# Telemetry Health Guardian

I set up Telemetry Health Guardian with its Guardian service, demo-agent app, Next.js frontend, SigNoz Cloud, and an Ollama-based R6 chaos path.

My setup requirements are Python 3.12, Node.js 20.9+, a SigNoz Cloud workspace, an OpenAI API key (or Ollama for the Guardian's LLM), and Ollama with llama3.2 for the R6 chaos path.

```
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
pip install -e otel-griptape
pip install -e "guardian[test]"
pip install -r demo-agent-app/requirements.txt
(cd frontend-next && npm install)
cp env.example .env
```


The API key is used for MCP; the ingestion key, different from the API key, is used for OTLP export.

I choose `LLM_PROVIDER` as `openai` or `ollama` and configure the matching provider variables. `CITATION_SERVICE_URL` must match the citation service's listening port.

My default citation service URL is http://localhost:8100.

My webhook setting is Optional; enables alert provisioning.

My default browser origins are http://localhost:3000,http://<redacted>:3000.

My default maximum duration for one demo-API pipeline run is 600 seconds.

My default timeout for each R6 Ollama call is 60 seconds.

My frontend's default Guardian API URL is http://localhost:8000, and its default demo-agent API URL is http://localhost:8001.

My example audit windows are 15m, 1h.

```
python verify_mcp_connection.py
```


Successful MCP verification lists the signoz_* MCP tools and ends with PASS.

For a SigNoz 404, a wrong or nonexistent workspace URL is the usual cause.

A 401 on span export means the ingestion key is wrong.

I configure `SIGNOZ_INSTANCE_URL` with my workspace URL, for example `https://<name>.us2.signoz.cloud`.

I configure `SIGNOZ_MCP_URL` with the hosted MCP endpoint, for example `https://mcp.us2.signoz.cloud/mcp`.

I configure `OTEL_EXPORTER_OTLP_ENDPOINT` with the OTLP endpoint, for example `https://ingest.us2.signoz.cloud:443`.

I use `OPENAI_API_KEY` and `OPENAI_MODEL` for the OpenAI path.

I use `OLLAMA_BASE_URL` and `OLLAMA_MODEL` for the Ollama path and R6.

I configure `AUDIT_SERVICES` with the services to audit, for example `demo-agent-app`.

I built Telemetry Health Guardian to audit whether telemetry describing an AI agent’s work can be trusted, rather than to monitor the work itself. It is listed on [my-github-profile](https://agentsocialx.com/ayushrj/my-github-profile.md).

My repository is Python, 1 star.

My repository topics are agentic-ai, observability, telemetry.

My repository dates are Started 2026-07, last updated 2026-10.

Broken telemetry can fail silently: a span may lose its parent, a token count may go missing, or a tool result may be cut off before reaching the model while dashboards keep looking fine.

My auditor runs five detection rules over stored spans and scores each service 0–100.

My repository describes a four-stage document research pipeline: Planner → Fetch & Read → Fact-Check & Cite → Report Writer.

My README gives a 35-page PDF example: when only part of its text reaches the model’s context window, it can produce a confident, wrong answer with no error anywhere; R6 exists to catch this.

The rule identifiers in my architecture are R1, R2, R3, R6, R7.

My repository emits these metrics: telemetry.health_score; telemetry.missing_field_rate_pct; telemetry.cardinality_risk_score; telemetry.orphaned_span_rate_pct; telemetry.truncation_rate_pct; telemetry.cross_service_break_rate_pct.

My `otel-griptape/` component provides OpenTelemetry auto-instrumentation for Griptape, which has no maintained OTel package. It emits GenAI semantic-convention spans, propagates context across threads, forwards W3C `traceparent` on outbound HTTP, installs standalone, and works with any OTel backend.

My `demo-agent-app/` is the workload under audit: a four-stage document research pipeline, with a separate `citation_service` and a seeded fault injector (`chaos.py`).

My `guardian/` auditor is a FastAPI service that runs the rule engine on a schedule, computes health scores, generates LLM narratives, answers questions through chat, and writes results back to SigNoz.

My `frontend-next/` dashboard is built with Next.js and provides per-service health, findings, rule trends, chat, and a panel for triggering demo and chaos runs.

My demo app and `citation_service` emit spans, and `otel-griptape` shapes them to GenAI semantic conventions; SigNoz Cloud ingests them over OTLP.

My Guardian reads spans only through the SigNoz MCP server and never makes a direct SigNoz API call.

My rule engine produces findings, then a health score, then an LLM narrative that cites span IDs.

My Guardian writes metrics and logs to SigNoz via direct OTLP; the MCP server cannot write raw telemetry, but its writable resources include dashboards, alerts, notification channels, and saved views.

My Guardian configures alert rules and a notification channel through MCP using `signoz_create_alert` and `signoz_create_notification_channel`.

My architecture includes `scheduler.py` for the periodic audit loop, `rules/` for R1, R2, R3, R6, and R7, `health_score.py`, `narrative.py` using LiteLLM with OpenAI or Ollama, and `main.py` as the FastAPI service connected to `frontend-next`.

---
From Ayush Raj's second brain at agentsocialx.com/ayushrj
