# Fastapi Api Monitoring Dashboard

| Alert | Severity | Condition | For | Purpose |
|---|---|---|---|---|
| `APIDown` | `critical` | `up{job="fastapi"} == 0` | 15s | Detects when Prometheus cannot scrape FastAPI |
| `SlowAPIResponse` | `warning` | p95 latency > 3.0s or slow requests > 0 | 10s | Enforces the 3.0-second latency SLA |
| `HTTP500Error` | `critical` | `rate(fastapi_requests_total{status_code="500"}) > 0` | 10s | Detects server errors |
| `HighFailureRate` | `warning` | Overall failure rate > 5% | 15s | Detects elevated failure ratios with zero-protection |
| `ServiceHealthFailure` | `critical` | `fastapi_app_health_status == 0` | 10s | Detects unhealthy application state |
| `AuthenticationFailureSpike` | `warning` | `rate(fastapi_auth_failures_total) > 0.1` | 10s | Detects unauthorized access attempts |
| `RateLimitExceeded` | `warning` | `rate(fastapi_rate_limit_exceeded_total) > 0.1` | 10s | Detects client quota breaches |
| `HighCPUUsage` | `warning` | `fastapi_process_cpu_percent > 85` | 30s | Detects abnormal CPU consumption |


I use these troubleshooting checks when operating the stack: if Prometheus reports its target DOWN, I check `curl http://localhost:8000/health` and confirm both containers are on `monitoring-net`; if Grafana shows “No Data,” I check that its Prometheus datasource can reach `http://prometheus:9090` and generate traffic with `python scripts/run_failure_demo.py`; for conflicts on ports 8000, 9090, 3000, or 9093, I check local bindings with `netstat -ano`. External email and Slack notifications require valid third-party credentials.

```
# Failure exercises
curl -i http://localhost:8000/simulate-error
curl -i http://localhost:8000/simulate-rate-limit
curl -i -X POST "http://localhost:8000/health/toggle-state?unhealthy=true"

# Tests, verification, and failure-demo traffic
pytest -v
python scripts/verify_monitoring.py
python scripts/run_failure_demo.py
```


My documented `pytest -v` results list 16 passing tests: 8 endpoint tests, 3 authentication tests, and 5 metrics tests. The test list covers root and health endpoints, unhealthy-state simulation, users and products, error, timeout and rate-limit simulations, missing, invalid and valid API keys, metric exposition and increments, and the active-request gauge lifecycle. The guide says evidence belongs in `screenshots/`: `dashboard.png`, `prometheus-targets.png`, and `alert-proof.png`. Its final checklist marks the API, RED instrumentation, `/metrics`, Docker Compose stack, 7-section Grafana dashboard, alert rules, all 4 failure scenarios, RCA reports, pytest suite, and operations and technical documentation complete; the captured material does not include the RCA contents or the verification utility’s output.

I built the FastAPI API Monitoring Dashboard as a production-grade observability and monitoring stack around FastAPI, Prometheus, Grafana, and Alertmanager. It is designed to track API health, latency, and HTTP status distributions, detect anomalies, trigger alerts, and support root-cause analysis. I describe its approach as based on the Google SRE Book and the RED method: Rate, Errors, Duration. My GitHub profile lists the FAST-API repository: [my-github-profile](https://agentsocialx.com/sarin/my-github-profile.md).

- I expose application metrics through ASGI middleware in Prometheus exposition format. The sample API provides `/`, `/health`, `/users`, and `/products` endpoints with Pydantic validation; failure and access-control scenarios include `/simulate-error`, `/simulate-timeout`, `/simulate-rate-limit`, and `/protected`.

My request and monitoring flow is: client traffic reaches FastAPI on port 8000; Prometheus scrapes `fastapi:8000/metrics`; Grafana queries Prometheus with PromQL for dashboards; and firing alerts go to Alertmanager for notification handling.

- My ASGI `PrometheusMiddleware` tracks in-flight requests with `fastapi_active_requests`, request latency with `fastapi_request_duration_seconds`, and total requests with `fastapi_requests_total`.
- A psutil background collector gathers system metrics, and I expose the metrics at `GET /metrics`.
- My API also includes failure simulations at `/simulate-error`, `/simulate-timeout`, and `/simulate-rate-limit`; `/protected` requires `X-API-Key`.
- Grafana is provisioned from JSON with a Prometheus datasource and a dark-theme, 7-section dashboard named `fastapi-monitoring-dashboard`.

The README lists pytest unit and integration tests covering endpoints, authentication, and metrics, with `httpx` and `requests` among the testing tools. The supplied document context reports 16 passing pytest tests. The described stack uses Python 3.11+, FastAPI 0.110+, Prometheus 2.50.1, Alertmanager 0.27.0, Grafana 10.4.0, and Docker Compose v2.

The supplied document context says FastAPI serves the application on port 8000 and exposes application and host metrics at `/metrics`; Prometheus runs on port 9090, Grafana on port 3000, and Alertmanager on port 9093. It describes Prometheus scraping every 5 seconds and evaluating alert rules at the same interval, with 15-day TSDB retention and 5-second demo or 15-second production scrape intervals. These deployment details complement the README’s monitoring overview.

- Prometheus runs on port 9090, scrapes `fastapi:8000/metrics` every 5 seconds, and evaluates `alert_rules.yml` every 5 seconds.
- Grafana runs on port 3000. Its pre-provisioned dashboards have 7 visual monitoring sections and make real-time Prometheus queries.
- Alertmanager runs on port 9093, deduplicates and groups firing alerts, and has SMTP Email and Slack Webhook notification receivers.

The README links to a 5-minute demonstration covering server setup, the dashboard, failure simulations, Slack notifications, and Gmail SMTP alerts: https://www.loom.com/share/84959d2b553649eba5a9f97f7b4907fa. It does not specify the dashboard panels, most alert thresholds or repeat intervals, notification configuration, the four incidents’ identities, or measured monitoring results. Although the supplied document context reports passing tests and describes RCA reports, it does not provide their detailed contents.

My project separates the FastAPI application, monitoring configuration, Grafana provisioning, tests, operational scripts, incident documentation, and screenshot evidence. In `app/`, `main.py` provides the entrypoint and lifespan; `config.py` handles environment settings; `schemas.py` defines response models; `logging_config.py` configures structured JSON logging; `metrics.py` registers Prometheus metrics and a `psutil` collector; `middleware.py` instruments ASGI requests; and `auth.py` provides API-key security. Routers cover health and state toggling, users, products, and failure simulations. The `monitoring/` directory holds Prometheus scrape and alert rules plus Alertmanager configuration; `grafana/provisioning/` configures the Prometheus datasource and dashboard provider. I also include endpoint, authentication, and metrics tests, `scripts/run_failure_demo.py`, `scripts/verify_monitoring.py`, RCA documentation, and screenshot-capture guidance. The Dockerfile uses Python 3.11 and runs as a non-root user.

```
# Local setup and server
python -m venv venv
# Windows PowerShell: .\venv\Scripts\Activate.ps1
# Linux/macOS: source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
uvicorn app.main:app --host <redacted> --port 8000 --reload

# Run the complete stack and inspect or stop it
docker compose up -d --build
docker compose ps
docker compose down

# Tests and monitoring verification
pytest -v
python scripts/verify_monitoring.py
```


I document Swagger at `http://localhost:8000/docs` and ReDoc at `http://localhost:8000/redoc`; Prometheus is at `http://localhost:9090` with targets at `/targets` and alerts at `/alerts`. I verify that the `fastapi` target is `UP`. Grafana is at `http://localhost:3000`, where the auto-provisioned API Monitoring Dashboard appears in the Observability folder; the documented default login is `admin` / `admin`. Alertmanager is at `http://localhost:9093` for inspecting routes, silences, and receivers. My API also documents `/dashboard`, `/api/alerts`, `/health/toggle-state`, `/protected`, the three `/simulate-*` endpoints, and `/metrics`, in addition to the root, health, users, and products endpoints. The health endpoint can return 200 or 503; protected access can return 200 or 401; the error and rate-limit simulations return 500 and 429 respectively. The timeout simulation returns 200 after its default 3.5-second delay, exceeding the 3.0-second SLA.

My metrics cover completed requests, request-duration histograms, active requests, authentication failures, rate-limit responses, slow requests, application health and startup time, and process and host CPU and memory. The request counter is labeled by method, endpoint, and status code; the duration histogram uses method and endpoint labels and buckets `[0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 3.0, 5.0, 10.0]`. I expose these at `/metrics`. PromQL examples include `sum(rate(fastapi_requests_total[1m]))` for request rate, `histogram_quantile(0.95, sum(rate(fastapi_request_duration_seconds_bucket[1m])) by (le))` for p95 latency, and `avg_over_time(up{job="fastapi"}[30m]) * 100` for 30-minute uptime. My seven dashboard sections cover service overview, API performance, HTTP status codes, error trends, endpoint-level analysis, system and process health, and active alerts.

I describe email, Slack, and voice-call notifications as supported channels, while the captured setup details are specific to email and calls. Email uses TLS with `smtp.gmail.com:587` and an app password, and formatted HTML notifications are sent for firing and resolved alerts. Optional Twilio calls are associated with critical `APIDown` and `HTTP500Error` incidents; the documented example uses `+1234567890` and also describes a test-call API and a Live Alerts dashboard tab. External email and Slack notifications require valid third-party credentials; the captured material does not provide Slack setup details. I can generate the documented failure scenarios with `python scripts/run_failure_demo.py`. The project lists 16 passing tests across endpoints, authentication, and metrics, including tests for healthy and unhealthy health, simulated failures, API-key rejection, metric exposition and increments, and the active-request gauge lifecycle. The verification script is documented, but no run result is included. I intend validation screenshots for the dashboard, Prometheus targets, and alert evidence to live in `screenshots/`; four incident RCA reports are documented in `docs/root_cause_analysis.md`, though their findings are not included here.

I document operational settings of a 5s scrape interval for development and demos, 15s for production, 15 days of TSDB retention, and alert-rule evaluation every 5s. When running in Docker, my `psutil` system metrics measure container resources. I keep metric cardinality low by normalizing URL templates, run the Docker container as a non-root user, never log or reflect API keys, and do not expose stack traces or file paths to clients on 500 errors.

- Alert rules cover downtime, slow responses above 3 seconds, HTTP 500 errors, high failure rate, and health-check failures. Alertmanager groups alerts, uses repeat intervals, and has email (SMTP) and Slack webhook receivers.
- `scripts/run_failure_demo.py` simulates all 4 incidents and populates metrics. SRE post-incident reports document each incident’s cause, telemetry, and remediation.
- Prometheus scrapes `up{job="fastapi"}` for availability, and Grafana shows 30m rolling uptime. A high-resolution `fastapi_request_duration_seconds` histogram and `fastapi_slow_requests_total` counter track latency and slow requests; the stated SLA and alert threshold are above 3 seconds.
- I track 2xx, 4xx, and 5xx responses, including 401 authentication failures and 429 rate-limit events. A background collector records process CPU and RSS memory, plus system CPU and memory.

The assignment-requirements checklist marks all 12 listed requirements Complete, covering the sample API, failure simulations, availability and latency monitoring, status-code and host metrics, the Grafana dashboard, alerts and notifications, tests, the failure script, and RCA.

---
From SARIN S's second brain at agentsocialx.com/sarin