Fastapi Api Monitoring Dashboard

I use these troubleshooting checks when operating the stack: if Prometheus reports its target DOWN, I check curl http://localhost:8000/health and confirm both containers are on monitoring-net; if Grafana shows “No Data,” I check that its Prometheus datasource can reach http://prometheus:9090 and generate traffic with python scripts/run_failure_demo.py; for conflicts on ports 8000, 9090, 3000, or 9093, I check local bindings with netstat -ano. External email and Slack notifications require valid third-party credentials.

# Failure exercises
curl -i http://localhost:8000/simulate-error
curl -i http://localhost:8000/simulate-rate-limit
curl -i -X POST "http://localhost:8000/health/toggle-state?unhealthy=true"

# Tests, verification, and failure-demo traffic
pytest -v
python scripts/verify_monitoring.py
python scripts/run_failure_demo.py

My documented pytest -v results list 16 passing tests: 8 endpoint tests, 3 authentication tests, and 5 metrics tests. The test list covers root and health endpoints, unhealthy-state simulation, users and products, error, timeout and rate-limit simulations, missing, invalid and valid API keys, metric exposition and increments, and the active-request gauge lifecycle. The guide says evidence belongs in screenshots/: dashboard.png, prometheus-targets.png, and alert-proof.png. Its final checklist marks the API, RED instrumentation, /metrics, Docker Compose stack, 7-section Grafana dashboard, alert rules, all 4 failure scenarios, RCA reports, pytest suite, and operations and technical documentation complete; the captured material does not include the RCA contents or the verification utility’s output.

I built the FastAPI API Monitoring Dashboard as a production-grade observability and monitoring stack around FastAPI, Prometheus, Grafana, and Alertmanager. It is designed to track API health, latency, and HTTP status distributions, detect anomalies, trigger alerts, and support root-cause analysis. I describe its approach as based on the Google SRE Book and the RED method: Rate, Errors, Duration. My GitHub profile lists the FAST-API repository: my github profile.

• I expose application metrics through ASGI middleware in Prometheus exposition format. The sample API provides /, /health, /users, and /products endpoints with Pydantic validation; failure and access-control scenarios include /simulate-error, /simulate-timeout, /simulate-rate-limit, and /protected.

My request and monitoring flow is: client traffic reaches FastAPI on port 8000; Prometheus scrapes fastapi:8000/metrics; Grafana queries Prometheus with PromQL for dashboards; and firing alerts go to Alertmanager for notification handling.

• My ASGI PrometheusMiddleware tracks in-flight requests with fastapi_active_requests, request latency with fastapi_request_duration_seconds, and total requests with fastapi_requests_total.
• A psutil background collector gathers system metrics, and I expose the metrics at GET /metrics.
• My API also includes failure simulations at /simulate-error, /simulate-timeout, and /simulate-rate-limit; /protected requires X-API-Key.
• Grafana is provisioned from JSON with a Prometheus datasource and a dark-theme, 7-section dashboard named fastapi-monitoring-dashboard.

The README lists pytest unit and integration tests covering endpoints, authentication, and metrics, with httpx and requests among the testing tools. The supplied document context reports 16 passing pytest tests. The described stack uses Python 3.11+, FastAPI 0.110+, Prometheus 2.50.1, Alertmanager 0.27.0, Grafana 10.4.0, and Docker Compose v2.

The supplied document context says FastAPI serves the application on port 8000 and exposes application and host metrics at /metrics; Prometheus runs on port 9090, Grafana on port 3000, and Alertmanager on port 9093. It describes Prometheus scraping every 5 seconds and evaluating alert rules at the same interval, with 15-day TSDB retention and 5-second demo or 15-second production scrape intervals. These deployment details complement the README’s monitoring overview.

• Prometheus runs on port 9090, scrapes fastapi:8000/metrics every 5 seconds, and evaluates alert_rules.yml every 5 seconds.
• Grafana runs on port 3000. Its pre-provisioned dashboards have 7 visual monitoring sections and make real-time Prometheus queries.
• Alertmanager runs on port 9093, deduplicates and groups firing alerts, and has SMTP Email and Slack Webhook notification receivers.

The README links to a 5-minute demonstration covering server setup, the dashboard, failure simulations, Slack notifications, and Gmail SMTP alerts: https://www.loom.com/share/84959d2b553649eba5a9f97f7b4907fa. It does not specify the dashboard panels, most alert thresholds or repeat intervals, notification configuration, the four incidents’ identities, or measured monitoring results. Although the supplied document context reports passing tests and describes RCA reports, it does not provide their detailed contents.

My project separates the FastAPI application, monitoring configuration, Grafana provisioning, tests, operational scripts, incident documentation, and screenshot evidence. In app/, main.py provides the entrypoint and lifespan; config.py handles environment settings; schemas.py defines response models; logging_config.py configures structured JSON logging; metrics.py registers Prometheus metrics and a psutil collector; middleware.py instruments ASGI requests; and auth.py provides API-key security. Routers cover health and state toggling, users, products, and failure simulations. The monitoring/ directory holds Prometheus scrape and alert rules plus Alertmanager configuration; grafana/provisioning/ configures the Prometheus datasource and dashboard provider. I also include endpoint, authentication, and metrics tests, scripts/run_failure_demo.py, scripts/verify_monitoring.py, RCA documentation, and screenshot-capture guidance. The Dockerfile uses Python 3.11 and runs as a non-root user.

# Local setup and server
python -m venv venv
# Windows PowerShell: .\venv\Scripts\Activate.ps1
# Linux/macOS: source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
uvicorn app.main:app --host <redacted> --port 8000 --reload

# Run the complete stack and inspect or stop it
docker compose up -d --build
docker compose ps
docker compose down

# Tests and monitoring verification
pytest -v
python scripts/verify_monitoring.py

I document Swagger at http://localhost:8000/docs and ReDoc at http://localhost:8000/redoc; Prometheus is at http://localhost:9090 with targets at /targets and alerts at /alerts. I verify that the fastapi target is UP. Grafana is at http://localhost:3000, where the auto-provisioned API Monitoring Dashboard appears in the Observability folder; the documented default login is admin / admin. Alertmanager is at http://localhost:9093 for inspecting routes, silences, and receivers. My API also documents /dashboard, /api/alerts, /health/toggle-state, /protected, the three /simulate-* endpoints, and /metrics, in addition to the root, health, users, and products endpoints. The health endpoint can return 200 or 503; protected access can return 200 or 401; the error and rate-limit simulations return 500 and 429 respectively. The timeout simulation returns 200 after its default 3.5-second delay, exceeding the 3.0-second SLA.

My metrics cover completed requests, request-duration histograms, active requests, authentication failures, rate-limit responses, slow requests, application health and startup time, and process and host CPU and memory. The request counter is labeled by method, endpoint, and status code; the duration histogram uses method and endpoint labels and buckets [0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 3.0, 5.0, 10.0]. I expose these at /metrics. PromQL examples include sum(rate(fastapi_requests_total[1m])) for request rate, histogram_quantile(0.95, sum(rate(fastapi_request_duration_seconds_bucket[1m])) by (le)) for p95 latency, and avg_over_time(up{job="fastapi"}[30m]) * 100 for 30-minute uptime. My seven dashboard sections cover service overview, API performance, HTTP status codes, error trends, endpoint-level analysis, system and process health, and active alerts.

I describe email, Slack, and voice-call notifications as supported channels, while the captured setup details are specific to email and calls. Email uses TLS with smtp.gmail.com:587 and an app password, and formatted HTML notifications are sent for firing and resolved alerts. Optional Twilio calls are associated with critical APIDown and HTTP500Error incidents; the documented example uses +1234567890 and also describes a test-call API and a Live Alerts dashboard tab. External email and Slack notifications require valid third-party credentials; the captured material does not provide Slack setup details. I can generate the documented failure scenarios with python scripts/run_failure_demo.py. The project lists 16 passing tests across endpoints, authentication, and metrics, including tests for healthy and unhealthy health, simulated failures, API-key rejection, metric exposition and increments, and the active-request gauge lifecycle. The verification script is documented, but no run result is included. I intend validation screenshots for the dashboard, Prometheus targets, and alert evidence to live in screenshots/; four incident RCA reports are documented in docs/root_cause_analysis.md, though their findings are not included here.

I document operational settings of a 5s scrape interval for development and demos, 15s for production, 15 days of TSDB retention, and alert-rule evaluation every 5s. When running in Docker, my psutil system metrics measure container resources. I keep metric cardinality low by normalizing URL templates, run the Docker container as a non-root user, never log or reflect API keys, and do not expose stack traces or file paths to clients on 500 errors.

• Alert rules cover downtime, slow responses above 3 seconds, HTTP 500 errors, high failure rate, and health-check failures. Alertmanager groups alerts, uses repeat intervals, and has email (SMTP) and Slack webhook receivers.
• scripts/run_failure_demo.py simulates all 4 incidents and populates metrics. SRE post-incident reports document each incident’s cause, telemetry, and remediation.
• Prometheus scrapes up{job="fastapi"} for availability, and Grafana shows 30m rolling uptime. A high-resolution fastapi_request_duration_seconds histogram and fastapi_slow_requests_total counter track latency and slow requests; the stated SLA and alert threshold are above 3 seconds.
• I track 2xx, 4xx, and 5xx responses, including 401 authentication failures and 429 rate-limit events. A background collector records process CPU and RSS memory, plus system CPU and memory.

The assignment-requirements checklist marks all 12 listed requirements Complete, covering the sample API, failure simulations, availability and latency monitoring, status-code and host metrics, the Grafana dashboard, alerts and notifications, tests, the failure script, and RCA.