I evaluate PolicyLens with both an offline deterministic rule‑based evaluator and a live LLM‑based evaluator (qwen2.5 3 B + nomic‑embed‑text) on 50 synthetic cases (offline) and the first 10 cases (live). The suite records recall, classification accuracy, grounding, false‑compliance, missing‑evidence detection, reviewer agreement and latency.
- Policy retrieval recall: offline 0.920, live 0.976
- False compliance rate (PASS where truth is not PASS): offline 0.000, live 0.000
- Average latency per case: offline <0.1 s, live 146 s
Database runs on host port 5433 (container PostgreSQL port 5432).
Backend runs on port 8010 (seeds demo users and the 9 sample policies on first start).
Frontend runs on port 5174 (proxies /api to :8010).
Passwords need 10+ characters, at least one letter and one number, and can't contain the username.
Usernames are 3–32 lowercase characters.
I created PolicyLens, an AI‑driven policy and compliance evidence agent hosted on my GitHub @kolipakamahendra1-wq. It automates policy identification, requirement matching, evidence verification, and audit‑package generation, while human reviewers retain final approval.
The repository language is Python.
Started 2026-10, last updated 2026-10.
The default is Ollama qwen2.5:3b for evaluation and nomic-embed-text for embeddings, so there are no paid API calls.
Statuses: PASS, FAIL, UNKNOWN, NEEDS_HUMAN_REVIEW.
Never claim compliance without evidence – true.
Cite the exact policy section – true.
Separate policy text from interpretation – true.
Mark uncertain decisions UNKNOWN – true.
Never modify production systems – true.