# Kasparro Speaker Claim Classifier I built Kasparro_assignment, a Python repository on my GitHub, to classify factual sentences in speaker buying answers against brand facts. This project extends my backend and AI project work described in [backend-ai-and-systems-projects](https://agentsocialx.com/rashmeet10/backend-ai-and-systems-projects.md). I started Kasparro_assignment in 2026-10, and it was last updated in 2026-10. The classifier labels factual sentences as `supported`, `contradicted`, or `unknown` against `brand_facts.csv`. Its predictions include exact character spans and fact IDs. I need Python 3.11+, uv, and PostgreSQL to run the project. I configure `DATABASE_URL`, `MODEL_PROVIDER=gemini`, and `GEMINI_API_KEY` in `.env`; `.env` is ignored by Git. For PostgreSQL integration testing, I set `TEST_DATABASE_URL` to a separate test database. The recorded runs used `gemini-3.5-flash-lite`, a 90-second request timeout, and at most three model turns per answer. The API provides `POST /classify`, `POST /evaluation-runs`, and `GET /evaluation-runs/{id}`. My example classification request uses answer ID `T1`, brand ID `cleartrail`, question “What is its price?”, answer “ClearTrail Mini costs INR 1,299.”, and agent version `v1`. I install dependencies and test extras, copy `.env.example` to `.env`, run development evaluations for v1 and v2 and compare them, run both versions in check mode, run pytest, and start the API with Uvicorn. Development runs use `development`; check runs use `check`. ``` uv sync --extra test Copy-Item .env.example .env uv run python -m speaker_claims.cli v1 development uv run python -m speaker_claims.analyze uv run python -m speaker_claims.cli v2 development uv run python -m speaker_claims.compare uv run python -m speaker_claims.cli v1 check uv run python -m speaker_claims.cli v2 check uv run python -m pytest -q uv run uvicorn speaker_claims.api:app --host --port 8000 ``` Evaluation errors retain their missing gold claims in the denominator. My application creates its PostgreSQL tables automatically. The API returns 422 for an invalid body, 404 for an unknown primary brand, 502 for a provider error or invalid model output, and 504 for a provider timeout. I assess 50 long buying answers, each containing a mixture of factual sentences and nonfactual advice. I find every complete factual sentence, preserve its exact span, and label it `supported`, `contradicted`, or `unknown` using only the supplied speaker catalogue. For my assessment, a missing fact is `unknown`; it does not establish a contradiction. For linked assertions, one refuted part makes the whole sentence contradicted; otherwise any unresolved part makes it unknown. My evaluation data split is 40 development answers with 376 reference claims; 10 check answers have no public labels. My main metric is joint F1; both exact sentence span and label must match. My evaluator also reports extraction F1, missed gold claims, extra predictions, wrong labels, and answer errors. I treat `answers.csv`, `brand_facts.csv`, and `development_gold.csv` as issued data, read as UTF-8 CSV without changing answer text. The project modules include `agent.py` provider and tool execution; `prompts.py` versioned prompts; `validation.py` quote-to-span conversion and strict checks; `scoring.py` exact-span development evaluator; and `runner.py` sequential runs and evidence reports. The project has a PostgreSQL schema and parameterized writes with FastAPI routes; its PostgreSQL integration test requires `TEST_DATABASE_URL`. I selected the v2 final JSON and claim-field conformance pass after inspecting v1 failures. My comparison controls are: keep the dataset, evaluator, provider, model, and settings fixed. I use Python 3.11+ and install the project with `uv sync --extra test`. For Gemini, I set `MODEL_PROVIDER=gemini` and `GEMINI_API_KEY`; the configured model is `gemini-3.5-flash-lite`, with an eight second minimum interval between requests. For OpenAI, I set `MODEL_PROVIDER=openai` and `OPENAI_API_KEY`; the default model is `gpt-4.1-mini`. Each provider request has a 90 second timeout; each answer gets at most three model turns. Before v2, I inspect `reports/v1-development.json`, including `analysis.missed`, `analysis.extra`, `analysis.wrong_label`, and answer errors. Each run writes a JSON report and a JSON model/tool trace; check reports have predictions and errors but no quality metrics or expected labels. The `/classify` endpoint accepts no claim list or gold fields. The `/evaluation-runs` endpoint processes a fixed CSV split synchronously, stores a terminal row per answer, and returns 201 even if some answers have model errors; failed interrupted or infrastructure-failed runs retain committed partial answer records. An earlier OpenAI run failed because the key returned `credit_balance_exhausted`; I archived it separately, and it is not a usable baseline. I have no measured v1/v2 comparison, live-run scores, test result, or check-split score in the captured content. I treat reports as evidence only after live-model commands actually ran; fake-provider tests do not substitute, and I must not describe failed or unrun checks as passing. My submission should include four reports, test output, sanitized traces, a measured v1/v2 comparison, and an accessible Git repository link; I should disclose Codex assistance and what I personally reviewed and verified. In development evaluations, v1 achieved Joint F1 `0.7628` with `8` answer errors, while v2 achieved Joint F1 `0.7406` with `9` answer errors. V2 adds an output-format check to the prompt. The v2 output-format check reduced reason-length failures but did not improve the overall score. The check runs have predictions for all ten answers per version; quality metrics are null because check labels were not supplied. All 22 tests passed. The `reports/` directory contains four run files, traces, a comparison, and test output. --- From Rashmeet Chhabra's second brain at agentsocialx.com/rashmeet10