I built Kasparro_assignment, a Python repository on my GitHub, to classify factual sentences in speaker buying answers against brand facts. This project extends my backend and AI project work described in backend ai and systems projects.
I started Kasparro_assignment in 2026-10, and it was last updated in 2026-10.
The classifier labels factual sentences as supported, contradicted, or unknown against brand_facts.csv.
Its predictions include exact character spans and fact IDs.
I need Python 3.11+, uv, and PostgreSQL to run the project.
I configure DATABASE_URL, MODEL_PROVIDER=gemini, and GEMINI_API_KEY in .env; .env is ignored by Git.
For PostgreSQL integration testing, I set TEST_DATABASE_URL to a separate test database.
The recorded runs used gemini-3.5-flash-lite, a 90-second request timeout, and at most three model turns per answer.
The API provides POST /classify, POST /evaluation-runs, and GET /evaluation-runs/{id}.
My example classification request uses answer ID T1, brand ID cleartrail, question “What is its price?”, answer “ClearTrail Mini costs INR 1,299.”, and agent version v1.
I install dependencies and test extras, copy .env.example to .env, run development evaluations for v1 and v2 and compare them, run both versions in check mode, run pytest, and start the API with Uvicorn. Development runs use development; check runs use check.
uv sync --extra test
Copy-Item .env.example .env
uv run python -m speaker_claims.cli v1 development
uv run python -m speaker_claims.analyze
uv run python -m speaker_claims.cli v2 development
uv run python -m speaker_claims.compare
uv run python -m speaker_claims.cli v1 check
uv run python -m speaker_claims.cli v2 check
uv run python -m pytest -q
uv run uvicorn speaker_claims.api:app --host <redacted> --port 8000
Evaluation errors retain their missing gold claims in the denominator.
My application creates its PostgreSQL tables automatically.
The API returns 422 for an invalid body, 404 for an unknown primary brand, 502 for a provider error or invalid model output, and 504 for a provider timeout.
I assess 50 long buying answers, each containing a mixture of factual sentences and nonfactual advice.
I find every complete factual sentence, preserve its exact span, and label it supported, contradicted, or unknown using only the supplied speaker catalogue.
For my assessment, a missing fact is unknown; it does not establish a contradiction.
For linked assertions, one refuted part makes the whole sentence contradicted; otherwise any unresolved part makes it unknown.
My evaluation data split is 40 development answers with 376 reference claims; 10 check answers have no public labels.
My main metric is joint F1; both exact sentence span and label must match.
My evaluator also reports extraction F1, missed gold claims, extra predictions, wrong labels, and answer errors.
I treat answers.csv, brand_facts.csv, and development_gold.csv as issued data, read as UTF-8 CSV without changing answer text.
The project modules include agent.py provider and tool execution; prompts.py versioned prompts; validation.py quote-to-span conversion and strict checks; scoring.py exact-span development evaluator; and runner.py sequential runs and evidence reports.
The project has a PostgreSQL schema and parameterized writes with FastAPI routes; its PostgreSQL integration test requires TEST_DATABASE_URL.
I selected the v2 final JSON and claim-field conformance pass after inspecting v1 failures.
My comparison controls are: keep the dataset, evaluator, provider, model, and settings fixed.
I use Python 3.11+ and install the project with uv sync --extra test.
For Gemini, I set MODEL_PROVIDER=gemini and GEMINI_API_KEY; the configured model is gemini-3.5-flash-lite, with an eight second minimum interval between requests.
For OpenAI, I set MODEL_PROVIDER=openai and OPENAI_API_KEY; the default model is gpt-4.1-mini.
Each provider request has a 90 second timeout; each answer gets at most three model turns.
Before v2, I inspect reports/v1-development.json, including analysis.missed, analysis.extra, analysis.wrong_label, and answer errors.
Each run writes a JSON report and a JSON model/tool trace; check reports have predictions and errors but no quality metrics or expected labels.
The /classify endpoint accepts no claim list or gold fields.
The /evaluation-runs endpoint processes a fixed CSV split synchronously, stores a terminal row per answer, and returns 201 even if some answers have model errors; failed interrupted or infrastructure-failed runs retain committed partial answer records.
An earlier OpenAI run failed because the key returned credit_balance_exhausted; I archived it separately, and it is not a usable baseline.
I have no measured v1/v2 comparison, live-run scores, test result, or check-split score in the captured content.
I treat reports as evidence only after live-model commands actually ran; fake-provider tests do not substitute, and I must not describe failed or unrun checks as passing.
My submission should include four reports, test output, sanitized traces, a measured v1/v2 comparison, and an accessible Git repository link; I should disclose Codex assistance and what I personally reviewed and verified.
In development evaluations, v1 achieved Joint F1 0.7628 with 8 answer errors, while v2 achieved Joint F1 0.7406 with 9 answer errors.
V2 adds an output-format check to the prompt.
The v2 output-format check reduced reason-length failures but did not improve the overall score.
The check runs have predictions for all ten answers per version; quality metrics are null because check labels were not supplied.
All 22 tests passed.
The reports/ directory contains four run files, traces, a comparison, and test output.