trade-intelligence-copilot
Trade Intelligence Copilot
Ask a plain-English question about Pakistan's sugar import records, and this tool looks the answer up in the actual records and shows the exact rows it used. When the records cannot support an answer, it says so instead of inventing one, the mistake most such tools make. It was graded on a 28-question test with answers computed from the records beforehand; every score, including its misses, is published below.
Verified end to end against a live Google Cloud Vertex AI project: ingestion, cited answers, the refusal path, and the full evaluation run were all exercised for real.
Overview
An agentic retrieval-augmented copilot over agricultural-trade data, with a real, published evaluation layer. Ask a plain-English question about Pakistani sugar-sector import data and get a grounded, cited answer, or an honest refusal when the data does not support one.
Most RAG demos answer confidently and are never measured. Two things make a trade copilot trustworthy: it must ground numeric answers in the actual database rather than guessing, and it must refuse when the data cannot support an answer.
This project measures exactly those properties end to end. It ships a 28-item test set with ground-truth gold answers, runs the whole system against the live Gemini API, and publishes the real metrics, including a check of the LLM judge against hand labels so the judge itself is validated.
Approach and architecture
An LLM planner (gemini-2.5-flash) reads a question and routes to tools: text-to-SQL over trade data, vector retrieval over a regulations corpus, an HS-code lookup, or several in one query. The final answer is synthesized only from the gathered evidence and cites its sources, or it refuses.
Key features
- Agentic planning that picks the right tools per question.
- Text-to-SQL passed through a read-only guard (single SELECT only, write and DDL keywords blocked, multiple statements rejected) on an immutable read-only SQLite connection.
- Grounded answers that cite the SQL query and rows, the HS-lookup rows, and the regulation chunks used.
- A deterministic no-evidence refusal path that fires before any synthesis call.
- A FastAPI service with POST /ask, GET /health, and a UI showing the answer, the tool-call trace, citations, and per-query cost and latency.
Results
| Metric | Result |
|---|---|
| SQL answer correctness (vs ground truth) | 15 / 15 = 100% |
| Retrieval hit rate | 8 / 10 = 80% |
| Refusal precision / recall | 1.00 / 1.00 |
| LLM-as-judge faithfulness (gemini-2.5-pro) | 19 / 22 = 86.4% |
| Judge vs human agreement (12 items) | 11 / 12 = 91.7% |
| Mean cost per query | $0.00098 |
| Latency p50 / p95 | 7.1 s / 9.9 s |
Reading the numbers honestly
SQL correctness is 100% only after a fix. In the first run the model wrote a name match without wildcards and returned 0 shipments instead of 88; tightening the schema prompt fixed it, and the gold numbers are computed directly from the database.
Faithfulness below 100% is real. Two of the misses were enumeration questions where the model padded its list with plausible product names beyond its evidence, which is exactly the failure a faithfulness metric exists to surface. Refusal was perfect on this set, but the set is small, so read it as the path works, not it never fails.
Data
The trade data is built from the ZaraiLink dataset into a compact SQLite database: companies, an HS-code hierarchy, and 2,232 import transactions over a 12-month window (Nov 2024 to Oct 2025). The regulations corpus is 16 authored plain-language explainers written for this demo.
Tech stack
Challenges
- pgvector was unavailable (PostgreSQL 18 without the extension, no Docker), so a small exact cosine store over a normalized numpy matrix replaced it; the trade-off is documented.
- The LLM judge was initially blind to the HS-lookup rows and flagged faithful answers as unfaithful; adding those rows to the citations fixed the harness and left only genuine misses.
- Text-to-SQL matched names literally and returned zero rows; because gold answers come from the database, the eval caught it immediately.
- Faithful is not the same as correct: a zero-row answer was judged faithful while being wrong, forcing a clear split between grounding and ground-truth correctness.
What I learned
- Ground-truth gold answers computed from the database are what make an LLM eval trustworthy.
- An LLM judge must be validated, and must see exactly what the answerer saw.
- Enumeration is where grounded models leak; point-fact answers stayed grounded.
- A cheap deterministic guard beats an expensive model call for refusal.