trade-intelligence-copilot
Trade Intelligence Copilot
Ask a plain-English question about Pakistan's sugar import records, and this tool looks the answer up in the actual records and shows the exact rows it used. When the records cannot support an answer, it says so instead of inventing one, the mistake most such tools make. It was graded on a 28-question test with answers computed from the records beforehand; every score, including its misses, is published below.
Verified end to end against a live Google Cloud Vertex AI project: ingestion, cited answers, the refusal path, and the full evaluation run were all exercised for real.
Overview
An agentic retrieval-augmented copilot over agricultural-trade data, with a real, published evaluation layer. Ask a plain-English question about Pakistani sugar-sector import data and get a grounded, cited answer, or an honest refusal when the data does not support one.
Key features
- Agentic planning that picks the right tools per question.
- Text-to-SQL passed through a read-only guard (single SELECT only, write and DDL keywords blocked, multiple statements rejected) on an immutable read-only SQLite connection.
- Grounded answers that cite the SQL query and rows, the HS-lookup rows, and the regulation chunks used.
- A deterministic no-evidence refusal path that fires before any synthesis call.
- A FastAPI service with POST /ask, GET /health, and a UI showing the answer, the tool-call trace, citations, and per-query cost and latency.
Results
| Metric | Result |
|---|---|
| SQL answer correctness (vs ground truth) | 15 / 15 = 100% |
| Retrieval hit rate | 8 / 10 = 80% |
| Refusal precision / recall | 1.00 / 1.00 |
| LLM-as-judge faithfulness (gemini-2.5-pro) | 19 / 22 = 86.4% |
| Judge vs human agreement (12 items) | 11 / 12 = 91.7% |
| Mean cost per query | $0.00098 |
| Latency p50 / p95 | 7.1 s / 9.9 s |
Reading the numbers honestly
SQL correctness is 100% only after a fix. In the first run the model wrote a name match without wildcards and returned 0 shipments instead of 88; tightening the schema prompt fixed it, and the gold numbers are computed directly from the database.
Faithfulness below 100% is real. Two of the misses were enumeration questions where the model padded its list with plausible product names beyond its evidence, which is exactly the failure a faithfulness metric exists to surface. Refusal was perfect on this set, but the set is small, so read it as the path works, not it never fails.
Data
The trade data is built from the ZaraiLink dataset into a compact SQLite database: companies, an HS-code hierarchy, and 2,232 import transactions over a 12-month window (Nov 2024 to Oct 2025). The regulations corpus is 16 authored plain-language explainers written for this demo.