Project overview
Auditable AI Data Analysis
An AI pipeline that turns a raw dataset and a business question into a finished, checked analysis, with no code written by the user. Every number is calculated by code, every step is checked automatically, and every judgment call is explained so a person can review it and change it.
The machine does the coding. People keep the judgment.
- Python
- LLM agents
- AI guardrails
- Data cleaning & quality
- Statistics
- Observability (Langfuse)
- Testing (700+ tests)
- DuckDB
How it works: three steps
A short brief
Clean, then analyse
A checked result
↻ Disagree with a decision? Write your choice into the brief and run again. Your expertise corrects the machine.
Why the result can be trusted
The AI decides what to look at; tested code produces every figure. The AI never calculates a number: every figure it writes is traced to a tool result.
Work only moves forward if it passes. Problems are flagged for a person, or the run stops and says why.
Each judgment call shows the options, the choice, the reasoning and the AI's confidence.
Uncertain choices are marked for review, and any decision can be overruled and the analysis re-run.
What you get: one answer, with more depth on demand
The answer in plain words
Up to four findings with charts, suggested actions, and a green / yellow / red rating of how far to trust it.
For decision makers · HTML or PDF · open a live example
The full reasoning
All charts and results, every data problem and fix, every decision with its effect, and a cost overview.
For analysts and reviewers · e.g. the decisions
The evidence
The cleaned data, removed rows with the reason, and a complete log of every step and every AI attempt.
For audit and reuse
Architecture
Like a well-run analysis team: AI specialists make the judgment calls, code does the arithmetic, and checkpoints sign off each stage.
Each has one job: find problems, fix them, review the fix, plan the analysis, interpret the results.
Reads the data and calculates every number with tested tools. Same input, same answer.
Automatic, rule-based inspections between stages. Nothing moves on without passing.
The four checkpoints: what each one protects against
Counts every row, catches broken lines and wrong encoding. Stops a bad file before any AI cost.
What it checks
- Lines in the file = rows loaded + rows rejected
- Rejected lines: flag above 0.1%, block above 1%
- Unique column names; file hashes recorded
- Every column has a type, or a reason why not
- Statements in the brief hold in the data
No rows lost or values blanked without explanation; re-running the fixes gives the same result.
What it checks
- Types and columns match the diagnosis
- Every changed cell and removed row is accounted for, with a reason
- Untouched columns are identical to the raw data
- Row loss: flag above 5%, block above 20%
- The target is never filled in; replaying the log reproduces the result
The columns exist, the methods can run on this data, and excluded data is not used.
What it checks
- Every column exists; every method is a tested tool
- The target is the brief's; IDs, personal data and excluded columns are not analysed
- A column that gives the answer away is excluded
- Enough rows for each question, or it is marked not possible
Each number in the text is traced to a tool result; no overclaiming. Only checked text reaches the report.
What it checks
- Every question in the plan is answered
- Every finding cites tool results that ran
- Every number matches a logged result; a number the AI worked out itself is rejected
- A finding without real evidence cannot be called strong
Why this design matters
AI is good at judgment but can make up numbers. Here the two are separated: the AI never calculates a figure, and code never makes a judgment call. Each hand-over between stages follows a fixed format, so the checkpoints can verify it mechanically.
Where the person comes in
Every judgment call is written to an editable decision list (decisions.yaml), and the analysis plan can be edited too. Change a choice, or write it into the brief, and run again. The machine redoes the work; your call is the one that counts.
Monitoring every AI call
Each run folder is the audit trail of one analysis. On top of that, every AI call can be traced in Langfuse: which specialist made it, with which model, what it was sent and answered, the tokens, the cost and the time. That shows where money and time go, and what changed after a model or prompt was swapped. Tracing is optional and can never stop a run.
Show as a table
| Specialist | Calls | Cost | Share | Time |
|---|---|---|---|---|
| Analyst | 5 | $0.565 | 65% | 113 s |
| Diagnostic | 1 | $0.126 | 15% | 39 s |
| Reviewer | 2 | $0.100 | 12% | 23 s |
| Planner | 1 | $0.041 | 5% | 20 s |
| Coder | 1 | $0.034 | 4% | 9 s |
What it shows, and what it is for
- Cost control: the analyst is two thirds of the cost, because each round of tool calls re-sends the earlier results. That is the first place to save.
- Model choice: each specialist has its own model. On the cheaper model, the coder, reviewer and planner cost about half of what they did in an earlier all-Opus run (on a different dataset, so a first indication, not a measurement).
- Debugging: the exact prompt and reply behind a weak finding or a blocked checkpoint.
- Next: attach checkpoint outcomes as scores, so quality and cost sit side by side.
Only what the AI is sent is traced: column names, summaries and the brief, never the rows.
Validation: how I know it works
I tested the pipeline the way you would test a new analyst: on public datasets that many people have already analysed, comparing what it found and decided with what others report.
How it was validated
- Three well-studied public datasets from Kaggle: customer churn, Titanic survival and mobile money fraud, each with many published analyses to compare against.
- The findings: the pipeline found the same main drivers as public analyses, with closely matching figures. For example, 42.7% churn for month-to-month contracts, where about 43% is commonly reported.
- The cleaning decisions: compared with common practice. Unclear values were kept and explained rather than silently dropped, and in the fraud data it excluded misleading balance columns because the dataset's own notes warned about them.
Limits of this validation
- Informal: compared with common practice, not scored against a fixed answer key.
- Well-known data: the AI may have seen analyses of these datasets; unseen data is the real test.
- Few runs: three datasets, one run each, so consistency across repeated runs is not yet measured.
- Scope: it shows associations, not causes, and is a prototype: local, single user, files only.
Next step: a systematic evaluation on data with known answers, including unseen data.
Let's work together
I would look forward to collaborating on implementing it for an organisation's needs: adapting it to your data, your rules and your review process.