IRS Pub 17 Tax Q&A
Retrieval-augmented question answering over 333 pages of IRS tax year 2025 publications, built to measure what each retrieval technique is actually worth rather than to assemble a stack.
Measure first, then change one thing at a time.
Most RAG projects stack every technique at once and report one number. This one starts from a deliberately naive pipeline, builds an evaluation set that can prove it wrong, and adds a single change per step. A technique stays only if it moves the metric by more than noise.
The corpus is Pub 17, Pub 501, Pub 502, the Form 1040 instructions and Schedules 1, 1-A, 2 and 3. Tax year 2025 postdates model training: it adds the Schedule 1-A deductions, a $2,200 child tax credit and a $40,000 SALT cap. A model answering from memory gets these confidently wrong, so the eval measures retrieval, not recall of pretraining.
Two paths: ingest once, answer on request.
Reranking did most of the work.
Section-aware chunking cut the PDFs along their own outline instead of fixed 350-word windows, fixing 8 of the baseline’s 10 misses. Over-fetching 20 dense candidates and rescoring them with a cross-encoder then took recall@1 from 63.3% to 90.0%. That is the ceiling: dense recall@20 is also 98.3%, so the reranker ordered the pool as well as the pool allowed.
Show every configuration as a table
| Configuration | recall@1 | recall@3 | recall@5 | MRR@5 | loose r@5 | answer acc. |
|---|---|---|---|---|---|---|
| Naive 350-word windows, dense only (baseline) | 51.7% | 78.3% | 83.3% | 0.642 | 86.7% | 86.7% |
| Section-aware chunking | 63.3% | 86.7% | 91.7% | 0.753 | 95.0% | — |
| + cross-encoder reranking, 20 → 5 | 90.0% | 98.3% | 98.3% | 0.939 | 98.3% | 98.3% |
| Heading path in the embedded text (rejected) | 55.0% | 85.0% | 93.3% | 0.696 | 96.7% | — |
| Fold sections under 40 words (rejected) | 61.7% | 83.3% | 90.0% | 0.734 | 93.3% | — |
| Ablation: naive chunks + reranking | 90.0% | 93.3% | 96.7% | 0.924 | 98.3% | — |
| Hybrid full-text (RRF), no rerank (rejected) | 46.7% | 73.3% | 81.7% | 0.605 | 90.0% | — |
| Hybrid full-text (RRF) + rerank (rejected) | 90.0% | 98.3% | 98.3% | 0.939 | 98.3% | — |
| Generated section context, no rerank (rejected) | 60.0% | 90.0% | 95.0% | 0.745 | 96.7% | — |
| Generated section context + rerank (rejected) | 90.0% | 95.0% | 95.0% | 0.922 | 96.7% | — |
The ablation matters more than the headline. Naive chunks with reranking reach 90.0% at recall@1 too, so once the reranker is in place, section chunking is worth about one question. It stays because it costs nothing at query time and never measured worse, but the claim it supports is small.
The one remaining miss is out of reach, not misranked. The question spells out “additional child tax credit”; its answer chunk says “ACTC” and sits at dense rank 49, outside the 20-candidate pool. Widening the pool to 50 would fix it, and it wasn’t done: that would be tuning the pipeline to a question it is scored on.
Two adopted, two rejected.
Section-aware chunking
Chunks follow the PDF outline, so tables stay whole and worked examples keep their heading.
Cross-encoder reranking
bge-reranker-base rescores 20 dense candidates and keeps 5.
Hybrid full-text search
Postgres tsvector fused by Reciprocal Rank Fusion. Tax prose repeats “credit” and “tax” on nearly every page, so term frequency pulled noise into the top five. With reranking it changed nothing and added 15 ms at p50.
Contextual enrichment
Claude wrote a short context for each of 675 sections, prepended before embedding. Every difference was two questions, within noise. The contexts cost $2.44 and are committed, so the rows reproduce for free.
An eval built to be falsifiable, not plausible.
- Gold is anchored on pages and verbatim spans, not chunk ids. Chunk ids don’t survive a re-chunk, which would silently invalidate the baseline the new chunker is measured against.
- A hit means the chunk contains the answer. The first harness accepted page overlap alone and overstated every configuration. Section chunking made the gap matter: the 1040 instructions bookmark every form line, producing fragments like “Line 1a” that sit on the right page and hold nothing.
- Every question is checked against the PDFs.
verify_questions.pyconfirms each gold page contains the answer verbatim and exits nonzero on any failure, so CI gates on it. - Gold lists every page that genuinely answers. The corpus duplicates heavily. Scanning for every answer-bearing page grew gold from 85 to 188 pages and moved the baseline from 75.0% to 86.7%: the retrieval hadn’t changed, the ruler had.
- Answers are graded on exact keys, not by a model. Dollar amounts, rates and yes/no answers don’t need an LLM judge, and one would make the metric depend on a second model.
Three gates before any model call.
POST /ask wraps the pipeline in FastAPI. Every outcome, answered, refused or failed, is written to query_log with the caller, role, tokens, cost, per-stage latency and retrieved chunk ids.
CI that spends nothing, CD to a public image.
- CI verifies the corpus against pinned SHA-256 checksums (the IRS republishes PDFs under the same URLs), runs the retrieval eval and fails if strict recall@5 drops below 96.7%. A second job ingests into a pgvector service container and runs the service tests with generation faked. No API key, no spend.
- On its first run on GitHub’s CPU runners, CI reproduced the GPU numbers exactly: 90.0% recall@1, 98.3% recall@5, the same single miss. Service tests passed 10 of 10.
- CD builds the API image with both models baked in and publishes it to GitHub Container Registry on a version tag.
Where the remaining points are.
- Query expansion for abbreviations. The one retrieval miss is “ACTC” against a spelled-out question, and dense, full-text and context-enriched retrieval all failed to reach it.
- A bigger eval with a held-out split. At n=60 one question is 1.7 points, so only the chunking and reranking gains clear noise.