Written from 111 named sources · Aug 11 · first result Context Optimization for Citation-Critical Generation: A Production Architecture Prepared for the technical founder. Assume nothing about the current pipeline survives contact with this document unchanged. The Verdict Up Front You are passing the full corpus into every generation call and paying for it three times: in tokens, in latency, and — most expensively — in silent citation drift you cannot detect without reading every output by hand. The fix is not a bigger context window. It is not a better prompt. It is an engineered context supply chain: a four-stage pipeline in which evidence is captured immutably, retrieved deterministically, cited transactionally, and verified independently before delivery. Three things you can ship this week, in dependency order: Prompt caching on a stable prefix — up to 90% reduction on repeated input tokens and up to 85% latency reduction on long prompts, with zero change to what the model sees [41][19]. This buys you runway. It fixes nothing about correctness. An immutable evidence registry with deterministic quote matching — zero LLM calls, layered on top of your current full-context pipeline as a post-hoc audit gate. This makes silent drift a hard error. Hybrid retrieval (dense + BM25) with reranking to replace full-corpus passing, only after (2) proves your citation baseline. Do not invert that order. If you cut context before you can measure citation recall, you will trade an expensive-but-correct system for a cheap-and-wrong one and have no instrumentation to notice. The Core Paradigm Shift 1.1 Why brute-force context fails Three failure axes, each independent: Cost explosion. Prompt caching is priced as a discount on repeated prefixes, not on volume. Cache reads cost roughly 0.1× base input rate; a 5-minute cache write costs 1.25× and a 1-hour write 2× [24]. Without caching, a repeated 100K-token prefix is billed at full rate on every call — and every regeneration, verification pass, and revision multiplies it. One production measurement: a workload with a shared prefix moved from 0% to 99.3% cache hit rate and dropped input cost roughly 90% ($6.72 → $0.57 per 1,000 requests) purely by marking the prefix and moving the dynamic field to the tail [85]. The corollary is brutal: without deliberate prefix discipline, you are paying full price and don't know it. Latency creep. Anthropic's own numbers on a 100K-token cached prompt: time-to-first-token drops from 11.5s to 2.4s [44]. That 11.5s is what full-context passing costs you on every call, and it compounds across the retrieve → generate → verify → regenerate loop. Silent citation drift. This is the one that ends careers. Empirical results are unambiguous: LLMs are competent at document-level attribution and poor at span-level attribution. Prompt-based citation generation reaches only 12.80 Snippet-F1 on ASQA, while posthoc span alignment against the source reaches 61.87 — a ~4.8× improvement from a mechanism that involves no better model, only better grounding architecture [45]. In the same evaluation, 81.8% of BioASQ citations targeted only the first two of five context documents, a primacy bias consistent with the lost-in-the-middle phenomenon [45]. Larger windows do not fix this. Independent evaluation places effective context at 50–65% of advertised capacity for most models, with degradation measurable at every length increment rather than only near the limit [7]. The NoLiMa benchmark found 11 of 13 models advertising ≥128K context dropped below 50% of their short-context baseline by 32K tokens [7]. A 1M-token window is capacity, not comprehension. 1.2 Why context must be a supply chain A supply chain has provenance at every hop, inspection gates between stages, and the ability to recall a defective batch. A context bucket has none of these. When a citation is wrong, you cannot tell whether extraction failed, retrieval failed, packing dropped the evidence, or the answer model received good evidence and refused to use it. The distinction that matters operationally: reducing the amount of information sent (retrieval, routing, compression) is a fundamentally different lever from reducing the cost of transmitting the same information (caching). Caching preserves your evidence byte-for-byte and therefore cannot degrade citation recall. Retrieval can. Compression will, if applied to citeable text. The 4-Stage Context Architecture Stage 0 — Ingestion: the evidence registry Before any optimization, you need an authoritative object that is not the model's output. The registry is that object. Every span carries: Field Purpose source_id Stable document identity source_version_hash SHA-256; prevents citations silently surviving a restated filing page_number, paragraph_id Human-auditable anchors char_start, char_end Offsets into canonical extracted text verbatim_text Exact bytes; the sole basis for quotations table_ref {table_id, row, col} for cell-level attribution fact_type ACTUAL / LIMIT / FORMULA / DEFINITION / TEMPORAL embedding Dense vector sparse_terms BM25-indexed tokens Two properties are load-bearing. First, the registry is append-only: source revisions create new entries; old entries are superseded, never overwritten. Second, spans are validated at ingestion, not at read time. The VeNRA architecture formalizes this as Double-Lock Grounding: Lock 1 verifies that source_text[char_start:char_end] equals verbatim_text (catching OCR shifts and table-boundary misalignment); Lock 2 verifies that the metric name tokens overlap the source chunk above a semantic threshold, rejecting "phantom metrics" where a real number is attached to a fabricated metric name [71]. Facts failing either lock are stamped UNALIGNED and excluded from retrieval. Be calibrated about what this buys. In VeNRA's own audit, 67.4% of candidate facts contained no hard numeric anchor at all; hard-anchor checking is one guardrail inside a broader support validator, not proof that every zero-anchor fact is verified [71]. The same audit rejected 11 of 132 candidate facts, most commonly for insufficient source overlap or invented hard values [71]. Non-negotiable structural rule for tables. A value like "12.4%" is not evidence. The citeable unit is table title + column headers + row label + relevant cells + footnotes. Retrieval must expand a cell into its minimum interpretive envelope, or you will cite a number without its period, denominator, or GAAP-vs-adjusted qualifier. Stage 1 — Retrieval and Routing, with KV-cache reuse Retrieval. Dense-only retrieval fails predictably in citation-heavy technical domains. Dense embeddings optimize for semantic similarity, not exact matching, so they miss product codes, clause references, version strings, and rare identifiers; a query for "Section 4.2.1(b) of the Master Services Agreement" should return that clause, not conceptually adjacent contract language [13]. Worse, mathematically opposite terms cluster together: "Net Income" and "Net Sales" occupy proximate vectors because their linguistic contexts are near-identical [71]. Financial-domain studies confirm generic embeddings struggle with finance-specific polarity ("short" vs. "long") [71]. There is also a theoretical bound: fixed-length dense representations have known limitations for precise retrieval from long documents [42]. Hybrid retrieval fixes the complementary blind spots. Run dense and sparse in parallel, fuse via Reciprocal Rank Fusion: $$\text{RRF}(d) = \sum_{c \in \mathcal{C}} \frac{w_c}{k + \text{rank}_c(d)}$$ with $k = 60$ as the standard constant [30]. RRF operates on rank positions, not raw scores, which sidesteps the score-incompatibility problem: BM25 scores are unbounded positive; cosine similarity is bounded in $[-1,1]$. Naïvely weight-averaging them gives BM25 dominant weight by default [30]. On the WANDS e-commerce benchmark, baseline BM25 (0.6983 NDCG) and pure KNN (0.6953) are statistically indistinguishable, while basic RRF reaches 0.7068 and field-boosted hybrid reaches 0.7497 [30]. Anthropic's own contextual-retrieval work reports that combining contextual embeddings with contextual BM25 reduced top-20 retrieval failure by 49%, and adding reranking pushed that to 67% (5.7% → 1.9%) [44]. Add a deterministic lexical gate before candidates enter context. Compute token-intersection recall rather than Jaccard — with verbose financial chunks ($ C \gg Q $), the Jaccard union denominator forces the score toward zero even on perfect matches [71]: $$R_{lex} = \frac{\sum_{t \in \text{set}(Q)} \min(\#(t,Q), \#(t,C))}{ Q }$$ Critically, strip non-discriminative domain stop-words ("net", "total", "per") from $Q$ first. Retained, "net" lets a query for Net Income score $R_{lex} = 0.50$ against a chunk discussing Net Sales and pass the gate; filtered, the same pair scores 0.0 and is correctly blocked [71]. KV-cache reuse. This is the highest-leverage cost lever and the most fragile. Understand the mechanics: prefill computes Key/Value tensors for every input token; prompt caching persists those tensors across requests, indexed by a hash of the token prefix, so a matching prefix skips prefill entirely [11]. The match is byte-exact and prefix-ordered. Anthropic renders requests as tools → system → messages, and any change at one level invalidates that level and everything after it [24]. The engineering discipline follows directly: Position Content Change frequency 1 (top) Tool definitions Near-never 2 System instructions, output schema, citation policy Low 3 Corpus manifest, source-family metadata, style guide Low 4 Retrieved evidence spans for this section Per section 5 (bottom) Section query, timestamps, session IDs Every call A single dynamic value placed upstream — a timestamp in the system prompt, a reordered JSON key in a tool schema, a session ID — invalidates the entire prefix behind it. One documented incident: editing one line of a tool description dragged the system prompt and all few-shot examples into a full cache rebuild, with cache_creation_input_tokens spiking every call while reads went to zero [82]. Track the read/write ratio as an SLO, not a curiosity; the break-even on the 5-minute tier is roughly one hit ($h^* = (w-1)/(1-r) \approx 0.28$, rounding to 1), so any repeated prefix pays back immediately — but only if it actually matches [85]. Minimum cacheable prefix lengths vary by model (512 to 4,096 tokens depending on generation); below the floor the request succeeds silently without caching and returns no error [11][24]. Verify by checking that both cache_creation_input_tokens and cache_read_input_tokens are non-zero at some point [24]. Use 1-hour TTL when your evidence registry serves batch report generation or human-in-the-loop review with gaps exceeding five minutes; 5-minute TTL for active interactive sessions [24][41]. Stage 2 — Compression and Structuring Here is where most teams destroy their citation integrity in the name of cost. The hard rule: never compress citeable evidence. Compression is lossy, and the losses are exactly the ones that matter — a compressor can remove a negation, a version number, an identifier, or a relation between entities [5]. In a financial or regulatory report, dropping "not" or "except" or "subject to" inverts a claim while leaving fluent prose behind. Some implementations keep originals locally and let the model retrieve them, which "changes delete-and-hope into hide-and-recover" — but that only helps when the model notices something is missing, and a confidently wrong answer does not trigger retrieval [5]. Segment the payload into budgets with distinct enforcement: Budget Contents Compressible Citeable Evidence Original spans, tables, definitions, footnotes, exceptions No Yes Background Section summaries, chronology, entity aliases Yes, with source pointers No Instruction Output schema, refusal behavior, citation syntax No, but cache aggressively No Verification Claim/evidence pairs, contradiction candidates No Internal only Where compression is safe — background narrative, boilerplate, duplicated headers — the LLMLingua family is the mature option. LongLLMLingua reports up to 4× fewer tokens with up to 21.4% performance gain on selected NaturalQuestions settings, 94.0% cost reduction on LooGLE, and 1.4×–2.6× end-to-end latency acceleration for ~10K-token prompts at 2×–6× compression [15]. Its tooling supports segment-level control via <llmlingua> tags with per-segment rate and an explicit no-compress flag — use that capability to fence off the evidence budget [43][14]. Three caveats before you enable it. Microsoft labels LLMLingua research-oriented rather than a turnkey integrity solution [17]. Compression is an additional inference stage and must save more latency than it adds [5]. And critically: compression mutates the prefix and can break provider prefix caching, which "can already reduce repeated-input cost without another stage" [5]. Sequence matters — establish caching first, then evaluate whether compression adds anything on top. Start with deterministic wins that carry zero semantic risk: duplicate removal, repeated-header stripping, boilerplate elimination, metadata normalization, template deduplication. Retrieval and reranking improvements are "cheaper and introduce less risk" than learned compression [5]. The measurement that matters is not compression ratio. It is how much context can be removed before the complete workflow becomes less reliable [5]. Stage 3 — Generation: Span-Grounded Ledgering The model composes; it does not remember. It receives typed evidence records with stable IDs and must bind every claim to those IDs. It may not invent a page number, because it never sees one it didn't receive. Structured intermediate output: { "claim_id": "clm_0142", "claim_text": "Gross margin declined 290 basis points year-over-year.", "claim_type": "computational", "evidence_bindings": [ { "evidence_id": "ev_01J8X2K4M7N3P5Q9R1S2T3U4V5", "role": "primary_support", "quoted_span": "Gross margin decreased from 45.2% to 42.3%", "confidence": "high" } ], "verification_status": "pending" } Two enforcement mechanisms, with a clear default. Constrained decoding enforces citation grammar at inference time via a finite-state automaton tracking whether the model is generating a claim, a document ID, or a snippet, restricting next-token selection to grammar-consistent tokens [45]. Retry up to 3 times with temperature increased by 0.5 on failure. This guarantees structural compliance and eliminates an entire class of formatting defects. Grammar-constrained decoding has matured considerably — recent work reports 17.71× faster offline preprocessing while preserving state-of-the-art online mask computation efficiency [58]. The cost is inference complexity. Posthoc span alignment is the recommended production default. Generate the claim and an approximate citation, then align the generated snippet to the closest verbatim span using word-level Jaccard similarity at threshold 0.7. This lifted Snippet-F1 from 12.80 to 61.87 on ASQA [45]. Because your registry already stores character offsets, alignment is a lookup rather than a search. The EFSG work at ACL 2026 reports the same directional finding from the opposite angle: sealing evidence into a fact pool before generation and showing each sentence only its committed passage yielded 0.612 sentence-support — high groundedness — while nugget coverage remained low at 0.126, isolating retrieval breadth rather than generation faithfulness as the bottleneck [59]. That asymmetry is the single most useful diagnostic in this entire document. Groundedness and coverage fail independently. Architecture fixes groundedness. Only retrieval breadth fixes coverage. Decompose compound claims before binding. "Revenue increased due to enterprise demand" is two propositions: revenue increased and enterprise demand was a stated driver. A citation supporting only the first cannot commit the combined sentence. Stage 4 — Decoupled Verification-in-the-Loop Verification must be external to generation. Asking the generator to self-check its own citations reintroduces the failure it was meant to catch — CiteGuard's design places the support decision with an external, gold-validated verifier rather than the generator [34]. Broader work on LLM-as-a-judge documents self-preference bias, prompt sensitivity, and self-inconsistency as reasons not to trust a model grading its own output [50][34]. Three tiers, cheapest first: Tier 1 — deterministic quotation matching. Character-level substring search of the quoted text against verbatim_text at the cited evidence_id, with whitespace normalization. Zero LLM calls, microsecond latency. This catches paraphrase drift, token substitution, and hallucinated quotations. It is the single highest-value control in the architecture because it makes silent drift a hard error, and it can be deployed against your current pipeline today. Tier 2 — hard anchor verification. Extract typed signals from the claim — monetary values, dates, percentages, identifiers, quoted strings — and verify exact presence in the cited span. This is Eywa's V_hard: verification of "deterministic values such as dates, monetary values, quoted strings, identifiers, URLs, IP addresses, and percentages" [60]. Also enforce polarity: negation markers in the source must be preserved in the claim [60]. Zero LLM calls. Tier 3 — entailment verification. An external model judges whether the cited span actually supports the claim, catching cases where the quotation is real but doesn't mean what the prose asserts. A distilled 8B detector achieves 91.4% F1 at 18× lower per-claim latency than the GPT-4o teacher it was distilled from, retaining 96.2% of teacher performance and enabling ~$0.003/query deployment [69]. Type-routed verification is where generic NLI is provably insufficient. FinGround's central finding: existing detectors treat all claims uniformly and miss 43% of computational errors requiring arithmetic re-verification against structured tables [69]. Computational claims carry the highest hallucination rate (28.4%) yet are the most amenable to automated verification once properly typed — formula reconstruction against a template library, operand retrieval from table cells, and recomputation with ±0.5% rounding tolerance yields 90.2% F1 on that class [69]. Ablating the six-type taxonomy roughly doubles hallucination rates, with the largest impact on the hardest benchmark (4.9% → 11.7%) [69]. Recovery, not deletion. CiteGuard's policy is verify → re-attribute → flag [34]. Most citation failures are misattribution, not fabrication: the claim is right and the pointer is wrong. Re-attribution proposes an alternative span from the sealed pool via BM25 rerank, then re-verifies before moving the pointer. Cap at 3 attempts, then flag [UNVERIFIED]. This matters because a lexical ranker cannot distinguish support from contradiction — on SciFact, an answer and a refuting passage share nearly identical vocabulary (Jaccard 0.064 vs. 0.055), while the verifier separates them cleanly (mean $P(\text{attributable})$ 0.83 vs. 0.50) [50]. The ranker proposes; the verifier disposes. 2.5 Provenance Non-Amplification The formal guarantee that makes the loop safe: repair operators must be typed and closed over tool-produced provenance. If every repair operation either creates no new entry, or creates one only by invoking a tool and applying a deterministic mapping to its output, then no repair can fabricate provenance-less content [61]. Free-form reflective repair has no such property — it can fix one unsupported claim while introducing another. Measured error-introduction rate on regenerated claims is roughly 4.1% per claim, compounding in multi-claim answers, which is why full re-generation is triggered above a threshold of claims requiring repair [69][60]. Attribution Integrity: ACID-Style Claims Treat each claim as a transaction with ACID-analogous properties. This is a design discipline, not a database guarantee — but the analogy is load-bearing because it tells you exactly which invariants to enforce. Property Meaning for a claim Enforcement Atomicity A claim commits with all its bindings or none. No half-cited sentences. Compound-claim decomposition; single-transaction commit Consistency Every committed claim satisfies all invariants: valid evidence ID, byte-exact quote, preserved polarity, allowed source version Tiers 1–2 as hard gates Isolation The claim is judged against sealed evidence, not against other claims or the model's parametric memory Evidence sealed before generation; generator sees only its committed span Durability The claim's provenance survives independently of the model, the prompt, and the session Immutable registry; append-only; version-hashed Concretely, the commit gate: Gate Deterministic check Failure caught Ledger existence Cited span_id exists in the requested source version Invented or stale citation IDs Quotation integrity Quotation is byte- or normalization-equivalent to registry span Fabricated quotations, OCR mutation, truncated qualifiers Anchor integrity Page, table, section, offsets resolve to canonical artifact Wrong page, wrong table row Polarity Negation and uncertainty markers preserved "does not" → "does" inversion Version Source version permitted by as_of_date policy; supersedes checked Citing a superseded restatement as current Coverage Every material factual claim has ≥1 binding Unsupported assertions Entailment Cited span supports, not merely resembles, the claim Silent citation drift Render integrity Final citation marker maps to approved span_id Post-processing citation corruption The critical engineering point, stated plainly: retrieval relevance is not entailment. A passage containing the same entity, metric, or legal term may still fail to support the generated proposition. Audit envelope. For downstream regulatory or client review, each committed claim should ship a citation chain with three independently verifiable signals: a content-addressable stable identifier, a cryptographic signature over the canonical serialization (HMAC-SHA256 is the current common practice), and a re-fetchable source reference so the verbatim evidence is recoverable even if the original document moves [49]. Three independent checks means a forgery must defeat all three; a failure in any one tells you something different — ID lookup failure means the catalog moved, signature mismatch means the envelope was modified, source 404 means the document went away and the stored excerpt is your fallback [49]. What provenance does not give you. Provenance establishes source support, not external truth. If a source contains a false, ambiguous, or superseded claim, the architecture preserves where it came from and whether the extracted belief is supported by that source — but world-level truth verification remains outside the memory layer [60]. A perfectly attributed citation to a paper with fabricated data will pass every gate here. Source curation is a separate, architectural concern: restrict retrieval to a trusted corpus, check publication recency, and triangulate across independent sources where the stakes justify it [20]. Architectural Tradeoff Matrix 4.1 Strategy comparison Strategy Latency Cost Attribution rigor Impl. complexity Failure mode if misapplied Verdict Prompt caching Up to 85% lower on long repeated prefixes; TTFT 11.5s → 2.4s on 100K cached [44] Cache reads ~0.1× base input; writes 1.25× (5-min) / 2× (1-hr) [24] Neutral — content byte-identical; does not fix attention or attribution Very low: prefix ordering, cache_control markers, hit-rate telemetry Any upstream byte change silently invalidates the whole prefix; timestamps in system prompt drive hit rate to zero [82][85] Enable immediately Hybrid retrieval + rerank Adds indexing and rerank time (~100–400ms for top-20 rerank); may lower generation latency via smaller prompt [13] Largest durable v