AI Research & Fact-Checking System

Ask a language model whether a claim is true and it will answer confidently either way. That is the actual failure: not that models are sometimes wrong, but that a wrong answer is indistinguishable in tone from a right one.

For anything that touches a decision — a market claim in a report, a compliance statement, a technical assertion in documentation — an assistant that cannot say "the evidence disagrees" or "I could not verify this" is worse than no assistant, because it launders a guess into something that reads like a finding.

A verification pipeline rather than a chat wrapper. A claim enters, gets decomposed into independently checkable sub-claims, and each one is researched separately against multiple retrieval sources. Evidence is scored for support, contradiction or irrelevance, and agreement between independent sources — not model confidence — determines the verdict.

The output is a structured verdict with per-claim citations, an explicit uncertainty band, and a list of what could not be checked. "Insufficient evidence" is a first-class result the system is designed to return, not a failure mode.

Why agreement beats confidence#

The core design decision is that the model never issues the verdict. It classifies evidence; arithmetic issues the verdict.

That sounds like a small distinction and it changes everything. A model asked "is this true?" produces a fluent answer shaped by its priors. A model asked "does this passage support, contradict, or fail to address this specific sentence?" is doing a much narrower job that it is genuinely good at. Aggregating those narrow judgements across independent sources gives a verdict you can audit line by line.

verdict.pypython
def aggregate(evidence: list[Evidence]) -> Verdict:
    support = [e for e in evidence if e.label == "SUPPORTS"]
    contradict = [e for e in evidence if e.label == "CONTRADICTS"]

    sources_supporting = {e.source_domain for e in support}
    sources_contradicting = {e.source_domain for e in contradict}

    if len(sources_supporting) >= 2 and not sources_contradicting:
        return Verdict.SUPPORTED
    if len(sources_contradicting) >= 2 and not sources_supporting:
        return Verdict.CONTRADICTED
    if sources_supporting and sources_contradicting:
        return Verdict.DISPUTED
    return Verdict.INSUFFICIENT_EVIDENCE

Counting distinct domains rather than passages is deliberate. Five paragraphs from one content farm is one source, and treating it as five was the first bug worth fixing.

Cost control as a design constraint#

Naïve agent loops are unbounded by construction: each retry costs money and there is nothing stopping the loop from taking twelve of them. Three constraints keep this predictable:

Tiering alone cut cost per verified claim by roughly two thirds with no measurable accuracy change — which is the sort of result that only shows up when you have an evaluation set to measure against.

What the architecture looks like#

Retrieval, scoring and aggregation are separate services behind a queue rather than one long function. That is not architectural ceremony: verification runs take tens of seconds, and a synchronous request that occasionally takes a minute is a support ticket waiting to happen. Persisting each state transition also means a failed run resumes from its last good node instead of restarting and re-spending the budget.