Most people read research papers wrong. They start at the abstract, read every word in order, reach the results, feel lost, and quit. A 2022 Journal of the Association for Information Science and Technology study found that a majority of researchers themselves admit to skimming citations without verifying them — and citation chains full of unverified links are exactly how errors compound into whole literatures.
This article codifies a different way: a nine-pass protocol where each pass has one job, one trigger for moving on, and one artifact it produces. It is the reading workflow used to produce this site’s deep dives, and it pairs with the AI agents concept glossary — the vocabulary, with this as the method. Everything below works with a human reader, an AI assistant, or both.
How should you read a research paper? Not linearly. Run nine passes with distinct jobs: (1) an intake protocol that captures provenance before you invest attention; (2) an attention residual check on your own reading — note where your understanding skipped; (3) the contradiction finder; (4) the citation chain walk; (5) the gap scanner; (6) the methodology audit; (7) the assumption killer; (8) the master synthesis; (9) the so-what test. With AI assistance, the same passes become prompts — and the whole protocol fits into an afternoon instead of a week.
The Protocol at a Glance
Each pass answers one question and stops. The discipline is refusing to do pass 7’s job during pass 2.
| Pass | Question it answers | Artifact you keep |
|---|---|---|
| 1. Intake protocol | What is this paper, and is it worth my next eight passes? | Provenance card: venue, authors, date, code link |
| 2. Attention residual check | What did I silently skip or assume while reading? | List of unexamined claims carried into later passes |
| 3. Contradiction finder | What does this paper claim that conflicts with what I know or with itself? | Contradiction ledger with severities |
| 4. Citation chain | Do the foundational cites actually support the claims hung on them? | Verified / misapplied / broken buckets per citation |
| 5. Gap scanner | What did the authors NOT test, measure, or mention? | Gap list: measurement, dataset, baseline, theory |
| 6. Methodology audit | Would this design survive its own scrutiny? | Audit table: design, statistics, reproducibility |
| 7. Assumption killer | Which implicit premises, if false, collapse the result? | Load-bearing assumptions ranked by fragility |
| 8. Master synthesis | What does this paper change about the whole field map? | Updated knowledge map + your positioning |
| 9. So-what test | Who should act differently because this exists? | Decision memo: action, owner, urgency |
Pass 1: The Intake Protocol
The intake protocol is triage before investment: identity, provenance, and a go/no-go decision, captured before any deep reading. It prevents the classic failure — spending four passes on a paper that was never worth one.
How to run it: pull title, authors, venue, date, and version (arXiv v1/v2 differences matter); check for code and data release; scan the abstract for the delta claim (“we show X” — what is new); note conflicts of interest and funding. Thirty seconds on the provenance card answers whether the next eight passes are warranted. With AI: paste the PDF and ask for intake only — “return venue, authors, date, version, code link, the one-sentence delta claim, and any funding disclosure” — and nothing else.
Example: a paper claiming a 40% latency reduction clears intake only if its v-number and benchmark availability are disclosed; a v1 arXiv preprint with no code, no data, and a corporate lab affiliation gets logged and parked, not read.
Pass 2: The Attention Residual Check
Your reading has a residual stream too: claims flow through your comprehension with most edits unnoticed, and what you skip silently becomes load-bearing later. This pass makes the skipped material explicit by instrumenting your own attention.
How to run it: read once at normal speed, without stopping. Immediately after, list every claim you cannot restate in your own words — that list is your attention residual: the places your understanding glided. With AI: ask the model to black out one random paragraph at a time and quiz your summary of the whole; discrepancies localize the skips. The output feeds passes 3 and 7 — contradictions and assumptions hide preferentially in skipped text.
Example: after reading a scaling-law paper, you restate everything cleanly except the data-mixing section — which turns out to be where the entire result’s fragility lives. Your attention residual was pointing at it all along.
Pass 3: The Contradiction Finder
A contradiction finder hunts internal and external conflicts: places the paper contradicts itself (claims vs. results, abstract vs. tables) and places it contradicts known results or your own verified knowledge.
How to run it: extract every claim as a numbered sentence; then cross-examine: claim vs. claim, claim vs. table, abstract vs. conclusion. External: check the paper against the two or three papers it must not conflict with (its own prior version, the baseline it beats, the survey it cites). With AI: “list every pair of statements in this paper that are in tension, with locations” — then verify each hit yourself, because models over-flag and under-flag in predictable patterns (they over-flag hedging, under-flag units).
Example: an abstract claims “significant improvement across all tasks” while Table 3 shows a regression on two of nine tasks with a footnote dismissing them — the contradiction ledger’s first entry writes itself.
Pass 4: The Citation Chain
The citation chain is the paper’s load-bearing structure made explicit: which claims rest on which prior work, and whether those prior works actually said what is attributed to them.
How to run it: build the chain claim-by-claim: for each foundational claim, follow the citation to the primary source and read its abstract with the claim in mind — bucket each link verified (supports), misapplied (exists but says something else), or broken (untraceable or retracted). Prioritize: methods statistics, benchmark numbers, and any “it is known that” sentence. With AI: one prompt per citation — “does this cited abstract support this specific claim?” — then spot-check the model’s verdicts against the sources; automated DOI resolution catches the broken bucket instantly.
Example: a paper hangs “attention flows are interpretable” on a 2017 citation; pulling the 2017 abstract shows it demonstrated interpretability for one narrow probe, not the general claim — a misapplied link, quietly carrying weight it cannot bear.
Pass 5: The Gap Scanner
The gap scanner reads for absence: what was not measured, not compared, not stated, not tested. Gaps are where the paper’s real limitations live — and where your opportunity might.
How to run it: interrogate the four standard gap classes: measurement (which metrics are missing), dataset (which populations/domains absent), baseline (which obvious rivals untested), theory (which mechanisms unexplained). Rule: a limitation admitted by the authors is a constraint; one found by you is a gap — collect both, separately. With AI: “list what this paper does not measure, does not compare against, and does not explain, in three separate lists” — then diff against the paper’s own limitations section; your list minus theirs is the true gap set.
Example: a quantization paper benchmarks quality and memory but omits latency-under-load — the gap scanner flags it, and the missing measurement turns out to be the deciding factor for production adoption.
Pass 6: The Methodology Audit
The methodology audit subjects the paper’s experimental design to the scrutiny it applies to others: datasets, baselines, statistical treatment, and reproducibility.
How to run it: work the audit table — (a) design: is there a control; are ablations real ablations; (b) statistics: how many seeds, is the error bar a standard deviation or standard error, are significance claims backed by tests; (c) reproducibility: is code/data out, are hyperparameters complete, does the reported number recompute. With AI: “audit this methods section: list every place the evaluation could be unfair to the baseline” — the classic catches are cherry-picked baselines, tuned-ours-untuned-theirs, and seed selection. Keep the audit table as a reusable artifact for when the next version of this paper appears.
Example: the error bars turn out to be standard errors across three seeds — the “clear win” overlaps the baseline once you recompute; the audit table records it, and the paper’s headline drops from breakthrough to incremental.
Pass 7: The Assumption Killer
The assumption killer hunts implicit premises: statements the paper needs to be true but never argues. If one is false, the result collapses regardless of execution quality.
How to run it: convert every method-and-setup sentence into “assume X” form; then attack each: is X universally true, or true only in this paper’s regime? Rank by fragility (how easily false) times leverage (how much of the result rests on it). The audit from pass 6 feeds this directly — unrandomized splits and unreported seeds are assumptions wearing lab coats. With AI: “list every assumption this paper needs but does not state, ranked by how much of the result depends on it” — then manually prune; models list twenty, the load-bearing ones are usually two.
Example: the paper assumes static documents; your target corpus mutates hourly. Nothing in the evaluation fails — but the assumption killer marks the whole transfer invalid, saving you a quarter of engineering time.
Pass 8: The Master Synthesis / Knowledge Map
Master synthesis integrates the paper into your existing knowledge map: the living graph of papers, claims, tools, and open problems you maintain. A paper not mapped is a paper half-read.
How to run it: place the paper’s contribution on your map as nodes and edges: which claims it establishes (nodes), which prior results it builds on, contradicts, or supersedes (edges), and which open problems it opens or closes. Then re-derive override edges: what you believed last month that this paper changes. With AI: hand the model your map fragment for this subfield plus the paper summary; ask it to propose the updated subgraph — additions green, contradictions red, supersessions struck through — and review like a PR.
Example: the map shows three papers claiming mutually incompatible effective approaches spanning 2023-2026; the synthesis pass surfaces the hidden variable (dataset drift between studies) that reconciles all three — a contribution none of the individual papers made.
Pass 9: The So-What Test
The so-what test is the final gate: who should act differently because this paper exists, and what exactly should they do? A paper that changes nobody’s actions can still be good science — but you should know that before you cite it as if it mattered.
How to run it: write the decision memo: one action, one owner, one urgency, per audience. For builders: does this change architecture, tooling, budgets, or nothing? For researchers: which open problem advances? For your own roadmap: adopt, watch, or ignore. If the memo comes back empty after honest effort, file the paper as reference material — that is a legitimate outcome, not a failure of the protocol.
Example: a meticulous attention-approximation study passes eight passes beautifully and fails the ninth: nobody’s decision changes — adopt-nothing, watch-nothing, ignore. Logged, mapped, closed.
Running the Protocol With AI: One Prompt Pack
Each pass compresses to a repeatable prompt. Together they form a complete audit; separately they are still useful one-off tools.
| Pass | The prompt |
|---|---|
| Intake | “Venue, authors, date, version, code link, funding. One-sentence delta claim. Nothing else.” |
| Attention residual | “Black out each paragraph in turn and quiz my summary; report every discrepancy.” |
| Contradiction finder | “List every pair of statements in tension, with locations.” |
| Citation chain | “For citation [n]: does the cited work support this exact claim? verified / misapplied / broken.” |
| Gap scanner | “What is not measured, not compared, not explained? Three lists.” |
| Methodology audit | “Where could this evaluation be unfair to the baseline? List each with fix.” |
| Assumption killer | “List unstated assumptions ranked by how much of the result depends on each.” |
| Master synthesis | “Here is my subfield map; return the updated subgraph with adds/contradictions/supersessions.” |
| So-what | “Write the decision memo: action, owner, urgency, per audience.” |
FAQ
Does the order of the passes matter?
Yes. Intake must come first (attention is the scarce resource) and the so-what test must come last (it consumes the artifacts of all previous passes). The middle passes have some slack: the contradiction finder and gap scanner can swap, since both feed the assumption killer; but doing synthesis before the citation chain produces maps built on misapplied citations — the most common AI-assisted mistake.
How long does the full protocol take per paper?
Twenty minutes to two hours unassisted depending on density, but distribution matters more than the total: intake is minutes, synthesis and citation-chain walks dominate. With AI on the mechanical passes (chain verification, gap listing, assumption enumeration), a defensible full run compresses to roughly 30-45 minutes — the human keeps contradiction judgment, assumption pruning, and the so-what verdict, which are the passes where model errors are costliest.
Can large context models just read the whole paper?
Context capacity is not comprehension. Long-context models still exhibit skipped-reasoning failures and lost-in-the-middle effects — the exact phenomena passes 2 and 4 exist to catch. A model can hold the entire paper and still misattribute a citation; the protocol exists because capacity does not certify verification.
Why keep artifacts for every pass?
Because artifacts compound. The contradiction ledger from this paper seeds pass 3 of the next; the knowledge map accumulates across years; the audit table template improves with each use. The protocol’s value is not any single run but the residue it leaves — the same way a good lab notebook outlives any one experiment.
