1. Why “read it start to finish” fails
Original analysis and writing. Every outside idea is paraphrased and credited in the sources at the end.


Most of us were taught to read a paper the way we read a novel: begin at the top, push through the middle, stop at the end. For a modern machine learning paper that is a poor strategy. The typical submission runs between eight and thirty pages once appendices are counted, packs its key claim into a single sentence, and buries the awkward details (failed runs, narrow test conditions, compute budgets) several pages away from the headline number. Reading linearly means you spend your best attention on background material and arrive at the evidence tired.
The best-known fix is older than the current AI boom. In a widely taught 2007 essay, the computer scientist S. Keshav proposed reading in up to three passes instead of one long slog. The first pass is a quick bird’s-eye scan that he says should take roughly five to ten minutes; the second is a careful read that skips things like proofs and can take up to an hour for an experienced reader; the third attempts to virtually re-create the work and can take hours. The point of the structure is that most papers should die after pass one, and that is a good thing. Your time is the scarce resource, and a paper that does not earn pass two has not earned your afternoon.
This guide adapts that idea to AI papers specifically, because they have quirks classical papers lack: benchmark numbers that move by fractions of a point, baselines that may have been tuned less carefully than the proposed method, and claims about “reasoning” or “understanding” that are really claims about performance on a particular dataset. A ten-minute pass is not about understanding everything. It is about forming an accurate, defensible opinion of what the paper claims and how much trust that claim deserves.
There is also a newer reason to build the skill yourself. Many readers now paste a paper into a chatbot and ask for a summary. That is a fine way to triage, but recent research shows it has a systematic weakness, which we will examine below. The short version: summaries tend to sound more confident and more general than the original study. Knowing how to check a paper directly is the safeguard.


2. The ten-minute routine, minute by minute
Minutes 0–2: title and abstract. Read the title, then the abstract twice. On the first read, underline nothing; just ask what category of paper this is. Is it a new method, a new benchmark or dataset, an empirical analysis of existing models, a theoretical result, or a survey? The category tells you what “good” looks like. A method paper must beat strong baselines fairly. A benchmark paper must show the benchmark is hard, valid and not already saturated. An analysis paper must show its finding is not an artifact of one model or one prompt. On the second read, write the claim in one sentence of your own words. If you cannot, that is already information: either the paper is unclear or you lack background.
Minutes 2–4: introduction and contributions. Most introductions end with a bulleted list of contributions. Treat that list as the author’s own scorecard, then look for each item later in the paper. If a contribution says “state-of-the-art results across benchmarks” but you find only two benchmarks in the tables, note the gap. If it says “we show that X emerges at scale,” check how many model sizes were actually tested. Look as well for the research gap the authors say they are filling. A weak gap (“nobody has combined these two ideas”) is less convincing than a concrete failure of existing methods.
Minutes 4–7: figures, tables and headings. Skip the method section on the first pass. Instead, read every section heading to understand the structure, then study the main results table and the one or two most important figures. Captions in good papers are nearly self-contained, so read them fully. We will return to what to look for in the next part.
Minutes 7–9: limitations and conclusion. Jump to the limitations section, the conclusion, and any “broader impact” or “discussion” text. Honest authors state where the method fails, which settings were not tested, and what they could not afford to run. Then skim the related work and the reference list, ticking off papers you know. A paper that ignores the obvious competitor is a yellow flag; a paper that cites mostly its own group’s earlier work deserves a closer look at who else has replicated it.
Minute 9–10: the verdict. Write three lines: what the paper claims, what evidence supports it, and what you would need to see before believing it. Then choose: stop, bookmark for a second pass, or commit to a deeper read. This tiny act of writing is the habit that separates reading from merely scrolling.

3. The five questions your first pass must answer
Keshav’s five Cs remain the best checklist for the end of pass one. They are category, context, correctness, contributions and clarity. In an AI setting I translate them into practical questions.
Category: what kind of paper is this, and therefore what standard applies? Context: which prior work does it build on, and is the comparison set current? In a field where strong open models arrive monthly, a paper that benchmarks against systems that were state of the art two years ago is describing a world that no longer exists. Correctness: do the assumptions look reasonable? Ask whether the test data could have leaked into training, whether results come from a single run or several seeds, and whether the evaluation measures what the title says it measures. Contributions: are they real and specific, or restatements of “we trained a bigger model”? Clarity: could you explain the idea to a colleague? Murky writing sometimes hides murky science.


4. Reading figures, tables and claims in AI papers
The results table is where a ten-minute reader earns their keep. Start with the bolded row and work outwards. How large is the margin over the strongest baseline, and is it larger than the run-to-run noise? Papers that report a single number with no variance make it impossible to tell a real gain from luck, and in machine learning random seeds alone can swing results by amounts comparable to the headline improvement. Look for error bars, confidence intervals or at least a statement about the number of runs.
Next, inspect the baselines. Were they tuned with the same care and compute as the proposed method? Were they given the same data? A frequent trick, usually unintentional, is to compare a heavily engineered new system with default-setting competitors. The ablation table, where components are removed one at a time, is often more honest than the main table, because it shows which parts of the proposed method carry the weight. If the ablation shows one tweak accounts for nearly all of the gain, the rest of the architecture may be decoration.
Third, read the axes. Many figures in the field use truncated y-axes, log-scale compute plots, or cherry-picked examples. A qualitative figure of beautiful model outputs tells you what the model can do at its best, not how often. Ask what the failure cases look like and whether the authors showed any. Fourth, check the scope of the claim against the scope of the experiment. This is the single most useful habit, and it connects directly to the research on AI summaries below. If the experiments covered English-language text, three model sizes and one benchmark family, the honest claim is narrow, even if the abstract speaks in general terms.
Finally, note the cost. How much compute, data and money did the result need? A technique that only works with a million-dollar training run may be an important scientific data point, but it is not the same kind of finding as one a graduate student can reproduce in a weekend. Reproducibility information such as released code, weights, data and exact hyperparameters is worth a few seconds of checking, because it determines whether anyone can test the claim independently.


5. What recent research says about AI summaries
Using a chatbot to summarize a paper is tempting and often useful, but a 2025 study in Royal Society Open Science by Uwe Peters and Benjamin Chin-Yee gives good reason for caution. The researchers tested ten prominent language models, including several versions of ChatGPT, DeepSeek, LLaMA and Claude, and compared roughly 4,900 model-written summaries against the source texts. According to the paper, the summaries were nearly five times as likely as human-written ones to contain broad generalizations, with an odds ratio of 4.85. In plain terms, when a study said something held for a specific group or setting, the summary often said it held generally.
Two further findings matter for how you work. First, the newer models in the sample tended to generalize more, not less, than older ones. Second, prompts that explicitly asked for accuracy did not fix the problem and in some cases made it worse, according to press coverage of the study. The authors suggested mitigation steps such as lowering the temperature setting and benchmarking models specifically for generalization accuracy, and they reported that the Claude models produced the most faithful summaries in their tests. Treat that last point with the usual care: it is one study, with specific model versions, on a specific task, and the authors themselves note that testing should be extended to other scientific tasks and to non-English text.
The practical lesson is not “never use AI.” It is that an AI summary tends to delete the qualifiers (“in this cohort,” “on this benchmark,” “under these assumptions”) that tell you how far a result travels. Those qualifiers are exactly what a ten-minute human pass is built to recover. There is also a learning cost. A 2025 study in PNAS Nexus, as reported by a secondary source I could not verify at the original, found that people who learned a topic from AI summaries came away with shallower knowledge than people who consulted the sources directly. I would treat that as suggestive rather than settled, but it fits common experience: summaries hide what you did not know to ask.


6. Putting paper tools to work, without outsourcing your judgment
Used well, research tools make the ten-minute routine faster. If you have connected paper tools to your assistant, such as alphaXiv, SciSpace or Lacuna Research, each can take over a different chore. A tool that pulls a paper’s full text lets you ask targeted questions (“what datasets were used in Table 2?”) instead of requesting a vague summary. A literature-search tool helps you find the three to five most recent papers on a topic, which is also Keshav’s own advice for starting a survey: do one quick pass on each, then read their related-work sections to find the shared key references. Data-extraction features can pull the same fields (model size, dataset, metric, compute) from many papers into one table, which makes comparisons across a literature quick. Author and research-direction tools help you see who works on what and which ideas cluster together.
The rule I would apply is simple: let tools fetch and organize, but make the judgments yourself. Ask them narrow, checkable questions that quote the paper’s own numbers, and verify anything that matters in the actual figures and tables. Keep notes and a personal library if the tool supports it, because the second and third passes get much faster when your first-pass verdicts are saved. And ask for the limitations explicitly, since that is where summaries are weakest.
A good prompt for a connected tool looks like this: “List every claim in the abstract. For each, name the table or figure that supports it, the exact setting tested, and any limitation the authors state.” This forces the output back toward scope and evidence, which is where overgeneralization creeps in. Then open the paper and spot-check two of the answers against the source. If both match, you can trust the tool’s output a little more for that paper; if either drifts, read the section yourself.
7. A reusable template and when to go deeper
Copy this into your notes app and fill it in for every paper: Claim (one sentence). Category (method, benchmark, analysis, theory, survey). Evidence (which table or figure, how many runs, how large the margin). Scope (models, data, languages, scale tested). Weak points (baselines, leakage risk, missing ablations, cost). Verdict (stop, bookmark, deep read). Three lines of real content beat three pages of highlighting.
Go to a second pass when the paper is central to your work, when the claim surprises you, or when many people are citing it. In the second pass, read the method section and the experimental setup with care, but skip long proofs and appendix minutiae unless they affect the claim. Make notes in the margin, list the references you want to follow up, and try to explain the main idea to someone else. Reserve the third pass, in which you mentally rebuild the work and challenge each assumption, for papers you will review, replicate or build on directly. For most papers, you will never need it.
The deeper habit this routine builds is calibrated trust. You will not become a perfect judge of research in ten minutes, but you will stop confusing a confident abstract with a well-supported result, and you will notice quickly when a summary, human or machine, claims more than the evidence allows. In a field producing hundreds of papers a week, that ability to triage honestly, and to say “interesting, but narrower than advertised,” is worth more than speed alone.
