Fact-Check: Can an AI Scientist Auto-Publish Papers? Claims vs Evidence (2026)

Last Updated: October 2026

The clip goes like this: an AI scientist proposes an idea, writes the code, runs the experiments, drafts a manuscript, and a conference accepts it, all without a human touching anything. It is a compelling story, and fragments of it are true. A fully AI-generated paper really did clear a peer-review threshold, and the work behind it really did appear in Nature in 2026.

The trouble is everything the clip leaves out: the venue was a workshop, humans chose which papers to submit, the paper was withdrawn on purpose, and independent testers found frequent coding failures, stale citations and invented numbers. “Auto-publish” compresses a narrow, carefully staged experiment into a claim about science itself.

You will leave with a claim-by-claim verdict table, a primary-source account of what Sakana, Intology, OpenAI’s PaperBench and the Agents4Science conference actually measured, and an audit checklist for judging the next headline yourself.

What this covers: the verdict, how autonomous research pipelines work, the ICLR workshop experiment in detail, independent evaluations, the other systems in the field, why simulated review is not peer review, and what to trust.

The Verdict at a Glance

Short answer: an AI scientist can generate a paper that passes a lenient peer-review screen, and it cannot yet publish science on its own authority. No system has shown a fully AI-generated paper clearing the bar for a main-track top-tier venue under independent verification. The honest rating is “partly true, heavily scoped.”

The table rates each viral claim against the primary evidence reviewed for this article. Evidence ratings are ours: Supported means a primary source and independent scrutiny agree; Partly true means the fact holds but the framing overreaches; Unverified means the only source is the developer; False means the evidence contradicts it.

Viral claim	Rating	What the evidence shows
“An AI wrote a paper that passed peer review”	Partly true	One of three AI papers scored above a workshop acceptance threshold (reviewer scores 6, 7, 6) at an ICLR 2025 workshop. Workshop, not main track.
“No humans were involved”	Misleading	Humans supplied the topic, selected 3 papers for submission, ran internal reviews and code checks, and handled the ethics process.
“It was published”	False	The paper was withdrawn by design and never posted. Nothing was published.
“It can replace the research process end to end”	Unsupported	The Nature paper itself says outputs are inconsistent and the system works only in computational domains.
“It produces novel discoveries”	Unverified	An independent evaluation found novelty checks misclassifying known ideas as new. No independent confirmation of novel findings.
“AI reviewers can stand in for human reviewers”	Partly true	An automated reviewer reached 69% balanced accuracy on ICLR decisions, versus 66% human-to-human agreement in one cited study. Accuracy is not understanding.
“An AI paper was accepted to ACL main”	Unverified	Reported by Intology about its Zochi system; the claim comes from the developer.
“Papers cost $15 and an afternoon”	Partly true	The v1 paper claimed under $15 per paper. An independent study measured $6 to $15 plus about 3.5 hours of human involvement.

Treat the table as the thesis. The rest of the post shows the evidence behind each row, and flags the numbers we could not verify.

Context and Background

The idea of automated science is older than large language models. Systems such as Adam and Eve in the 2000s closed the loop between hypotheses and lab robots for narrow biological questions. What changed after 2023 is that language models can write code and prose, which makes the whole paper automatable for computational fields where an “experiment” is a script.

The reference point is Sakana AI’s The AI Scientist, introduced in August 2024 by Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune and David Ha (arXiv:2408.06292). Its abstract claims that each idea is developed into a full paper for under $15. That framing, a cheap paper factory, is what the virality is built on.

Since then the field has split into three kinds of claim, and mixing them up is the root of most bad fact-checks:

Capability claims. Can an agent run a research pipeline and produce a coherent manuscript? Largely yes, for machine learning.
Quality claims. Is the output correct, novel and useful? Mixed, with weak independent evidence.
Governance claims. Should AI papers be submitted, reviewed and published? This is a policy question that the technology cannot answer by itself.

A useful companion on measuring agent behaviour is our guide to AI agent evaluation harnesses and trajectory evals. The same discipline, test the process and not just the final artefact, applies here. For a similar exercise on another sweeping AI claim, see our fact-check of whether AI replaced software engineers.

A note on corrections. The earlier version of this article, published in April 2026, contained claims we can no longer support: an “AGoP” framework, a “CMU AI Scientist v2”, a DeepMind origin for Agent Laboratory, and a list of precise success rates without citations. We could not find primary sources for those and have removed them. The rewrite below uses only what we could trace to a paper, a developer post or a conference report.

How an Autonomous AI Scientist Actually Works

An autonomous research pipeline is a loop of language-model calls wrapped around a code sandbox. It brainstorms an idea, checks the literature, writes and debugs experiment code, summarises results into figures, drafts a LaTeX paper, and scores the draft with an automated reviewer. Humans choose the starting topic and, in every documented case, the papers that leave the system.

Figure 1: The AI scientist loop. Human input bookends the automated stages: a topic goes in, and a person chooses what goes out.

The diagram is deliberately explicit about the two human steps. In the first public version, the system started from a human-written code template for each research area, then iterated ideas on top of it. In the second version, it dropped that dependency and used a search procedure over experiments. In both, the paper reaches a venue only because a person selected it.

Version 1: templates and a linear pipeline

The 2024 system leaned on a template: a small working codebase for a topic such as diffusion modelling or language modelling. The model proposed modifications, ran experiments through a coding assistant, plotted results and wrote the paper section by section. An LLM reviewer then scored the manuscript, loosely modelled on conference review forms.

Templates were both the strength and the limit. They gave the agent a working baseline to modify, which is why the experiments mostly ran. They also meant the “research” was usually a variation on code a human had already written, within a narrow area.

Version 2: agentic tree search

The follow-up, AI Scientist-v2 (arXiv:2504.08066), removed the template requirement and generalised across machine-learning domains. According to the authors, it uses a progressive agentic tree search guided by an experiment-manager agent, and a vision-language model reviews figures to improve their content and appearance. The code is open source.

Tree search matters because experiments fail. Instead of a single run that either works or crashes, the system keeps a tree of candidate implementations, expands promising nodes, and debugs failing ones. This is the same idea behind many coding agents: spend more compute exploring, keep what works.

The automated reviewer

Both versions rely on an LLM reviewer as the quality filter. In the 2026 Nature write-up, as summarised by the Vector Institute, the reviewer combines five separate LLM judgments into a meta-review and a consensus, similar to an area chair. Tested on real ICLR submission data, it predicted accept or reject with 69% balanced accuracy, compared with 66% balanced agreement between human expert reviewers in a comparable study.

That figure is real and notable, and it is also easily over-read. Predicting whether a human committee accepted a paper is not the same as finding its errors. We return to this in the section on simulated review.

What “autonomous” does and does not mean

“Autonomous” in these papers means no human edits the manuscript text during generation. It does not mean no human decides what to run, what to submit, whether the code was sound, or whether the result is worth a reader’s time. The distinction between autonomous generation and autonomous science is the single most important fact in this fact-check.

The ICLR Workshop Experiment: What Happened, Step by Step

The headline event is a controlled experiment, not a surprise acceptance. Sakana, with researchers at the University of British Columbia and the University of Oxford, submitted three fully AI-generated papers to a workshop at ICLR 2025. They did so with the cooperation of ICLR leadership and the workshop organisers, and with institutional review board approval from the University of British Columbia, according to Sakana’s own account.

Figure 2: The ICLR 2025 workshop experiment as documented by Sakana. Every arrow that starts or ends with “Humans” is a place where the claim “no human involved” breaks down.

The sequence in Figure 2 is documented in the developer post and corroborated in reporting by TechCrunch. The details below are what survive when you read the primary material instead of the clip.

The numbers

The workshop received 43 papers in total, and reviewers were told that three of them might be AI-generated. The accepted AI paper received ratings of 6 (“marginally above acceptance threshold”), 7 (“good paper, accept”) and 6, an average of 6.33. Sakana states this ranked roughly in the top 45% of submissions, which is a long way from a standout.

Only one of the three papers was accepted; the other two fell below the threshold. The accepted paper, titled “Compositional Regularization: Unexpected Obstacles in Enhancing Neural Network Generalization,” reports a negative result: an idea that did not work as hoped. We will come back to why that matters.

Where the humans were

Sakana’s own description is candid. Humans gave the system a broad topic, and the system produced papers. Humans then selected three to submit. The AI handled hypotheses, experiments, code, data analysis, figures and the manuscript with no human edits to the text. Humans also ran their own internal reviews and code checks, and asked the AI to repeat some experiments.

Matthew Guzdial, quoted by TechCrunch, called the framing “a bit misleading” because the selection step means the result reflects humans plus AI more than AI alone. That is a fair reading. Selection from a larger pool is a form of human filtering, and filtering is where much of research judgment lives.

Why it was withdrawn

The protocol required that any accepted paper be withdrawn before publication. The papers were not posted on OpenReview, and the accepted one was desk-rejected after review. The authors have said they withdrew all submissions to avoid setting norms before the community is ready.

Because the organisers already knew about the experiment, no meta-review was performed. In a normal process a meta-reviewer or area chair can overrule individual reviewers, so the acceptance outcome is incomplete evidence. The word “published” in the viral claim is simply wrong.

The workshop caveat

Workshop acceptance rates run higher than main-track rates, in the 60 to 70% range according to Sakana’s own caveat, against roughly 20 to 30% for the main ICLR, ICML and NeurIPS tracks. Sakana’s authors reviewed their three papers against main-track standards and concluded that none met the bar.

Mike Cook, quoted by TechCrunch, also noted that newer workshops are often reviewed by more junior researchers, and that this workshop focused on negative results, which AI may find easier to write convincingly. That is a testable hypothesis rather than a settled fact, but it is a reminder that venue choice shaped the outcome.

Errors the authors disclosed

The accepted manuscript contained citation errors. A representative one: the system attributed an LSTM network to Goodfellow in 2016 rather than to Hochreiter and Schmidhuber in 1997. These are exactly the errors a careful reviewer would flag, and also the errors a lenient workshop review is likely to miss.

The takeaway is not that the experiment was dishonest. By most accounts it was unusually transparent. The takeaway is that the experiment answers a narrow question, “can an AI paper clear a lenient screen at a workshop?”, and the viral version answers a different one.

What Independent Evaluation Found

The developer’s results are one data point. The more valuable data come from people who had no stake in the system looking good. The most detailed independent study is “Evaluating Sakana’s AI Scientist: Bold Claims, Mixed Results, and a Promising Future?” by Joeran Beel, Min-Yen Kan and Moritz Baumgart (arXiv:2502.14297), revised as v3 in October 2025.

Execution reliability

The headline number: 42% of experiments failed because of coding errors, and other runs produced flawed or misleading results. That is a ceiling on any claim of autonomy. If four in ten experiments crash, then someone has to notice, and someone has to decide whether surviving runs are valid.

The authors also observed that code changes grew by only about 8% more characters per iteration on average. They read this as limited adaptability: the system tends to patch rather than redesign.

Novelty and literature

The evaluation found that literature reviews often misclassified established ideas as novel, with micro-batching for stochastic gradient descent as the cited example. This matters because novelty is the entire value proposition of research. A system that cannot reliably tell the new from the known will produce papers that look original and are not.

Citations were thin and old. Generated manuscripts had a median of five citations, and only 5 of 34 were from 2020 or later. A real paper in a fast-moving field would cite far more recent work.

Presentation defects and invented numbers

The study reports missing figures, repeated sections and placeholder text such as “Conclusions Here.” More seriously, some papers contained hallucinated numerical results: figures in the text that the experiments did not produce. In science, a fabricated number is the worst kind of error because it is invisible without checking the logs.

Cost and time

The authors measured a cost of USD 6 to 15 per paper and about 3.5 hours of human involvement. So the “under $15” claim holds up, with an important qualifier: it is a compute and API cost, and it excludes the human time to verify the output. The study’s overall verdict is that manuscripts can look human-written at a glance, while quality resembles a rushed undergraduate paper.

How to read these numbers

Two cautions. First, the evaluation tested the original template-based system on an earlier generation of models, so it should not be assumed to describe v2 or the 2026 Nature system. Second, a 42% failure rate is a failure rate for experiments, and a pipeline with retries can still deliver a finished paper. The right conclusion is “unreliable at the experiment level and unverified at the claim level,” not “never works.”

Figure 3: The rating method used in this post. A claim is only marked supported when evidence beyond the developer’s own report agrees.

Figure 3 shows how we applied the same standard to every row. “Self-reported only” is its own category because developer claims are often accurate and still deserve a different weight than independent replication.

The Nature Paper and the Scaling Claim

In March 2026, Nature published “Towards end-to-end automation of AI research” by Lu and colleagues (DOI 10.1038/s41586-026-10265-5), with authors from the University of British Columbia, Sakana AI, the Vector Institute and the University of Oxford. A UBC news post describes it as a system that runs a project end to end: generating ideas, checking the literature for novelty, writing and debugging code, analysing data, making graphs, drafting the manuscript and reviewing its own work.

We could not retrieve the Nature article text directly while writing this, so the specifics below come from the UBC news post and the Vector Institute summary. Treat figures from the summary as second-hand until checked against the paper.

The scaling claim

The authors report that papers from newer, more capable foundation models scored higher under the automated reviewer, and that giving the experiment search more compute also produced better papers. They frame this as a scaling trend for scientific quality, and describe the current system as a baseline.

This is a legitimate and interesting result, with a built-in circularity. The measure of quality is the same family of model that generated the paper. A better model might write papers that a model-based reviewer likes more without the papers being more correct. The 69% balanced accuracy against ICLR decisions offers partial reassurance, but it validates the reviewer as a predictor of committee behaviour, not as a detector of truth.

What the paper itself concedes

The limitations are explicit in the source material. Output quality is inconsistent. Common failures include weak ideas, incorrect implementations, experimental errors and hallucinated citations. The system only works in computational domains for now. No AI-generated paper has yet met the bar for main-conference or top-tier journal acceptance, according to the Vector Institute’s summary of the work.

The authors also flag risks that deserve more attention than the acceptance headline: overwhelming strained peer-review systems and adding hard-to-filter noise to the literature. Those are the real near-term harms, and they come from the system working well enough, not from it failing.

So is the claim “AI does science” supported?

For narrow, computational, benchmark-style machine-learning research, an AI scientist can run the full loop and produce a readable paper at a few dollars of compute. That is a genuine engineering achievement. It is not the same as an AI doing science in the sense of changing what a field knows, and the authors do not claim otherwise.

The Wider Field: Zochi, Agent Laboratory, PaperBench and Agents4Science

Sakana is not the only name in the clip. Four other data points shape the picture, and they differ sharply in how much independent verification stands behind them.

Intology’s Zochi and the ACL claim

Intology reported that its system Zochi produced a paper accepted to the ACL 2025 main conference, about automated multi-turn jailbreaking of language models with tree search (the paper is called Tempest). Coverage such as Forbes Argentina repeats the developer’s statements that humans contributed only some figures and style corrections, that the meta-review score was 4, and that this placed the paper in the top 8.2% of submissions, against an ACL acceptance rate of 21.3%.

We rate this unverified. The sources are the developer’s own statements as relayed by the press. If true, it would be a stronger claim than the Sakana workshop result because it is a main-track venue. We could not find an independent account of how much human work preceded submission, so we cannot say how autonomous it was.

Agent Laboratory

Agent Laboratory (arXiv:2501.04227) by Samuel Schmidgall and colleagues takes a different stance. Its abstract reports that the system reduces research expense by 84% compared with earlier autonomous methods, that o1-preview-driven runs gave the strongest results, and that human feedback at each stage meaningfully improved overall quality.

That last finding is the instructive one. A system designed for a human in the loop reports that the human makes it better. We corrected one error from our earlier article here: Agent Laboratory is not a Google DeepMind project as we previously wrote, and we have removed the specific success rate we cited because we could not trace it.

PaperBench: can agents replicate research?

If agents are going to publish, they should at least be able to reproduce known results. OpenAI’s PaperBench (arXiv:2504.01848) asks agents to replicate 20 ICML 2024 spotlight and oral papers from scratch. Replication is graded against rubrics with 8,316 individually gradable leaf nodes, using an LLM judge that reached an F1 of 0.83 against human grading.

The best result in the main setup was Claude 3.5 Sonnet (New) with a basic agent scaffold at 21.0%. A variant using o1 reached 24.4%, and 26.0% with a 36-hour limit. Eight ML PhD participants attempted a 3-paper subset: the best of three human attempts scored 41.4% after 48 hours, versus 26.6% for o1 on the same subset.

Read that carefully. On a task much narrower than discovery, rebuilding code for a published paper whose method is described, agents in that benchmark earned about a quarter to a fifth of the available credit. Those figures are from 2025 models and will have moved, so check the current leaderboard. The gap to humans was real at the time of publication.

Agents4Science: AI as author and reviewer

The Agents4Science conference at Stanford made the AI-as-author experiment public. According to the organisers’ paper (arXiv:2511.15534), 315 submissions arrived, 62 were incomplete and desk-rejected, and 253 complete papers were reviewed. A shortlist of 79 went to human expert review, and 48 papers were accepted, about 15% of submissions according to Science News.

The reviewer findings are the most useful part for this fact-check:

Three LLM reviewers (GPT-5, Gemini 2.5 Pro and Claude Sonnet 4) had an average pairwise Pearson correlation of only 0.48 with each other.
Mean absolute difference from human scores on the 79 papers was 0.91 for GPT-5, 1.09 for Claude Sonnet 4 and 2.73 for Gemini 2.5 Pro. The average scores ranged from 2.30 for GPT-5, the harshest, to 4.23 for Gemini, the most generous.
Fully AI-driven papers made up 23.3% of submissions but only 14.9% of accepted papers, so the most autonomous entries fared worse than average.
About 44% of submissions had no hallucinated references, which means more than half had one or more flagged.
Two papers attempted prompt injection to sway the LLM reviewers. Neither was accepted.

Reviewer Risa Wechsler, quoted by Science News, said the papers were technically correct but “neither interesting nor important,” and warned that technical skill can mask poor scientific judgment. An economist who reviewed a paper had to check an original source to catch an AI tool repeatedly giving the wrong date for a policy.

What the four have in common

Across these systems the pattern is stable. Generation is cheap and fast. Verification is expensive and still human. Where humans are removed from the loop, quality metrics fall, as the Agents4Science acceptance gap suggests, and where they stay, results improve, as Agent Laboratory reports.

Why Simulated Peer Review Is Not Peer Review

The most common technical sleight in the viral version is treating an automated reviewer score as equivalent to human acceptance. They are related, and they are different instruments. Understanding the difference is the core skill for reading any future headline.

What an LLM reviewer measures

A language-model reviewer reads the text and judges plausibility: is the structure sound, are the claims consistent with the figures described, does it resemble a decent paper? That produces a score correlated with committee decisions, which is how the 69% balanced accuracy was obtained. It is evidence that surface and structural features predict acceptance.

It is not evidence that the reviewer can detect that an experiment never ran, that a baseline was untuned, that a number was invented, or that an idea was published in 2019. Those checks require running code, reading logs and knowing the literature, and they are exactly where the independent evaluation found failures.

Reviewer disagreement is itself a warning

The Agents4Science data show how unstable model reviewers are. A pairwise correlation of 0.48 between three frontier models means that which paper “passes” depends heavily on which model reads it. The spread between 2.30 and 4.23 in average score implies calibration differences larger than the signal on many papers.

This is the same problem that affects any LLM-as-judge setup: position bias, verbosity bias and self-preference. If you want the mechanics and the mitigations, our guide on LLM-as-a-judge evaluation, bias and calibration covers them in detail. A paper-writing model scored by a similar model is a textbook self-preference setup.

Human peer review is also flawed

It would be too easy to treat human peer review as the gold standard. Human reviewers miss errors, reviewers disagree, and the 66% balanced agreement figure cited in the Nature write-up shows how noisy committee decisions are. Peer review is a coarse filter for plausibility and importance, not a certificate of truth.

That matters in two directions. It means an AI paper clearing a human workshop review is weaker evidence than it sounds, because the filter is lenient. It also means that the real test of research is replication and use by others over years, which no current autonomous system has been through.

The gap peer review cannot close

Even a perfect reviewer sees only the submitted manuscript. Selective reporting, where the agent ran twenty variants and wrote up the one that worked, is invisible in a single paper. A human researcher can be asked about it. A pipeline’s logs can reveal it, but only if someone reads them.

This is why audit trails matter more for AI-generated research than for human work. If you are building agents that run for hours and touch consequential outputs, the patterns in our piece on long-running governed AI agents apply directly: logged tool calls, checkpoints and approval gates.

An Audit Checklist for AI-Generated Papers

If a paper or a headline claims an AI scientist did the work, the right response is a gate sequence, not an opinion. The sequence below puts the cheapest, most decisive checks first. It is our synthesis of the failure modes documented above, not an official standard.

Figure 4: A gated audit for AI-generated research. Failing early gates is a rejection; failing later gates is a hold for expert judgment.

Figure 4 orders the checks by cost. The first three gates need no domain expertise, only access to logs and a bibliography. The last two need a specialist, which is exactly the scarce resource that the automation was supposed to save.

Gate 1 and 2: did it run, and do the numbers match?

Require the run logs, the code and the raw outputs. Every number in the paper’s tables should be traceable to a logged result. The independent evaluation found hallucinated numerical results, so this is not a formality. A script that extracts the tables from the manuscript and diffs them against the experiment outputs catches the worst class of error in minutes.

Gate 3: do the citations exist and say that?

Hallucinated references were present in more than half of the Agents4Science submissions, and misattributions appeared in Sakana’s own accepted paper. Resolve every reference against a bibliographic index and read the cited sentence. Reference existence is cheap to check automatically; whether the source supports the claim needs a person or a careful second model.

Gate 4 and 5: novelty and fair baselines

Novelty requires someone who knows the field. A related-work search by the same model that generated the idea is not independent, which is why micro-batching for SGD could be passed off as new. Baselines need equal tuning budgets and repeated seeds. A single-seed gain of a percent on a benchmark is noise, however fluent the prose.

Where AI Research Agents Genuinely Help

A fact-check that only subtracts is incomplete. Several uses hold up even under the strictest reading of the evidence.

Breadth of search. Tree-search over experiment variants lets a team try many small ideas cheaply. The value is in generating candidates for a human to triage, similar to how automated hyperparameter search already works.

First drafts and boilerplate. Agent Laboratory’s finding that human feedback at each stage raises quality describes the productive pattern: the human sets direction and verifies; the agent produces the draft, plots and scaffolding.

Replication attempts. PaperBench frames replication as a measurable task. Even at 21 to 26% rubric credit in the 2025 results, agents attempting replication produce partial implementations a human can complete and compare against the original claims.

Negative results. The accepted ICLR workshop paper reports an unexpected obstacle. Negative and null results are chronically under-published, and cheap automated runs could help fill that gap, provided each result is verified before it enters the literature.

Reviewer assistance. Used as a checker for formatting, missing references and internal consistency, a model reviewer is a useful first pass. Used as the decision maker, it is not.

For a broader view of how autonomous tool-using agents behave when their actions matter, our guide to Claude computer use and desktop agents covers the architecture of agents that act in environments, which is the same capability that powers lab-in-a-sandbox research loops.

Trade-offs, Gotchas, and What Goes Wrong

The authors of the leading system list the risks that matter, and we agree with their ordering.

Volume against review capacity. If a paper costs under $15 in compute, then submissions can scale faster than reviewers. Peer review is a volunteer system with finite capacity. The Agents4Science experience, where fully AI-driven papers were nearly a quarter of submissions in a conference designed for them, hints at the load. Venues are already adapting disclosure rules, though policies differ by venue and we have not verified the 2026 policy of each.

Pollution of the literature. A fabricated number that passes a lenient review becomes a citation, and later models train on it. The Nature authors flag hard-to-filter noise as a concern. Noise is more dangerous than wrongness because it is expensive to filter and cheap to produce.

Reviewer gaming. Two Agents4Science papers tried prompt injection on the LLM reviewers. Neither succeeded, but any pipeline that routes manuscripts through model reviewers needs the same defences as any system that passes untrusted text to a model, including stripping hidden instructions.

Selection effects in demos. Every documented success involves selecting a result. Three papers were generated, one was accepted; the number of papers generated before selection is not stated in the sources we reviewed. A system that makes ten papers and humans pick the best is a different system from one that must stand behind each output.

Evaluation by the thing being evaluated. Using a model reviewer to measure model quality creates a loop. Scaling laws for “scientific quality” are scaling laws for model-reviewer score until validated against human outcomes, replication or downstream citation.

Domain limits. The system works in computational domains where an experiment is a script. Biology, chemistry and physics add instrument time, materials and safety, and none of the evidence here transfers to them. Claims about “curing disease” from paper-writing agents are outside the data.

Human time hidden in the cost. The $6 to $15 figure excludes the roughly 3.5 hours of human involvement that the independent study measured. At any realistic hourly rate, the human time dominates the cost.

Practical Recommendations

For readers deciding what to believe, and for teams deciding what to build, the evidence supports a clear stance: use these systems as accelerators with a verifier attached, and distrust any claim that omits the verifier.

If you are a researcher or reviewer, ask for logs and code with any paper that discloses AI generation, and apply the gate sequence in Figure 4. If you run a venue, require disclosure of AI authorship and review model use, and keep humans on the decision. If you are a builder, log everything the agent does so the audit is possible.

If you are reading a headline, here is a short checklist:

Which venue, and what is its acceptance rate? Workshops are not main tracks.
Who chose the topic and who chose which output to submit?
Was the result independently replicated, or is the developer the only source?
Did anyone check that the numbers match the logs and that citations exist?
Was the paper actually published, or withdrawn, or merely accepted for review?
Is the quality metric produced by a model similar to the generator?
What does the developer’s own limitations section say?

We would revisit our ratings if a system shows a fully AI-generated paper clearing a main-track venue with disclosed process, independently audited logs and no human selection. The Nature paper’s scaling claim suggests that day may come, but the evidence as of October 2026 does not show it.

Frequently Asked Questions

Has an AI scientist really had a paper accepted by peer review?

Yes, with qualifiers. In 2025 Sakana submitted three AI-generated papers to an ICLR workshop, and one received reviewer scores of 6, 7 and 6, above the acceptance threshold. It was a workshop, humans chose the papers, and the paper was withdrawn by design. Intology separately reports a main-track ACL 2025 acceptance for its Zochi system, but that claim comes from the developer and we could not verify it independently.

Can AI auto-publish papers without human involvement?

Not in any documented case. Humans set topics, select which generated papers to submit, run code checks and handle ethics approval. The Sakana paper was withdrawn and never published. A person must also submit to a venue, and conferences such as Agents4Science were specially created for AI-authored work. Fully autonomous publication, where no human decides, has not been demonstrated or independently audited.

How much does an AI-generated paper cost?

Sakana’s original paper claims under $15 per paper, and an independent evaluation measured $6 to $15 in compute and API costs. That excludes human time: the same study estimated about 3.5 hours of human involvement per paper. Costs depend on the underlying models and how much search the system runs, and the figures refer to the 2024 system, so current costs may differ.

Are AI-written research papers accurate?

Often not reliably. The independent study found 42% of experiments failing from coding errors, hallucinated numerical results, outdated citations and misclassified novelty. The Agents4Science report found more than half of submissions had at least one flagged hallucinated reference. The authors of the leading system acknowledge weak ideas, incorrect implementations, experimental errors and hallucinated citations. Always verify against logs and sources.

Can an LLM replace a human peer reviewer?

Not yet. An automated reviewer reached 69% balanced accuracy predicting ICLR decisions, versus 66% agreement between human experts in one cited study, but predicting a committee is not the same as finding errors. At Agents4Science, three frontier-model reviewers correlated only 0.48 with each other, and their scores ranged widely. They are useful for first-pass checks, not final decisions.

What would convince you an AI scientist does real science?

Independent replication of its findings, a main-track acceptance with a disclosed process and no human selection, audited logs showing experiments ran as reported, and downstream use by other researchers. Novel results in wet-lab fields would be stronger still, since they require physical experiments. As of October 2026, none of these has been shown for a fully autonomous system.