How Can Researchers Use AI for Literature Reviews Without Losing Reliability?

Can AI find relevant scientific papers accurately?

AI can help surface candidate papers, but evidence from the supplied studies does not support relying on it as a complete or dependable literature-search system. Its usefulness depends on the task and tool; a fluent list of results can still omit relevant work, include irrelevant records, or fail to provide a reproducible search trail.

A PLOS ONE comparison examined ChatGPT’s biomedical literature-search performance against conventional methods, including Google and PubMed, across four search scenarios. Basic ChatGPT had limitations in consistency, accuracy, and relevance. Browsing, plugins, prompt engineering, and custom tools improved results, but limitations remained, and the tested systems did not meet the reliability level needed for broad adoption as scientific literature-search tools.

A separate JMIR Medical Informatics comparison makes the coverage problem concrete. For a systematic-review benchmark on Peyronie disease, ChatGPT identified 1,287 studies but only seven were directly relevant; Bing AI identified 19 relevant studies among 48, while the human benchmark contained 24. These results are specific to that comparison, not a universal score for every tool or field. They do show why a large result count is not the same as a useful search.

Use AI to propose synonyms, related concepts, or questions worth searching. Then run a documented search in the databases appropriate to the discipline, such as PubMed for biomedical literature or Google Scholar as another discovery route. Compare results, check whether known relevant papers appear, and use conventional citation searching where appropriate. AI can expand the searcher's vocabulary; it should not silently determine the evidence base.

How can I check whether AI-generated references are real?

Check each reference by matching its bibliographic details to a reliable database record and then opening the paper itself. A citation that looks complete may still be invented, inaccurate, or attached to a real paper that does not support the claim being made.

The JMIR analysis of ChatGPT and Bard generated 471 references across 33 prompts based on 11 systematic reviews in four fields. Citation precision was 9.4% for GPT-3.5, 13.4% for GPT-4, and 0% for Bard; the measured hallucination rates were 39.6%, 28.6%, and 91.4%, respectively. The authors conclude that language models should not be the primary or exclusive tool for systematic reviews. They state: “Any references generated by such models warrant thorough validation by researchers.”

A practical verification sequence is simple, though not optional:

1. Search the exact title in a scholarly database or publisher site. If no matching record appears, do not cite the reference as supplied.
2. Compare authors, title, journal, year, and other available publication details with the record. Do not assume that a correct-looking title validates the rest.
3. Open the original paper and confirm that it supports the specific claim for which you plan to cite it. Check the relevant methods, results, or discussion rather than relying only on an abstract or AI paraphrase.
4. Keep the verified record and note any discrepancy or correction. Exclude references you cannot substantiate.

This process separates two questions that are easily collapsed: “Does this paper exist?” and “Does this paper support this statement?” The first is bibliographic verification; the second requires reading and interpretation.

Is it safe to use AI to summarize research papers?

AI summaries can be useful for initial triage, but they are not a definitive source of truth. Their main value is speed in orienting a researcher to a paper; their main risk is compressing away a qualification, method detail, or result that changes how the evidence should be interpreted.

In a study published in The Annals of Family Medicine, physicians evaluated ChatGPT summaries of 140 peer-reviewed abstracts from 14 journals. The summaries were 70% shorter than the abstracts. Median physician ratings were 90 out of 100 for quality and 92.5 out of 100 for accuracy, yet hallucinations occurred in two abstracts and were mainly small but important methodological or results details. The authors concluded that summaries have rare but important inaccuracies that prevent them from being treated as a definitive source of truth.

The high ratings and the errors belong in the same account. A summary may be clear and broadly accurate while omitting the very detail that matters for a review's inclusion decision or conclusion. The study also found AI more modest at classifying relevance for individual articles than for whole journals, reinforcing the distinction between helping with triage and making the final judgment.

Use summaries to decide what to read next, identify terms to check, or orient yourself before opening a paper. For included studies and any source that carries weight in the final argument, inspect the original text and verify the methods, population, outcomes, limitations, and results that matter to your question. The synthesis—the judgment about how findings fit together—remains the researcher's responsibility.

Conclusion

The reliable role for AI in a literature review is bounded: it can help generate leads and reduce friction in early screening, but it cannot certify its own citations, establish search completeness, or replace critical reading. Treat AI output as a proposal that must earn its place in the evidence record.

Reproducibility requires more than saving a final list of papers. PRISMA-S provides a 16-item checklist for reporting literature searches, including database and platform names, full strategies as run, search dates, limits, filters, updates, citation searching, record counts, and deduplication methods. Pair that search record with the AI tool, version, prompts, and screening decisions. The result is a process another researcher can inspect—and, as far as the documented methods allow, repeat.