AI in single-cell biology: large models need independent evaluation

MIA: Lorin Crawford, Rethinking scale in single-cell foundation models; Primer: Davide D'Ascenzo

Broad Institute · 2025-10-02 · Source

Broad Institute research seminar held on September 24, 2025, published on October 2, 2025. Covers data scale, composition and evaluation in single-cell models. In English, approximately 1 hour 54 minutes; not a clinical-use authorization.

Research: what was the model built to do?

The 2024 scGPT study describes pretraining on data from more than 33 million cells. With further task-specific training, the researchers examine cell-type annotation, integration of datasets and prediction of responses to cellular perturbations. The inputs are gene-expression measurements; this should not be read as a complete simulation of everything a cell does.

Independent testing: what happens without further training?

A 2025 study evaluates scGPT and Geneformer without additional task-specific training, known as zero-shot use. Across five datasets, these models do not consistently outperform simpler methods in cell clustering, and technical differences between experiments remain challenging. The findings concern the evaluated model versions and tasks.

Evidence limits: the questions differ

Performance after task adaptation differs from performance without it. Some evaluation datasets also appeared in pretraining; the independent study identifies this overlap as a limitation. Its findings are not a verdict on every future model.

Editorial proposal: a small, explicit evaluation plan

For a biotechnology team, start by writing the research question before choosing a model name. What output is needed, and which experiment could an incorrect result affect? We suggest defining the existing comparison method, reserved test data and acceptance criterion in advance. Reporting failures alongside successful examples helps teams establish realistic expectations. These proposals are our editorial assessment, separate from the studies’ reported findings.

Clinical use and regulatory authorization are separate questions

The evidence considered here is research evidence. We do not present a published model comparison as authorization for diagnosis, treatment selection or medical-device use. Any proposed clinical application requires separate examination of its intended task, validation scope and applicable product documentation. This article offers neither individual treatment advice nor claims about the effectiveness of RAI products.

Responsible AI: make the evaluation traceable

Our editorial recommendation is to record the model version, data source, whether additional training was performed and the test date alongside each evaluation. Explaining which question was answered under which conditions is more useful than saying that a model worked. That record also makes it possible to repeat the comparison when a model version or dataset changes.