Bingo! Remember when Watson beat Ken Jennings at Jeopardy! and was suddenly going to cure cancer. I am still waiting. We're at the same point in the hype cycle with LLMs/Foundation Models, but there are many, many of those. AlphaFold was very good at predicting protein structures, but a protein is a one-dimensional chain of amino acids. The link between genotype and phenotype (our genetic variants and our traits) is highly nonlinear, multidimensional, and multifactorial. AI models can learn rules (or pretend to), but creating new insights, and testing them, is a far more challenging than using a probabilistic model to string together some words or concepts.
All the glory goes to the predictors, so little goes to the validators!
I work mostly in non-model organisms, and I find it a real challenge when these splashy announcements come out and I get asked when I can integrate them into projects... I want to leverage all the fancy new tools and data available for model organisms, but the amount of validation data is so scarce, I don't really know how I would evaluate if some tool is actually working.
Reminded me of Rachel Thomas' article about the Nature Comms ML Gene Annotation paper which had serious problems with the predictions, and an expert was available to fact-check one that she knew was wrong. What happens as we get fewer deep experts and more generalists?https://rachel.fast.ai/posts/2025-06-04-enzyme-ml-fails/
I find it somewhat perplexing, though, that you talk about skepticism and seem to tout acceptance rates through peer review as quality control for articles, yet peer review just manages to filter the obviously bad stuff with a lot of effort, but not much else beyond formatting issues, obvious plagiarism and obvious research design issues. As the actual deep sleuth-type detection of fraud, QRPs and so on is generally done in post-publication peer review. And acceptance rates are generally not much more than PR for the journals. Peer review as a filter for bad science is troubling enough already and should not be used to rate journals for actual quality. But beyond that, I completely agree with your point. They are probably not really scientists in the way people are used to. Where skepticism is not an important element, or maybe it is mostly due to a lack of domain expertise in molecular biology?
It seems the greatest threat of Ai psychosis lives in the minds of those who work in the industry that essentially believe they're the midwives of a new cyborg deity.
This is a general issue and challenge we face right now, amplified by the interplay between publishers, funding agencies, and researchers. While the way we do research has changed dramatically, the cultural confrontation between natural science research and engineering has overturned what we view as good practice. I have also been raising the issue to trainees and colleagues, as in this talk: https://www.youtube.com/watch?v=xT6nb_Qg1eA&t=1002s
Bingo! Remember when Watson beat Ken Jennings at Jeopardy! and was suddenly going to cure cancer. I am still waiting. We're at the same point in the hype cycle with LLMs/Foundation Models, but there are many, many of those. AlphaFold was very good at predicting protein structures, but a protein is a one-dimensional chain of amino acids. The link between genotype and phenotype (our genetic variants and our traits) is highly nonlinear, multidimensional, and multifactorial. AI models can learn rules (or pretend to), but creating new insights, and testing them, is a far more challenging than using a probabilistic model to string together some words or concepts.
All the glory goes to the predictors, so little goes to the validators!
I work mostly in non-model organisms, and I find it a real challenge when these splashy announcements come out and I get asked when I can integrate them into projects... I want to leverage all the fancy new tools and data available for model organisms, but the amount of validation data is so scarce, I don't really know how I would evaluate if some tool is actually working.
Reminded me of Rachel Thomas' article about the Nature Comms ML Gene Annotation paper which had serious problems with the predictions, and an expert was available to fact-check one that she knew was wrong. What happens as we get fewer deep experts and more generalists?https://rachel.fast.ai/posts/2025-06-04-enzyme-ml-fails/
I find it somewhat perplexing, though, that you talk about skepticism and seem to tout acceptance rates through peer review as quality control for articles, yet peer review just manages to filter the obviously bad stuff with a lot of effort, but not much else beyond formatting issues, obvious plagiarism and obvious research design issues. As the actual deep sleuth-type detection of fraud, QRPs and so on is generally done in post-publication peer review. And acceptance rates are generally not much more than PR for the journals. Peer review as a filter for bad science is troubling enough already and should not be used to rate journals for actual quality. But beyond that, I completely agree with your point. They are probably not really scientists in the way people are used to. Where skepticism is not an important element, or maybe it is mostly due to a lack of domain expertise in molecular biology?
It seems the greatest threat of Ai psychosis lives in the minds of those who work in the industry that essentially believe they're the midwives of a new cyborg deity.
Well done for reading this stuff so I don't have to... Nature has always always been a sucker for fads.
This guff is next level though. Really lame.
Variant effect prediction with language models doesn't really work. Good accuracies are mostly just data leakage: https://blog.genesmindsmachines.com/p/protein-language-models-are-bad-at
More generally, AI in biology is way overhyped: https://blog.genesmindsmachines.com/p/we-still-cant-predict-much-of-anything
This is a general issue and challenge we face right now, amplified by the interplay between publishers, funding agencies, and researchers. While the way we do research has changed dramatically, the cultural confrontation between natural science research and engineering has overturned what we view as good practice. I have also been raising the issue to trainees and colleagues, as in this talk: https://www.youtube.com/watch?v=xT6nb_Qg1eA&t=1002s