New College of Florida professor co-authors study questioning AI’s ability to replace statisticians

A peer-reviewed study co-authored by New College of Florida’s Bernhard Klingenberg found that major AI models are too sensitive to prompt wording to reliably replace trained statisticians. The findings, published in the International Statistical Review, matter for research, analytics and statistical education as AI use expands.

Why it matters: - The study challenges a growing assumption that AI can substitute for trained statisticians in research and business workflows. - The findings suggest prompt wording can change results enough to undermine confidence in AI-driven statistical analysis. - The work points to a narrow but useful role for AI in early-stage screening, not final quantitative judgment.

What happened: - Bernhard Klingenberg, a professor of statistics and director of the graduate program in Applied Data Science at New College of Florida, co-authored a peer-reviewed study on whether AI can replace statisticians. - The study, "Can AI Replace the Statistician? Evidence From a Controlled Experiment," was published in the International Statistical Review. - Klingenberg partnered with University of South Florida researchers Wolfgang Jank, Daniel Zantedeschi and Sonal Prabhune. - The team tested five prominent AI models across three phrasing scenarios and four complex data tasks. - The experiment used ground-truth data that the researchers knew but withheld from the AI models.

The details: - The study found that model accuracy shifted by 30 to 80 percentage points depending on how the same question was worded. - The researchers concluded the models were matching linguistic patterns rather than performing genuine statistical analysis. - The study identified a "confidence-competence inversion," where models were least helpful to users who asked vague questions and most needed guidance. - The researchers also found a "mirror paradox": more precise statistical language sometimes reduced performance. - GPT-4 performed better with vague or ill-formulated prompts and worse with more formal statistical terminology. - GPT-4o showed the opposite pattern, suggesting differences in architecture and training. - The tested LLMs performed poorly on quantitative tasks such as effect size estimation. - Some models scored as low as 15% accuracy on those complex tasks. - The models did better on basic pattern-recognition work, reaching 76% to 81% accuracy. - That level of performance suggests AI can help with initial data screening before a human expert steps in. - Klingenberg said the research exposes a blind spot in how AI handles data. - Jank said AI behaves more like a linguist than a statistician because prompt phrasing can drive answers more than the data itself.

Between the lines: - The study draws a sharp line between language fluency and statistical reasoning. - That distinction matters because many users may assume polished AI answers reflect sound analysis. - The results also imply that stronger prompting skills can change outcomes without changing the underlying data, which raises concerns about reliability in high-stakes settings. - The authors said the findings will need to be revisited as AI models continue to improve.

What's next: - The professors expect the study to be reexamined as newer AI models add capabilities. - New College of Florida says its Master of Science in Applied Data Science program is built to bridge technical theory and real-world application. - The program focuses on statistics, machine learning, databases and computing. - The three- or four-semester curriculum includes an applied project in every course. - New College says the program was ranked among Fortune’s top 10 national programs. - New College of Florida, founded in 1960, is the state’s designated public honors college and offers more than 50 undergraduate majors plus graduate programs in Applied Data Science, Marine Mammal Science and Educational Leadership. - The college also says it has a growing NAIA athletics program on a 151-acre Sarasota Bay campus. - More information is available in the company’s announcement.