The MAHA Commission Report: Documented by AI. Does it Matter? Yes, a Lot.
Let me start by confessing that I did not review the references in the MAHA Comission report I wrote about last week—except for mine.
The reference to my book, Food Politics, is a bit garbled (In Food Politics? No. This is Food Politics), but these are basically OK. It’s easy to make mistakes like that one and I rely on the help of many proofreaders and factcheckers to try to avoid such errors in my published work. I checked a couple of the other references related to food topics and they seemed basically OK too.
So I was surprised by the report from NOTUS that The MAHA Report Cites Studies That Don’t Exist,
This finding was immediately attributed by the New York Times and other sources to the report’s having been referenced by Artificial Intelligence (AI), a tool well known to be scientifically inaccurate and to make things up.
To immediately plagiarize (well, quote) Ted Kyle at ConscienHealth: The MAHA Report: Make America Hallucinate Again.
I was also surprised—no, dismayed—by the administration’s response to these discoveries.
White House Press Secretary, Karoline Leavitt told reporters…“I understand there were some formatting issues with the MAHA report that are being addressed,” Leavitt said. “But it does not negate the substance of the report, which, as you know, is one of the most transformative health reports that’s ever been released by the federal government, and is backed on good science that has never been recognized by the federal government.”
FoodFix also quotes the HHS Press Secretary:
Minor citation and formatting errors have been corrected, but the substance of the MAHA report remains the same — a historic and transformative assessment by the federal government to understand the chronic disease epidemic afflicting our nation’s children…“It’s time for the media to also focus on what matters.”
Formatting issues? Oh come on.
Calley Means, the top advisor to RFK Jr, posted “The least surprising thing about the MAHA Report is that the media and failed medical leaders are talking about footnotes instead of its actual content.”
Sorry. Footnotes matter. Everything in a report making policy recommendations depends on where its information comes from. Hallucinating references implies hallucinating data.
The MAHA Report is now being continually updated to fix the citation problem.
Some of the updates are introducing other errors.
Yikes.
The Washington Post has published details: The MAHA Report’s AI fingerprints, annotated.
I was interviewed by Reuters about all this:
Nobody has ever accused RFK Jr. of academic rigor…The speed (of the MAHA report) suggests that it could not have been vetted carefully and must have been whisked through standard clearance procedures. The citation problem suggests a reliance on AI.”
Science magazine headlined the downplaying of the fake citations and pointed out the irony:
Problems with the MAHA report’s integrity came to light even as Kennedy has threatened to prevent government scientists from publishing in leading medical journals like The Lancet, The New England Journal of Medicine, and JAMA, which he claims are “corrupt” and controlled by pharmaceutical companies. Kennedy has instead proposed a state-run alternative.
Discovery of the fake citations also came just days after President Donald Trump unveiled an executive order that called for “Restoring Gold Science Standards” to government activities…One goal, Trump wrote, is to ensure that “Federal decisions are informed by the most credible, reliable, and impartial scientific evidence available.”
Yeah, right.
All of this has led cartoonists like Clay Bennett to ridicule the report.
Here’s another good one from Carlos Muñoz.
Ridicule—or lack of credibility if you prefer—is one reason why this matters.
What I had drilled into me as a graduate student in molecular biology was the importance of reading references, and never under any circumstances citing a reference I hadn’t read.
Why? Because the credibility of my work depends on where I got my information—how I know what I claim to know.
When I managed the editorial process for the 1988 Surgeon General’s Report on Nutrition and Health, checking references was crucial to supporting the report’s recommendations. It took years to get the report out, not least because of the enormous amount of vetting involved—from scientists, but also government agencies.
This report, unfortunately, was a rush job. It astonished me that it got done in only three months (I really want to know who wrote it).
It’s one thing to make editorial errors in citing references (try as hard as I can to get them right, errors invariably get overlooked).
But this report had references that were made up. Hallucinated. This means nobody looked at them.
If its references are not reliable, nothing else in the report can be trusted either.
And that’s a shame. It said a lot of things that badly needed to be said.
Too many corners were cut in throwing this together at the last minute. I know this was a rush job because I have four versions of the report.
- Version 1: May 22 early morning. It has no references at all. My first reaction: What? No references?
- Version 2: Also May 22, but later in the morning. It has references but not much in the way of formatting.
- Version 3: May 22. This is the one released to the public.
- Version 4: This is the one online on May 31. Other versions, perhaps multiple, came in between and may come after.
None of this bodes well for the future of MAHA initiatiatives. Sad.