Search results: research bias

I learned about this from a tweet.

I recognized the clip.  It was from an interview I did in January in Toronto: TVO’s The Agenda: Battling bias in nutrition research (slso on YouTube, and in transcript).  Nam was the terrific interviewer.

But do not miss Minhaj’s last Patriot Act episode of 2019, “How America is Causing Global Obesity.”  This is a brilliantly researched account of obesity politics, from food industry influence to trade policy.

I couldn’t have done better myself and dearly wish I had his production team (and his performance ability).

The study: Matthews LG, Smyser CD, Cherkerzian S, Alexopoulos D, Kenley J, Tuuli MG, et al. (2019) Maternal pomegranate juice intake and brain structure and function in infants with intrauterine growth restriction: A randomized controlled pilot study. PLoS ONE 14(8): e0219596.

The findings (my emphasis): “There were no group differences in brain injury, metrics or volumes. However, treatment subjects displayed reduced diffusivity within the anterior and posterior limbs of the internal capsule compared with placebo. Resting state functional connectivity demonstrated increased correlation and covariance within several networks in treatment subjects, with alterations most apparent in the visual network in per-protocol analyses. Direct effects on health were not found.

Conclusion: In conclusion, maternal pomegranate juice intake in pregnancies with known IUGR was associated with altered white matter organization and functional connectivity in the infant brain, suggesting differences in brain structure and function following in utero pomegranate juice exposure, warranting continued investigation.

Funding: This work was supported by National Institute of Health Grants R01 HD29190 (D. M. Nelson), K02 NS089852 (C.D. Smyser), U54 HD087011 and P30 HD062171 (T.E. Inder), The Foundation for Barnes-Jewish Hospital (D. M. Nelson) and an unrestricted gift to Washington University School of Medicine from POM Wonderful, Los Angeles, CA. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Comment: This is a classic example of interpretation bias.  Studies of bias associated with industry funding find that it shows up mostly in the framing of the research question or in the interpretation, as this one demonstrates.  The study did not find anything significant but concluded that drinking pomegranate juice during pregnancy is good for the growing fetus.

Bottom line: Fruit juices (of any kind) are good for health as long as volumes are small.  Eating the fruit itself is better–less sugar, more fiber.

In a recent NutraIngredients-USA commentary it was argued that dietary supplement companies that build up a suite of research do so carefully, and plan for success​​. With a careful design of the research program, positive results at the bench can naturally translate into successful randomized, placebo controlled trials.

Longtime critic of industry funded research

Marion Nestle, PhD, nutrition professor at New York University and author of the influential book Food Politics​ as well her most recent work Unsavory Truth: How Food Companies Skew the Science of What We Eat, ​ isn’t buying. Nestle responded to NutraIngredients-USA on the subject of the commentary to say that she has reviewed hundreds of company-funded studies and in her view the inherent biases built into that system are next to impossible to overcome.

Indeed, on her blog (also titled Food Politics​​) Nestle has a frequent feature called “Industry funded research of the week.” The feature is used to document instances of bias, which to Nestle’s eye are thick on the ground.

“The overriding issue is that industry-funded research almost invariably comes out with results favorable to the sponsor’s interests. This is so predictable that I can often recognize the funder by the title of the paper,”​ Nestle told NutraIngredients-USA.

Nestle noted that the phenomenon is not by any means restricted to research on food and supplement ingredients. It has been noted in studies on tobacco, chemicals and pharmaceuticals.

“Studies of these industries show that the influence apparently occurs at an unconscious level; investigators did not intend to be influenced and do not recognize that they were influenced. But the evidence for funding effects is overwhelming,”​ she said.

Bias starts with study design

Nestle said a key issue is how the research question is framed. Companies investing in research naturally want to succeed and get some return on their investment. But the best science doesn’t come when the question is framed in such a way that a positive result is overwhelmingly likely, she said.

“I get letters all the time from trade associations asking for proposals for research that will demonstrate the benefits of their products. That is not the same as asking open-ended questions about effects. Companies want data on benefits for marketing purposes. That’s why I view industry-funded studies as about marketing, not science,”​ Nestle said.

Nestle said she has noted that some of the larger funders, such as the larger food companies or industry associations supporting categories of products like walnuts, almonds, strawberries or what have you, churn out research supporting their products of interest. While some might argue this adds to the totality of evidence and thus could be a good thing, Nestle said she doubts the value of these investments when taking the inherent biases into account. This can result in studies that seek to demonstrate things like substituting junk food calories with a serving of something like almonds or strawberries is a good thing to do. Yes, but so what? In Nestle’s view, this kind of bias is all but inevitable in research funded in this way.

“That’s what decades of research on the effects of drug-industry funding says, and the few studies looking at funding effects in nutrition find similar results,” ​she said.

Independent funding mechanism

What Nestle said she’d like to see is a mechanism for funding research into food and supplement ingredients that was divorced from a marketing plan.

“I want to see a firewall between the funder and the scientist. In ​Unsavory Truth, I talk a lot about various attempts over decades to create such firewalls and develop a pool of industry research funds managed by independent third parties. They have never worked well,”​ she said.

Nestle said she believes that only by making contributions to research compulsory, with the resulting fund to be managed by a credible third party, can research of undeniable quality be done. Something like the Beef Checkoff Program but for independent research funding, even if the results of those studies might not immediately support the marketing of the products.

“My idea of an ethically funded study is to ask for investigator-initiated proposals, appoint third party reviewers who decide who gets funded, and stay completely out of the process from then on. I worry when I see disclosure statements that the funder had no role in the study because that statement has been demonstrated to be false so many times. Food companies are funding research because they want specific results. That’s not how science is supposed to work,”​ Nestle said.

The press release from the Annals of Internal Medicine arrived last week under embargo, sent to me by several reporters: “New guidelines: No need to reduce red or processed meat consumption for good health.”

The press announcement was accompanied by 5 review papers, a sixth with this recommendation, and an editorial.  These are posted on the website of their publisher, the American College of Physicians, implying this organization’s endorsement of this guideline.

Collectively, these papers challenge dietary advice to eat less processed meat and red meat on the grounds of inadequate science.

These papers argue:

• Eliminating processed and red meat from the diet may reduce disease risks but the effects are small and evidence for them is of low quality and low certainty.
• Omnivores are unwilling to eat less meat.
• If you continue to think eating meat is bad for you in the face of this evidence, you are a victim of the “the less you know, the more you hold to your beliefs” syndrome.
• You can ignore dietary guidelines (regardless of source) and continuing eating meat as you always have.

This is a good example of what I call nutritional nihilism, an approach that insists that because observational studies are based on self-reported information and necessarily flawed, their conclusions are unscientific and should be discounted.  Therefore, because we can’t do more rigorous studies, we should not advise the public about diets best for health or the environment.

I have several concerns about all this:

• Yes, the effects are small but that is true of nutrition studies in general.  The small effects show benefits from eating less meat.  The authors could easily have interpreted their work as suggesting that eating less meat might be useful.  This is an example of interpretation bias.
• The authors took a strictly science-based approach to a problem strongly affected by social, economic, and political factors and values.
• The studies look at previous studies that compared people who eat meat to those who eat less. The authors excluded studies of vegetarians compared to meat-eaters.
• They look at meat outside its context of calories.
• The authors did not look at the totality of the evidence; they excluded laboratory and animal studies that can be more closely controlled.
• They excluded studies of environmental impact, which has a significant bearing on human dietary practices (meat production adds more greenhouse gases than vegetable production).
• The conclusions fall into the category of “everything you thought you knew about nutrition is wrong.”  This rarely happens.  Science usually works incrementally, not in one enormous reversal like this.

Do the authors really believe that all those other committees and commissions urging less meat were wrong and continue to be wrong?   Their strictly science-based approach seems unrealistic.

The papers come across to me as a concerted attack on dietary guidelines (national and international), on nutrition science in general, and on nutritional epidemiology in particular.

The meat industry and its supporters will love them.

Attacks on the quality of nutrition research have been coming from many sources lately: the food industry, of course, but also statisticians (John Ioannidis at Stanford is making a career of this), and some scientists (usually with ties to food companies).  The criticisms themselves are not new.

What is new is the vehemence and level of effort to discredit observational studies, particularly those based on self-reports of dietary intake.  Yes, nutritional epidemiology has flaws, but the methods have been useful in many instances, as argued convincingly by two of its leading practitioners.

In looking at nutrition research, I think it is essential to evaluate the totality of information available: laboratory, animal, human epidemiology and clinical studies—and to do this in the context of what people actually eat and the number of calories they consume, as well as adding in a hefty dose of common sense.

Common sense is what’s missing in these studies.  Do the authors really believe that:

• Meat eaters are healthier than vegetarians?
• Eating more meat is better for health?
• Meat eaters are less obese and have less heart disease and cancer than those who eat less?

If not, the conclusions make no sense.

Most of the authors report no financial ties to the food industry.  I would love to know the back story about why they chose to do these studies and to interpret them in this way.

Reactions (I will be adding to this list as they come in)

• The Harvard School of Public Health Nutrition Department issued a statement
• The True Health Initiative (THI) issued a press release outlining its concerns (the link is here)
• THI council members sent a letter to the journal editor stating objections (the link)
• THI collected quotes from members of its council and others (including mine)
• THI also produced graphics.  Here is one example.

• David Katz weighs in
• New York Times: 5 things you should know
• New York Times coverage: “Eat less meat, scientists said.  Now some believe that was bad advice.”
• From Dean Ornish
• Consumer Reports (I’m quoted)
• From the World Cancer Research Fund
• Vox’s crystal clear explanation
• NPR’s The Salt’s excellent analysis
• Aaron Carroll’s follow-up to his editorial

A reader, Gema Flores Monreal, who holds a doctorate in Food Science and Nutrition, pointed me to this study.  She noted that it examines the effects on blood pressure of eating 5 to 6 servings of dairy per day, twice what is typically recommended. 

it is easy to understand why a dairy company would want research like this.  People are consuming less dairy food, and the industry wants to reverse the decline.

The study:  Effect of high compared with low dairy intake on blood pressure in overweight middle-aged adults: results of a randomized crossover intervention study.  Rietsema S, and 11 other authors.  Am J Clin Nutr 2019;110:340–348.

Conclusions: “This intervention study shows that an HDD [high dairy diet] results in a reduction of both systolic and diastolic BP [blood pressure] in overweight middleaged men and women. If the results of our study are reproduced by other studies, advice for high dairy intake may be added to treatment and prevention of high BP.”

Funding: “Supported by the Public–Private Partnership Topconsortium voor Kennis en Innovatie (TKI) Agri & Food (TKI-AF-12104).”  FrieslandCampina, a Dutch multinational dairy cooperative, is part of the partnership.  Two of the authors are employed by FrieslandCampina.

Comment: As I discuss in Unsavory Truth, research like this has a high probability of producing biased results.  I’m reserving judgment about dairy foods and blood pressure until results like these are confirmed by independent research.

I love avocados but does their trade association really need to do research to encourage you to eat them?  Apparently so.

The study: Using the Avocado to Test the Satiety Effects of a Fat-Fiber Combination in Place of Carbohydrate Energy in a Breakfast Meal in Overweight and Obese Men and Women: A Randomized Clinical Trial.  Zhu L, et al. Nutrients 2019, 11, 952; doi:10.3390/nu11050952

Conclusion: “Replacing carbohydrates in a high-carbohydrate meal with avocado-derived fat-fiber combination increased feelings of satiety mediated primarily by PYY [satiety-inducing peptide] vs. insulin. These findings may have important implications for addressing appetite management and metabolic concerns.”

Funding: “This research was supported by the Hass Avocado Board, Irvine, CA, USA.”

Acknowledgments: “The planning, organization of the study as well as data analyses was performed solely by the investigators.”

Comment: Perhaps so, but, as I document in my book Unsavory Truth, an overwhelming body of research demonstrates that the biasing effect of industry funding occurs at an unconscious level and mostly occurs in the design of the research question.  The effect of the funding is usually unintentional and unrecognized, and typically denied.

Thanks to Effie Schultz for sending this one.

This is one of my ongoing updates of industry-funded studies such as those discussed in my most recent book, Unsavory Truth.

Here’s this week’s example:

Funding sources and outcomes of dairy consumption research – a meta-analysis of cohort studies: The case of type-2 diabetes and cardiovascular diseases.  Moshe Mishali, Mirit Kisner,Tova Avrech.  International Dairy Journal, accepted and in press 2019.

The study: The authors asked whether the source of funding was associated with the results of studies examining the association of dairy food intake with the risk of type 2 diabetes or heart disease.

Conclusion:  “This meta-analysis found that the funding source (i.e., food industry sponsorship versus neutral organisations sponsorship) did not affect the findings of studies in terms of the association between dairy consumption and the risk of developing type-2 diabetes and cardiovascular diseases…This meta-analysis suggests that research funded by the dairy industry might not deserve the same dismissive treatment that other industry-funded studies might merit.”

Funder: “This work was supported by the Israel Dairy Board Research – Fund (DBRF). Declaration of interest Moshe Mishali is a consultant for the Israel Dairy Board; Mirit Kisner was paid for her work by the Israel Dairy Board; Tova Avrech is Chief Health Officer at the Israel Dairy Board.”

Comment: This study was commissioned by the Israel Dairy Board specifically to address “The radicalised discourse that emerged in recent years [which] sees industry-funded research as inherently biased due to the obvious vested interests of any industry that initiates and funds certain studies…We sought to check if the suspicion is warranted when it comes to the dairy industry.”

These conclusions are consistent with those of Wilde et al (2012), although that study found that independently funded research included the only studies with results unfavorable to dairy (3 out of 16), whereas the industry-funded research came out 100% in favor of dairy products.

I would find the arguments about the benefits of dairy food more convincing if they were funded and conducted by investigators with no skin in the game.

I listened in yesterday to the webinar on USDA’s proposed relocation and reorganization of the Economic Research Service.  Participants included Scott Swinton (Michigan State University), Cathie Woteki (former undersecretary for research at USDA), Susan Offutt (former ERS head), Gale Buchanan (another former undersecretary for research, USDA), and Stephen Censky (current USDA deputy secretary). The former officials were unanimous in arguing that the proposal to relocate the agency outside of Washington DC and reorganize it into the USDA Secretary’s office was “ill-conceived,” made no sense, was done without appropriate consultation, was potentially illegal, would politicize the agency, and would damage, if not destroy, an agency that is the jewel of USDA. The USDA says the reasons for doing this are easier recruitment, cheaper rent, closer alignment with the Secretary’s policy initiatives, and getting the agency closer to stakeholders.  None of these bears up under even the most casual scrutiny. So what is this really about? I’m guess that this is about getting political control over—silencing—an agency that conducts independent, unbiased, nonpartisan research that risks leading to inconvenient truths. Here, for example, are some recent publications [with my comments].

• Participation in USDA’s Supplemental Nutrition Assistance Program (SNAP): Effect of Local Labor Market Conditions in Oregon: Local labor market conditions were strongly linked to the probability of leaving SNAP… When labor market conditions were measured at the local level, researchers estimated a greater responsiveness by SNAP recipients to labor market conditions, as opposed to what other researchers have measured at the State or national levels [if there aren’t jobs available, people will need to stay on SNAP].
• Household Food Security in the United States in 2017: An estimated 11.8 percent of American households were food insecure at least some time during the year in 2017, lacking access to enough food for an active, healthy life for all their members. That is down from 12.3 percent in 2016 [but still high].
• High-Poverty Schools Are More Likely To Adopt the Community Eligibility Provision of the USDA School Meal Programs: The Community Eligibility Provision (CEP) allows eligible schools to offer USDA school meals to all students at no charge. During the 2016-17 school year, 47 percent of U.S. school districts eligible to use CEP used the provision in at least one school [less than half are giving kids free meals].

ERS is not broke and does not need fixing.  The proposal must be understood as an attempt to destroy the ERS.  Participants called for:

• Congressional hearings
• An independent cost-benefit analysis
• Delay further action until then or, better yet, a full stop

I am a big user of ERS data and a great admirer of the work of ERS economists. Other views on the webinar and this issue

• Agri-Pulse
• Center for Progressive Reform
• Politico (behind paywall, alas)

Additions, September 24

The groups that did the WEbinar have forwarded links:

• USDA Research Relocation and Reorganization: Perspectives from Former USDA Chief Scientists and Administrators (464 people registered).
• The recording is online .
• The slides are online.
• The webinar webpage is online.
• The sign-on letter—organizations only—to Congress is at www.tinyurl.com/ERS-Strong
• Another sign-on letter for former administration officials and heads of statistical agencies is at www.tinyurl.com/StrongERS4USDA.

These ask Congress to:

• retain the ERS in the national Capitol region;
• maintain and strengthen the integrity and independence of the ERS as a statistical agency; and
• keep the budget and personnel for the USDA Economic Research Service at least at FY 2018 levels.

Additions, September 25 (thanks to the Hagstrom Report

▪ USDA — Secretary Perdue response to Roberts and Stabenow
▪ American Statistical Association — Count on Stats
▪ — Fact Sheet
▪ — Friends of Agricultural Statistics and Analysis Sign-On Letter Opposing USDA re-organization and re-alignment of the Economic Research Service
▪ — USDA Economic Research Service Sign-On Letter – Former administration officials and statistical agency leaders
▪ — NIFA Relocation Letter to Congress
▪ Center for Progressive Reform — Draining Washington of Science and Talent