Ecologists have just thrown a wrench into the pond of their discipline in the form of a manuscript currently published in the journal BMC Biology and posted online on October 4th. More than two hundred researchers analyzed the exact same data and came to different, sometimes even contradictory, conclusions.
In a series of 135 analyses, the majority concluded that the more chicks there are in a blue tit brood, the less each individual chick weighs. However, the strength of the relationship between the number of young animals and weight varies between one and ten. In another set of 81 analyses, one part found that vegetation cover on a plot affected the growth of eucalyptus trees, while another part found the opposite. Three-quarters believe grass has no impact on land use.
Is ecological research so fragile? “I have been interested in the reliability of the scientific literature in ecology and evolutionary biology for more than ten years,” explains Hannah Fraser, co-pilot of this unique experiment at the University of Melbourne (Australia). The variability in results is often attributed to the fact that environmental conditions are never exactly the same. However, we wanted to know to what extent individual decisions in data analysis can influence this variability. »
This may seem surprising, but when faced with a set of data, every researcher makes different decisions depending on what they find relevant. For example, it can only store certain observations. In the case of the “tits”, measurements were taken in the wild, but also others where chicks were added or removed by humans to study the development of the litter.
Then we need to consider the factors that are likely to influence the desired relationship between two variables. In the case of “Eucalyptus,” the distance of the plot to a mature tree that is likely to drop seeds is obviously important when assessing a tree’s chances of growth. Choosing these cofactors adds even more options.
“A surprising variability”
Finally, there is the choice between statistical methods. A linear relationship between two variables or another, more subtle one? A frequentist or Bayesian view (the name of two statistical schools)? Which software version to estimate the parameters? Still sources of variability.
To shed light on the situation, the authors proceeded methodically. Two datasets were proposed, “Tit” and “Eucalyptus,” each with a specific question that has never been studied before. Each “analyst” then became a critical reviewer of the others’ work, assigning a score from 0 to 100.
You still have 50% of this article left to read. The rest is reserved for subscribers.