A MISPLACED RELIANCE ON STATISTICAL ANALYSIS:

A recent trend in rock art studies is the partial acceptance of rock art as a legitimate subject for study by academics and professionals. Much of this is made possible by the application of statistical analysis of rock art which lends an air of scientific respectability to what was before a somewhat messy field of study. This is because the statistical analysis provides quantifiable data that can be pointed to as scientifically verifiable objective fact. I have personally never been completely comfortable with this type of rock art study but I had always put it down to the fact that, as an art historian, statistical analysis is seriously foreign territory, something I am not conversant with. My feeling has been that once we have finished the elaborate statistical analysis of rock art elements, and now know which elements are closer to which other elements, what have we really learned? We could have pretty much determined that by simple observation, and learned just as little by doing so.

On July 12, 2010, I published a posting titled The Writing On The (Cave) Wall concerning a recent example of this sort of study which was related in a February 17, 2010, article by Kate Ravilious in New Scientist magazine entitled “The Writing On The Cave Wall” which made the ambitious claim that writing had been discovered on the walls of the painted European caves. According to this report a pair of scholars at the University of Victoria, British Columbia, Canada, graduate student Genevieve von Petzinger and her supervisor April Nowell undertook as a Master’s project a numerical analysis of all the signs found in 146 sites in France covering a date range from 37,000 to 12,000 B.P. The signs were compiled in a database for analysis. They found that 26 of these signs appeared frequently in numerous sites. The most common sign was a line that was found at 70 percent of the sites and across all time periods. The next most common symbols were what they called “open angles” and dots being found at 42 percent of the sites. Having found quantifiable examples of common occurrence of these symbols in conjunction with each other they made the amazing announcement that they represent a form of written communication, and astoundingly the scientific community, including anthropology and archaeology, seems to accept this statement. Why would scientists fall for this? Because, Petzinger and Nowell got their results from a statistical analysis of numbers in a database.

An essay by Tom Siegfried in the March 27, 2010, issue of Science News titled "Odds Are, It’s Wrong: Science Fails to Face the Shortcomings of Statistics", addresses the question of accuracy and truth in science based upon statistical analysis. In a detailed explanation, and with a number of interesting examples, Siegfried points out that all scientific results that have been reached through a statistical analysis are literally only possible explanations, not proven truth. One example he cites is the 2007 meta-analysis published in the New England Journal of Medicine, which attributed an increased risk of heart of attack to users of the diabetes drug Avandia. Siegfried stated that “raw data from the combined trials showed that only 55 people in 10,000 had heart attacks when using Avandia, compared with 59 people per 10,000 in comparison groups. But after a series of statistical manipulations, Avandia appeared to confer an increased risk.” We have since seen much about this in the news including recent hearings by an FDA panel on the subject.

Now, as I stated above, I am not a statistician, and am not personally qualified to pass judgment on statistical work done by others. However, if I was interested in doing statistical analysis of rock art I would certainly read Siegfried’s essay before I tied my reputation to the results.

References:

Siegfried, Tom
2010     Odds Are, It’s Wrong: Science Fails to Face the Shortcomings of Statistics, Science News,
             Vol. 177, No. 7, (26-29). (March 27)

http://www.newscientist.com/article/mg20527481.200-the-writing-on-the-cave-wall.html?full=true