Anecdotal
I’m currently reading Cognitive Surplus: Creativity and Generosity in a Connected Age, by Clay Shirky. There seems to be a pattern emerging from certain pop-science books I’ve been reading in the past few years. Namely, a heavy reliance on fascinating anecdotes, counter-intuitive psychology experiments, and maybe a little behavioral economics thrown in for good measure. Cognitive Surplus most certainly fits the mold. Another book I’ve read recently, How We Decide by Jonah Lehrer, also fits. Most of Malcolm Gladwell’s work does too (indeed, he’s a master of the anecdote).
I don’t think there’s anything inherently wrong with this format. In fact, it can be quite entertaining and sometimes even informative. But sometimes I feel a bit uncomfortable with the conclusions that are drawn from all of this. Anecdotes, even well documented anecdotes, can make for great reading, but that doesn’t necessarily make them broadly applicable. Generalizing or extrapolating from anecdotes can lead to some problematic conclusions. This is a difficult subject to tackle though, because humans seem to be hard wired to do exactly that. The human brain is basically a giant heuristic machine.
This is not a bad thing. Heuristics are an important part of human life because we usually don’t always have all the information needed to use a more reliable, logical process. We all extrapolate from our own experiences; that is to say, we rely on anecdotal evidence in our daily lives all the time. It allows us to operate in situations which we do not understand.
Unfortunately, it’s also subjective and not entirely reliable. The major issue is that it’s rather easy to convince yourself that you have properly understand the problem, when in fact, you don’t. In other words, our incompetence masks our ability to recognize our incompetence. As a result, we see things like Cargo Cults. Security beliefs and superstitions are also heuristics, albeit generally false ones. But they arise because producing such explanations are a necessary part of our life. We cannot explain everything we see, and since we often need to act on what we see, we must rely on less than perfect heuristics and processes.
So in a book like Cognitive Surplus, there’s this instinctual impulse to agree with conclusions extrapolated from anecdotes, which is probably the source of my discomfort. It’s not that I doubt the factual content of the anecdotes, it’s that I’m not always sure how to connect the anecdote with the conclusion. In many cases, it seems like an intuitive leap, but as previously noted, this is a subjective process.
Of course, Shirky does not rely solely on anecdotal evidence in his book (nor do the other authors mentioned above). There are the aforementioned psychology experiments and behavioral economics studies that rely on the scientific notions of strictly controlled conditions and independent reproduction. The assumption is that conclusions extrapolated from this more scientific data are more reliable. But is it possible that they could suffer from the same problems as anecdotes?
Maybe. The data is almost always presented in an informal, summarized format (very similar, in fact, to the way anecdotes are formed), which can leave a lot of wiggle room. For instance, strictly controlled conditions necessary to run an experiment can yield qualifying factors that will make the results less broadly applicable than we may desire. I find this less troubling in cases where I’m already familiar with a study, such as the Ultimatum Game. It also helps that such a study has been independently reproduced countless times since it first appeared, and that many subsequent tests have refined various conditions and variables to see how the results would come out (and they all point in the expected direction).
Later in the book, Shirky references an economic study performed on 10 day-care centers in Haifa, Israel. I will not get into the details of the study (this post is not a review of Shirky’s book, after all), except to say that it was a single study, performed in a narrow location, with a relatively small data set. I don’t doubt the objective results, but unlike the Ultimatum Game, this study does not seem to have a long history of reproduction, nor did the researchers conduct obvious follow-up experiments (perhaps there are additional studies, but they are not referenced by Shirky). The results seem to violate certain economic assumptions we’re all familiar with, but they are also somewhat intuitive when you realize why the results came out the way they did. On the other hand, how do we know why they came out that way? I’m virtually certain that if you vary one particular variable of the experiment, you’ll receive the expected result. Then what?
I don’t mean to imply that these books are worthless or that they don’t contain valuable insights. I generally find them entertaining, helpful and informative, sometimes even persuasive. I like reading them. However, reading a book like this is not a passive activity. It’s a dialogue. In other words, I don’t think that Cognitive Surplus is the last word on the subjects that Shirky is writing about, despite a certain triumphal tone in his writing. It’s important to recognize that there is probably more to this book than what is on the page. That’s why there’s a lengthly Notes section with references to numerous papers and studies for further reading and clarification. Cognitive Surplus raises some interesting questions and it proposes some interesting answers, but it’s not the end of the conversation.
Update: I thought of a few books that I think are better about this sort of thing, and there’s a commonality that’s somewhat instructive. One example is The Paradox of Choice: Why More Is Less, by Barry Schwartz. Another is Flow: The Psychology of Optimal Experience by Mihaly Csikszentmihalyi. The interesting thing about both of these books is that they are written by researchers who have conducted a lot of the research themselves. Both of them are very careful in the way they phrase their conclusions, making sure to point out qualifying factors, etc… Shirky, Gladwell, etc… seem to be summarizing the work of others. This is also valuable, in its own way, but perhaps less conclusive? (Then again, correlation does not necessarily mean causation. This update basically amounts to heuristic, and one based on the relatively small sample of pop-science books I’ve read, so take it with a grain of salt.)
Again Update: I wrote this post before finishing Cognitive Surplus. I’m now finished, and in the last chapter, Shirky notes (pages 191-192):
The opportunity we collectively share, though, is much larger than even a book’s worth of examples can express, because those examples, and especially the ones that involve significant cultural disruption, could turn out to be special cases. As with previous revolutions driven by technology – whether it is the rise of literate and scientific culture with the spread of the printing press or the economic and social globalization that followed the invention of the telegraph – what matters now is not the new capabilities that we have, but how we turn those capabilities, both technical and social, into opportunities.
In short, I think Shirky is acknowledging what was making me uncomfortable throughout the book: anecdotes and examples can’t paint the whole picture. Shirky’s book is not internet triumphalism, but a call to action. I suppose you could argue that even the assertion that these opportunities exist at all is a form of triumphalism, but I don’t think so.
