Chasing the Tail

The Long Tail by Chris Anderson : An excellent article from Wired that demonstrates a few of the concepts and ideas I’ve been writing about recently. One such concept is well described by Clay Shirky’s excellent article Power Laws, Weblogs, and Inequality. A system governed by a power law distribution is essentially one where the power (whether it be measured in wealth, links, etc) is concentrated in a small population (when graphed, the rest of the population’s power values resemble a long tail). This concentration occurs spontaneously, and it is often strengthened because members of the system have an incentive to leverage their power to accrue more power.

In systems where many people are free to choose between many options, a small subset of the whole will get a disproportionate amount of traffic (or attention, or income), even if no members of the system actively work towards such an outcome. This has nothing to do with moral weakness, selling out, or any other psychological explanation. The very act of choosing, spread widely enough and freely enough, creates a power law distribution.

As such, this distribution manifests in all sorts of human endeavors, including economics (for the accumulation of wealth), language (for word frequency), weblogs (for traffic or number of inbound links), genetics (for gene expression), and, as discussed in the Wired article, entertainment media sales. Typically, the sales of music, movies, and books follow a power law distribution, with a small number of hit artists who garner the grand majority of the sales. The typical rule of thumb is that 20% of available artists get 80% of the sales.

Because of the expense of producing the physical product, and giving it a physical point of sale (shelf-space, movie theaters, etc…), this is bad news for the 80% of artists who get 20% of the sales. Their books, movies, and music eventually go out of print and are generally forgotten, while the successful artists’ works are continually reprinted and sold, building on their own success.

However, with the advent of the internet, this is beginning to change. Sales are still governed by the power law distribution, but the internet is removing the physical limitations of entertainment media.

An average movie theater will not show a film unless it can attract at least 1,500 people over a two-week run; that’s essentially the rent for a screen. An average record store needs to sell at least two copies of a CD per year to make it worth carrying; that’s the rent for a half inch of shelf space. And so on for DVD rental shops, videogame stores, booksellers, and newsstands.

In each case, retailers will carry only content that can generate sufficient demand to earn its keep. But each can pull only from a limited local population – perhaps a 10-mile radius for a typical movie theater, less than that for music and bookstores, and even less (just a mile or two) for video rental shops. It’s not enough for a great documentary to have a potential national audience of half a million; what matters is how many it has in the northern part of Rockville, Maryland, and among the mall shoppers of Walnut Creek, California.

The decentralized nature of the internet makes it a much better way to distribute entertainment media, as that documentary that has a potential national (heck, worldwide) audience of half a million people could likely succeed if distributed online. The infrastructure for films isn’t there yet, but it has been happening more in the digital music world, and even in a hybrid space like Amazon.com, which sells physical products, but in a non-local manner. With digital media, the cost of producing and distributing entertainment media goes way down, and thus even average artists can be considered successful, even if their sales don’t approach that of the biggest sellers.

The internet isn’t a broadcast medium; it is on-demand, driven by each individual’s personal needs. Diversity is the key, and as Shirkey’s article says: “Diversity plus freedom of choice creates inequality, and the greater the diversity, the more extreme the inequality.” With respect to weblogs (or more generally, websites), big sites are, well, bigger, but links and traffic aren’t the only metrics for success. Smaller websites are smaller in those terms, but are often more specialized, and thus they do better both in terms of connecting with their visitors (or customers) and in providing a more compelling value to their visitors. Larger sites, by virtue of their popularity, simply aren’t able to interact with visitors as effectively. This is assuming, of course, that the smaller sites do a good job. My site is very small (in terms of traffic and links), but not very specialized, so it has somewhat limited appeal. However, the parts of my site that get the most traffic are the ones that are specialized (such as the Christmas Movies page, or the Asimov Guide). I think part of the reason the blog has never really caught on is that I cover a very wide range of topics, thus diluting the potential specialized value of any single topic.

The same can be said for online music sales. They still conform to a power law distribution, but what we’re going to see is increasing sales of more diverse genres and bands. We’re in the process of switching from a system in which only the top 20% are considered profitable, to one where 99% are valuable. This seems somewhat counterintuitive for a few reasons:

The first is we forget that the 20 percent rule in the entertainment industry is about hits, not sales of any sort. We’re stuck in a hit-driven mindset – we think that if something isn’t a hit, it won’t make money and so won’t return the cost of its production. We assume, in other words, that only hits deserve to exist. But Vann-Adib�, like executives at iTunes, Amazon, and Netflix, has discovered that the “misses” usually make money, too. And because there are so many more of them, that money can add up quickly to a huge new market.

With no shelf space to pay for and, in the case of purely digital services like iTunes, no manufacturing costs and hardly any distribution fees, a miss sold is just another sale, with the same margins as a hit. A hit and a miss are on equal economic footing, both just entries in a database called up on demand, both equally worthy of being carried. Suddenly, popularity no longer has a monopoly on profitability.

The second reason for the wrong answer is that the industry has a poor sense of what people want. Indeed, we have a poor sense of what we want.

The need to figure out what people want out of a diverse pool of options is where self-organizing systems come into the picture. A good example is Amazon’s recommendations engine, and their ability to aggregate various customer inputs into useful correlations. Their “customers who bought this item also bought” lists (and the litany of variations on that theme), more often than not, provide a way to traverse the long tail. They encourage customer participation, allowing customers to write reviews, select lists, and so on, providing feedback loops that improve the quality of recommendations. Note that none of these features was designed to directly sell more items. The focus was on allowing an efficient system of collaborative feedback. Good recommendations are an emergent result of that system. Similar features are available in the online music services, and the Wired article notes:

For instance, the front screen of Rhapsody features Britney Spears, unsurprisingly. Next to the listings of her work is a box of “similar artists.” Among them is Pink. If you click on that and are pleased with what you hear, you may do the same for Pink’s similar artists, which include No Doubt. And on No Doubt’s page, the list includes a few “followers” and “influencers,” the last of which includes the Selecter, a 1980s ska band from Coventry, England. In three clicks, Rhapsody may have enticed a Britney Spears fan to try an album that can hardly be found in a record store.

Obviously, these systems aren’t perfect. As I’ve mentioned before, a considerable amount of work needs to be done with respect to the aggregation and correlation aspects of these systems. Amazon and the online music services have a good start, and weblogs are trailing along behind them a bit, but the nature of self-organizing systems dictates that you don’t get a perfect solution to start, but rather a steadily improving system. What’s becoming clear, though, is that the little guys are (collectively speaking) just as important as the juggernauts, and that’s why I’m not particularly upset that my blog won’t be wildly popular anytime soon.