The Age of Superfluous Information — Revisited

The Economist’s article, “Too much information: How to cope with data overload,” deals with information overload. (Hat tip Prasanna Viswanathan @prasannavishy for the link.) For a few years I have been concerned about it since I have a very low threshold for information. In 2005, I pondered the matter in a number of blog posts. I realize the irony in writing yet another blog post on information overload, but there you have it. The Economist article underlines my fears.

Allow me to quote from those posts. “The Age of Superfluous Information” (Oct 2005) —

“There is no more dangerous mistake than the mistake of supposing that we cannot have too much of a good thing.” Thus spake George Bernard Shaw. Excess is as damaging as shortage in most things that are considered good. More is better but only up to a point of satiation. Beyond the satiation point, the marginal utility of a good is negative, as an economist may put it. Particular instances of that generalization are not hard to find.

Food, for instance, is a good that in excessive quantities is a bad as the success of the dieting industry so starkly demonstrates. Yet tens of millions poor people around the world dying of malnutrition and starvation every year is the horrible demonstration of the problem at the other extreme.

The same holds for information.

What brings all this to mind is the so-called information revolution occurring globally. Information is a good which is also subject to the law of negative marginal utility beyond the satiation point. Information overload can be as debilitating as too little. For much of the world now the problem is no longer a shortage of information but rather a surfeit.

All living beings acquire information from their surroundings for survival. Information needs vary depending on the nature of the being and the sense-organs have co-evolved to perform the required function. The more complex the entity, the more sophisticated the sense organs are. The sophistication has two dimensions: the bandwidth and the filter.

Higher bandwidth means that more information per unit of time is sensed by the entity. But the information gathered is filtered severely for relevance and only a small percentage is passed on to the processing unit (the brain) for internalization and for response. Therefore the filtering mechanism had to evolve in step with the evolution of high bandwidth sense organs.

Here is my conjecture. Higher intelligence is marked by possessing very high bandwidth sense organs (channel capacity), and a very sophisticated filtering system which rejects most of the inputs and processes only a very small portion of the total information received

We humans sense the world around us mainly through our eyes and ears. Most of the megabits per second of information we receive is rejected and only a minute fraction is processed by our brains. The filtering mechanisms developed over our evolutionary history and did so gradually. Even the brain has an inbuilt capacity to forget. There are a few who suffer from a pathological condition which does not allow them to forget anything that they have ever seen or heard.

Now we are faced with a world where information is being generated and accumulated at an exponentially increasing rate and we face the possibility of information overload that could overwhelm our capacity to filter and meaningfully process it.
. . .
I conjecture that the age of information dieting industry is upon us. The day is not far off when you would have to go on an information diet.

In a follow up post, I addressed the issue of sorting and searching the large stock of public information and claimed that “in post-industrial world and increasingly so in the future, sorting and searching through information will occupy the role that manufacturing did in the growth of the old economy.”

Sorting and searching through information are uniquely human activities because only humans have an external store of information which needs to be accessed and acted upon. The notion of acting on information stored externally is not associated with non-human animals.

The larger the stock of information, the more expensive it is to search through it to locate the precise bit that is relevant at any particular instance. To make the task of searching more tractable, ordering the information in some fashion—called sorting—becomes paramount. Computer scientists have worked on the problem of sorting and searching for decades with phenomenally successful advancement in our understanding in this regard.
. . .
When the quantity supplied of any good is in excess of anything reasonably required or demanded, the variable of importance is quality. Basically a person’s information needs are really very simple. One can only read so much, listen to so much talk and music, watch so much video, and wish to know only so much about what is going on around in one’s neighborhood and in the world at large.
. . .
The bottom line is this: there is already so much information out there that even if no additional information were generated, each one of us could be occupied a little longer than forever to finish it. Information, as we well know, is a non-rival good. That is, my “consuming” a particular piece of information will not diminish the amount available to you. Compare this to a rival good such as food. Stock of food is enough to last the six billion humans for about 3 months. In other words, if we produce no additional food, all together humans would finish the stock in three months. Or, a single human can therefore finish this in 1.5 billion years. But it is not so in the case of information. Each of us would take the estimated 18 billion years to finish the information we already have before we ask for more.

Clearly, for an average human, about 0.00000000001 percent of the total information stock is more than enough. About 99.9999999999 percent of the available information is worthless. So how does one go about searching out the teaspoonful of useful information from the oceans of available information. That is the challenge and therein lie the opportunities. That is why firms like Google will make the big bucks. The opportunity is not so much in making information available but making the right information available.

Which brings me to the point which I started off with. Searching is only part of the story when it comes to information. The other part is sorting. If one can sort the information along some relevant dimension, then you have meaningful information. What is meaningful can only be defined in the context of the entity processing the information. From the same stock and flow of information, different entities define different subsets that are relevant and meaningful. This subset can be labeled private information as opposed to the vast store of public information. Private information is the top of the sorted list of public information. Internalizing the private information leads to what we can call a stock of knowledge associated with the individual.

It is useful at this point to remind ourselves of the distinction between information and knowledge. Information is a public good the stock of which is growing exponentially. Knowledge is a private good and its primary raw material is the private information which is a very vanishingly small subset of the available public information. Even though public information has no known bounds, there are limits to how much private information can processed by a human brain and thus there are limits to the acquisition of the private good we call knowledge.

Conflating knowledge and information is distressingly too common these days and so I would like to dwell on this distinction for a bit. Some say that today we have a knowledge economy. It is trivially true because it has always been a knowledge economy ever since humans evolved brains capable of processing information into knowledge and began using knowledge to organize and coordinate economic activities. What is novel is the unfathomably huge stock of information we have available today. What distinguishes one individual from another today is the capacity to figure out what is relevant information and to internalize it efficiently into knowledge. That capacity is one of the basic skills imparted by what we call education.

To summarize the story so far: from the vantage point of an individual, this is an age of superfluous information; only a tiny fraction is relevant and meaningful; searching through the information can be automated but efficiently sorting for relevance is a private skill; imparting that skill is a primary function of education.

And now a bit from the Economist article linked above.

[Researchers of the dense data fog] raise three big worries. First, information overload can make people feel anxious and powerless: scientists have discovered that multitaskers produce more stress hormones. Second, overload can reduce creativity. Teresa Amabile of Harvard Business School has spent more than a decade studying the work habits of 238 people, collecting a total of 12,000 diary entries between them. She finds that focus and creativity are connected. People are more likely to be creative if they are allowed to focus on something for some time without interruptions. If constantly interrupted or forced to attend meetings, they are less likely to be creative. Third, overload can also make workers less productive. David Meyer, of the University of Michigan, has shown that people who complete certain tasks in parallel take much longer and make many more errors than people who complete the same tasks in sequence.

Curbing the cacophony

What can be done about information overload? One answer is technological: rely on the people who created the fog to invent filters that will clean it up. Xerox promises to restore “information sanity” by developing better filtering and managing devices. Google is trying to improve its online searches by taking into account more personal information. (Some people fret that this will breach their privacy, but it will probably deliver quicker, more accurate searches.) A popular computer program called “Freedom” disconnects you from the web at preset times.

OK, I have added to your information overload. Sorry.

[I even have a category called “Information Overload.” How cool is that!]

2 thoughts on “The Age of Superfluous Information — Revisited

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s