Research Commentary by David Novak of The Spire Project

The Growth of the Internet Should Still Scare You.
By David Novak

Librarians among us wax romantically about a time when they knew all the prominent publications in their field - and certainly in their collection. Researchers can think back to a time they knew most of the important relevant research. Both times are passing. In the near future we will look back at today as the good old days of the internet.

I am concerned about our attitude to the growing quantity of information on the Internet.

The complete UN archival material is destined for the internet in the next eighteen months or so. All the primary search tools: UNCAPS, UNBISnet, UN Press Release Database (and UNIONS?) are already online. For financial reasons, government transparency and utility, all the 300+ shelves of past and current UN archival material will migrate to the net.

One resource - one publisher - one truckload of information.

Just how much information will the internet hold? Most everyone I've read that looks at this topic tends to wax into ever-grander sounding marketing slogans, but lets try again. There are several trends working to increase the quantity of information online.

1) Geographic boundaries that previously separated similar information are breaking down.

2) The number of people capable of publishing will grow exponentially for several years to come.

3) Vast quantities of information will continue to migrate onto the internet for at least another five years.

4) Internet information tends to stick around. There is no attrition rate as with newspapers.

5) And information duplication is at work: our search difficulties feed the creation of essentially equivalent information by other authors.

If this logic is not enough, we can also look at the trends in the growth of the internet. Or rather, let us miss this step. If I had the statistics I would certainly wax romantically about earlier days. Let us merely state the quantity of information is growing, fast.

Here is my predication: in 24 months, the internet will be 8 times its current size, the global search engines will index one in 50 documents, and most quality information will have less than 10 in-bound links.

Shift Gears:
So the internet is growing. What does this say about searching, research, and helping patrons find information?

The first item to register is there is certainly room for information on the internet. Like it or not, if your collection of in-house information includes an internet connection, you have a very vast range of resources at your finger tips. Ignoring a portion of your collection may be your salvation, but the information is there - if you want to dig it out.

The second item is a gradual evolution of search technique and search assistance. In a simple example, making lists has become passe. While it may appear useful, it does not help others. The Library of Congress is not organized with little lists of resources. Indeed, lists are conspicuously absent.

The list issue harkens back to the past - oh four years back - when hotlists and search guides were a brilliant place to start a search. Today, if you are not creating a definitive list - as part of a major meta- project (I'm thinking Argus, About.com, AlphaSearch, & W3VL) then don't bother. Others are doing a better job and your chatting is merely adding to the volume in the cafe.

The third item is to dispel the myth that the internet needs organizing. In the next 5 years, every state and nation in the world will undertake a major study of childcare reform. Absolutely every one of these documents will be placed on the internet. Do we really want to spend time cataloguing these documents? A list of 200+ childcare reform documents? Get real.

We don't have any good organization for newspaper articles. What about the internet screams at us a need for organization?

And this is where we need to end this story. Organizing the internet is not a problem that will be solved. It only looks that way to people unaware that babies put on a lot of weight fast. The truth is that despite much effort, much of the internet will forever be un-catalogued and not indexed. The volume, trends and logic has made this a predestined certainty. We will only ever arrange a fractional (and diminishing) percentage of the internet.

But then, perhaps just a little organization always was enough.

Two conclusions to consider: Firstly, we need to be very careful in our attempts to organize the internet. Do we have a clear idea of how to scale our projects. (Are we planning progressively tighter focus?) Secondly, growth reaffirms the importance of continuously polishing our internet search skills.
* * *
David Novak manages The Spire Project, an Internet research resource and thinktank.

The Spire Project - better ways to find information.
Like this? You should attend our public seminar and receive our bi-monthly update notice.
| SpireProject.com | SpireProject.co.uk | Project Background | Feedback. Copyright©David Novak 2002.