s the web grows larger and the tools we use begin to fail, there comes a need for more advanced internet research expertise. This is a growth industry.
I just finished a delightful article that I think will be published in the library periodical ONLINE. It was a piece tracking the evolution of research techniques, showing that internet research is in for a shake-up. I want to explore one of the themes that emerged in a little more detail.
The internet is growing. As it grows, the percentage of the web indexed by the giant global search engines, falls. That is, the coverage and opportunity to directly search for information is being reduced.
Search engines have been dealing with this dilemma for several years, in several ways.
1) Search engine databases have grown in size. Google now rests just under 2.5 billion records.
2) Popularity has been co-opted as a measure of value. This popularity extends to the process of selecting pages to index. Pages that are frequently linked are more likely to be indexed.
3) I've heard search engines selection criteria is being refined to place more emphasis on resources with other hallmarks of value - like inbound links from a selection of respected websites.
4) I think I detect a growing emphasis on recently crafted webpages. Search engines drop or index less frequently the pages that don't change.
5) I also detect a wider distribution with fewer pages from more domains. This leads to a thinning of coverage but casting the net wider.
I am sure there are other clever approaches to extend an index. But these approaches are not sufficient in themselves to dispel the effect of growth. There is a tremendous amount of information in the world, merging from different publishers, geographical regions, and purposes. I like to draw attention to how the number of people capable of publishing webpages grows exponentially and will continue for a few years yet. I've another article called "10 Billion or more" that explores this concept.
There are to be a great many webpages on the internet. If we accept this, we can consider how it affects the way we find information now.
A year ago I pushed the need to use punctuation when you search the internet. Quotes are particularly important but field searching and Boolean are also significant. We are in a transition. Future information research will be more challenging. I can taste it in some of my searches now.
Take a search for personal details. I wanted a phone number to a lecturer I met a year ago who works at the University of Technology, Sydney (UTS). Google did not list it. A search for her name in quotes did dig up a single page about her work but the page did not have any contact details. A visit to the UTS website seemed the next obvious step. We'll come back to why it was obvious. On the website there was a search function - that defaulted to an Inktomi search restricted to the UTS website. This is clever, since I recall Inktomi does a deep search of Australia (or at least once did), so may have indexed more UTS pages than Google. Again this search came up empty. But wait… Wouldn't UTS have a staff directory? Yes it does. Her phone number was listed there.
The Spire Project - better ways to find information.