Invisible Web

While search engines and directories provide access to many Web sites, there is a lot of information on the Web that is not indexed by these tools. Search engines find Web sites to index by using Web robots or spiders to crawl the Web, following links from one Web page to another. However, much information is located in databases that require users to complete search forms to access the information and Web robots skip or ignore these search forms when crawling the Web. Because entries in directories are compiled by humans, they provide better access to the information stored deep in the Web, but there are not enough people to index all of the information available via the Web, so even directories do not provide access to a lot of this hidden information.

The information in the hidden Web is typically of very high quality and is invaluable to any researcher, so how does one find these resources if not by use of search engines or directories? The University of California at Berkeley’s Library provides a tutorial that will explain the invisible Web in more depth and provide solutions for accessing this information.

Read through the "Invisible Web: What it is, Why it exists, How to find it, and it’s inherent ambiguity" at http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html.

After finishing this primer, you should be familiar with finding any information you may need on the Web.  Continue to the Resources page for more articles and tools that can help you streamline your search experience.