Search engine databases are selected and built by computer robot programs called spiders. Although it is said they “crawl” the web in their hunt for pages to include, in truth they stay in one place. They find the pages for potential inclusion by following the links in the pages they already have in their database (i.e., already “know about”). They cannot think or type a URL or use judgment to “decide” to go look something up and see what’s on the web about it. (Computers are getting more sophisticated all the time, but they are still brainless.)

If a web page is never linked to any other page, search engine spiders cannot find it. The only way a brand new page – one that no other page has ever linked to – can get into a search engine is for its URL to be sent by some human to the search engine companies as a request that the new page be included. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for “indexing”. This program identifies the text, links, and other content in the page and stores it in the search engine database’s files so that the database can be searched by keyword and whatever more advanced approaches are offered, and the page will be found if your search matches its content.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Pages that are excluded are referred to as the “Invisible Web” – what you don’t see in search engine results. The Invisible Web is estimated to be two to three times bigger than the visible web.

Copyright (C) 2005 by the Regents of the University of California