The Parts Of A Search Engine
- Search engines have three major elements. First is the spider, also called the crawler.
The spider visits a web page, reads it, and then follows links to other pages within the
site. This is what it means when someone refers to a site being "spidered" or "crawled."
The spider returns to the site on a regular basis, such as every month or two, to look for
changes.
- Everything the spider finds goes into the second part of a search engine, the
index. The index, sometimes called the catalog, is like a giant book containing a copy of
every web page that the spider finds. If a web page changes, then this book is updated new
information.
- Sometimes it can take a while for new pages or changes that the spider finds to be added
to the index. Thus, a web page may have been "spidered" but not yet "indexed." Until it is
indexed -- added to the index -- it is not available to those searching with the search
engine.
- Search engine software is the third part of a search engine. This is the program that sifts
through the millions of pages recorded in the index to find matches to a search and rank
them in order of what it believes is most relevant.