Search Engine Indexing

Google and others are indexing more pages than ever. Many of the modern search
engines now boast upward of a few billion indexed documents. Here is how Wikipedia
describes the process of search engine indexing:
Search engine indexing collects, parses, and stores data to facilitate fast and accurate
information retrieval. Index design incorporates interdisciplinary concepts from linguistics,
cognitive psychology, mathematics, informatics, physics and computer science. An
alternate name for the process in the context of search engines designed to find web pages
on the Internet is Web indexing.
Although web spiders do their best to obtain as many documents as possible for search
engines to index, not all documents get indexed. Search engine indexing is closely tied
to the associated search engine algorithms. The search engine indexing formula is a
highly secretive and intriguing concept to most SEO  enthusiasts.
Search engine indexing is not an entirely automated process.