 |
Search Engine Marketing
How Google Spiders The Web
November 2002
First and foremost, this isn’t a definitive article on how Google lists websites when you search. Google has over 100 criteria, on and off the page, for determining how a page ranks. Rather, this article is more a compilation of Q&A’s about how and when Google spiders the web, and what it means to your SERPs (Search Engine Results Pages, which are the results you get when you type in a query in a search engine).
Q.
|
When does Google spider the Web?
|
|
A.
|
Google utilizes robots on a monthly crawl of the entire Web to update the Google database, which currently indexes about 3 billion web pages. The spidering typically takes place every 30 days or so and usually begins during the first week of the month, though the time varies for different websites.
Then, also once a month, Google updates its index based on what its robots found during the crawl. This update often begins during the third week of the month and takes about a week.
Google now also sends out a "fresh bot" on a daily basis. It appears this spider finds updated pages that are already in the Google index, then includes those pages in SERPS for a brief period. Soon, those pages can disappear from the rankings until the next update.
|
Q.
|
How does Google find pages during the spider?
|
|
A.
|
One way to include your pages in Google is through the Add URL feature at google.com/addurl.html, though many question the effectiveness of that method. The best way to be found is via a link or links from web pages that are already included in Google’s index. During its crawl, the Google Bot will find your site via that link or links.
|
Q.
|
What does a date by my search result listing mean?
|
|
A.
|
It means that page has a "fresh" listing as a result of the "fresh bot" discussed above. Again, these results tend to last only a day or so, then tend to disappear until the next full updating of the Google index.
|
Q.
|
What is the "Google Dance?"
|
|
A.
|
Right before the monthly Google update becomes visible to casual Web surfers, a change becomes visible to Google-trekkies who closely follow this phenomenon. Google test servers begin to show different results than the main search URL (sometimes radically different) for the same query. The fluctuation of the results has become the Google Dance.
|
|
