Spider Simulator
The search engines operate with the spider as an agent to pick up information about web sites and collect it in a repository where it is analyzed and indexed. This information is used as an input when someone using the search engine searches and brings back the relevant sites based on the search text. I had posted about web spiders and also a reference link to how one can build spiders. I stumbled upon Spider Simulator which displays how a spider will look at a web site and collect information. I gave my site and there were some Meta information missing, so I updated it today so that my site’s presence is appropriately visible to the search engine spiders 
Sankar Said,
January 8, 2007 @ 12:13 pm
Good find Ramesh ..
these spiders or bots as they are called scan the pages in certain periodic fashion & the period itself is determined by the rate of change of the page’s content….. a site like techmasala.com for instance which is updated regularly gets to be scanned by the bot at shorter intervals (than it was when the site begun) and no wonder you see your page’s ranking zoom to “2″ from nowhere a couple of months back .. ….Great Going
BTW haver you wondered how although the google bots index your page contents after a certain period , you still get to find the content of your page( say a new post) shown in Google alerts that very day ! - could it mean google uses something else other than bots & spiders for the sake of aggregation ( for content to be used in Google News & Alerts ?)
Sankar
Ramesh Said,
January 8, 2007 @ 9:16 pm
Hi Sankar,
Oh yeah there is lot more stuff that sucks information like crazy in Google. Sure it could involve aggregators.
Ramesh