Spider Simulator
The search engines operate with the spider as an agent to pick up information about web sites and collect it in a repository where it is analyzed and indexed. This information is used as an input when someone using the search engine searches and brings back the relevant sites based on the search text. I had posted about web spiders and also a reference link to how one can build spiders. I stumbled upon Spider Simulator which displays how a spider will look at a web site and collect information. I gave my site and there were some Meta information missing, so I updated it today so that my site’s presence is appropriately visible to the search engine spiders 
