See also:
- Computers: Internet: E-mail: Spam: Preventing (63)
- Computers: Internet: Searching: Search Engines: Sitemaps (11)
- Computers: Internet: Web Design and Development: Authoring: Online Tools: robots.txt (5)
- Computers: Software: Internet: Site Management: Log Analysis (210)
List of Robot Agent Strings
pgts.com.au
A list from PGTS of Web robots with the identifying data they leave in Web site logs.
All About Search Indexing Robots and Spiders
searchtools.com
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
Robotstxt.org
robotstxt.org
Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
HTTP User Agent Index
siteware.ch
An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
Search Engine Robots and Other User Agents
jafsoft.com
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
User-Agents.org
user-agents.org
Large list of search engine spiders, similar web robots, and Web browsers: their web-log identification and links to their originators.
Robot IP Address
briandunning.com
Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.
Bots vs Browsers
botsvsbrowsers.com
This large database lists user agents in categories and distinguishes between robots and browsers.
User Agent String
user-agent-string.info
Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.
ACAP - Automated Content Access Protocol
the-acap.org
Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.
Search Engine IP Addresses
iplists.com
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
|
[Hide]▼
news.google.com
Neoseeker
But for what ever the reason, a much larger, massive-er, crazier giant robot statue is going to be built, in South Korea. Planned to be an epic-sized, ...
[Hide]▲
