# http://www.robotstxt.org/wc/norobots.html # http://www.google.com/bot.html ## http://crawler.archive.org/ User-agent: archive.org_bot Disallow: / User-agent: * Disallow: /fpapp/ Disallow: /fpapp_beta/ Disallow: /fphistory/ Disallow: /fphistory3/ Disallow: /fphistory3b/ Disallow: /fphistory4/ # Disallow: /cgi-bin/