Some web servers send a site navigation page or other response page with a “HTTP 200 OK” response instead of a “HTTP 404 Not Found” result for page-not-found conditions. To check on web server handling of page-not-found conditions, Slurp will occasionally send deliberately odd URLs built from random words to sites from which no 404 results have been seen.
http://help.yahoo.com/help/us/ysearch/slurp/slurp-10.html
I’m keeping this in mind when/if I finally start coding LiVE [the Link Integrity Verification Engine]. No more links leading to porn spam, thanks…
Leave a Reply