http://www.robotstxt.org/orig.html says:
Disallow: /help disallows both /help.html and /help/index.html
Now, google.com/robots.txt lists:
Disallow: /search
Allow: /search/about
Upon running robotparser.py
, it returns false
for both the above cases in Googles robots.txt
.
Would somebody please explain me, what's the use of Allow
in Allow: /search/about
as it would return a false
based on the Disallow
entry above it?