In robots.txt, why would there be "Allow: /*" below a Disallow rule?

by Md Shamsher Khan   Last Updated August 02, 2018 17:04 PM

I want to allow full access to all Google

I am specially confused about Allow: /* below Disallow:

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google
Disallow:


Answers 2


The Allow directive is a non-standard extension to robots.txt. The original robots.txt standard only includes Disallow directives. Google's robots do understand Allow directives, so it would be OK to use them in sections targeted at a Google robot. The Allow directive should only be used when it is more specific than Disallow directive. This will counteract the Disallow. It can be used to give finer grained allowance (maybe just a single page) of entire section that would otherwise be disallowed.

In your case, the Allow directive is saying the same thing as the Disallow directive, so it should be omitted.

You say you that you want to allow all Google robots to crawl all of your site. If that is the case, you would need a rule for the main Googlebot as well. I would suggest using this in your robots.txt

# Google Image
User-agent: Googlebot-Image
Disallow:

# Google AdSense
User-agent: Mediapartners-Google
Disallow:

# Googlebot
User-agent: Googlebot
Disallow:
Stephen Ostermiller
Stephen Ostermiller
October 05, 2013 10:47 AM

Google Image User-agent: Googlebot-Image Disallow: Allow: /*

Google AdSense

User-agent: Mediapartners-Google Disallow:

Angel
Angel
August 02, 2018 16:19 PM

Related Questions



robots.txt for website hosted in a subdirectory

Updated February 16, 2017 14:04 PM

Blocking vs noindex to reduce crawl requests

Updated December 09, 2016 08:01 AM