How to: Adding wildcard pages to robots.txt

Sometimes we want to exclude a page from search engines. For example, repetitive pages that might lead to page rank penalties.

If you want to exclude pages with a specific name from several locations in your site – for example you might have a comments.php or a help.php in multiple parts of your site, and you want to hide it from search engines, you need to modify robots.txt.

Here is how you do it:
1) Create or edit robots.txt in the root of your site

2) Add these lines to robots.txt

User-agent: *
Disallow: */comments.php

Where comments.php is the name of the page and the * (in the Disallow: */comments.php) indicates every path.

This will exclude:
Yoursite.com/comments.php
But also
Yoursite.com/any_path/comments.php

Important tip, you can use Google web master tools to verify your robots.txt

Share the love...Tweet about this on TwitterShare on LinkedInShare on Google+Share on Facebook

Amir Shevat

Amir Shevat is the global Startup Outreach lead in Google Developer Relations (g.co/launch). Previously, Amir Led Google Campus Tel Aviv and was the co founder of several startups.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *