Sometimes we want to exclude a page from search engines. For example, repetitive pages that might lead to page rank penalties.
If you want to exclude pages with a specific name from several locations in your site – for example you might have a comments.php or a help.php in multiple parts of your site, and you want to hide it from search engines, you need to modify robots.txt.
Here is how you do it:
1) Create or edit robots.txt in the root of your site
2) Add these lines to robots.txt
Where comments.php is the name of the page and the * (in the Disallow: */comments.php) indicates every path.
This will exclude:
Important tip, you can use Google web master tools to verify your robots.txt