Stop your https pages from being indexed in Google

How to create a dynamic robots.txt file with PHP and Apache Mod Rewrite to stop your secure https ssl pages from being indexed in Google.

(1.) Open up your existing robots.txt file and then create the following robots.php file

<?php if  ( $_SERVER['HTTPS'] ){ ?>
User-agent: *
Disallow: /

<?php }else{ ?>
# put the contents of your
# existing robots.txt file here
# ie...
# User-Agent: *
# Allow: /

<?php } ?>

(2.) Open your .htaccess file and add the following lines to it

<IfModule mod_rewrite.c>

	RewriteEngine on

	# uncomment this if needed
	# RewriteBase /

	# redirect robots.txt to robots.php
	RewriteCond %{REQUEST_URI} ^(/robots.txt)$
	RewriteRule ^(.*)$ robots.php [QSA,L]

</IfModule>

Now everytime googlebot follows a https link it will get a robots.txt file that tells it not to index those pages.

You can test that this is working by checking the difference between the normal robots.txt and the https robots.txt.

Example:

http://www.acidfanatic.com/robots.txt
https://www.acidfanatic.com/robots.txt