What robot.txt should I use if I DON'T want a post indexed?


#1

I am just wondering how I would go about not having a post indexed or seen on any of the search engines, not just Google but Yahoo, Bing, etc. Even the images. I want the post to be like a private post or a private page.

Is it better to set it up in the custom txt box or in the head of the template of the blog or just in the custom robot tags in the box on the side of blogger?


#2

You can use this example in your robots .txt to block your whole site or part of it. A good web development company will always make sure it has 100% control.

User-agent: * Disallow: /ajax/ Disallow: /album.php Disallow: /checkpoint/ Disallow: /contact_importer/ Disallow: /feeds/ Disallow: /file_download.php Disallow: /hashtag/ Disallow: /l.php Disallow: /live/ Disallow: /moments_app/ Disallow: /p.php Disallow: /photo.php Disallow: /photos.php Disallow: /sharer/

Also if you want to specifically blot a bot then you have to use something like this

User-agent: Applebot Disallow: /ajax/ Disallow: /album.php Disallow: /checkpoint/ Disallow: /contact_importer/ Disallow: /feeds/ Disallow: /file_download.php Disallow: /hashtag/ Disallow: /l.php Disallow: /live/ Disallow: /moments_app/ Disallow: /p.php Disallow: /photo.php Disallow: /photos.php Disallow: /sharer/