Custom Robots Txt for new blog


#1

Hi,

I have started a new blog on blogger platform and so far I have 10 posts on it. I would like to enable custom Robots Txt for my blog. I have not created sitemap yet. I have searched Google and have found a method which is different from MBT’s robots.txt file

MBT’s Robots.txt is here [http://prntscr.com/6zpl0j]

BUT I Found in Google that is here [http://prntscr.com/6zplmt]

Which one should I follow?

Regards,


#2

Hello @ahmeddcc159 you should only use pre-installed robots.txt file it is perfect you can see it in blogger >>Setting >> Search Preferences >>Crawlers and indexing


#3

You should not use any robots.txt file. The one which blogger automatically creates is just perfect. However, if you want to disallow some pages to index on Google, then you can easily do that by going to Blogger >> Settings >> Search Preferences >> Crawlers and Indexing and then, select yes option behalf custom robots.txt and then, paste bellow code in it:

User-agent: Mediapartners-Google Disallow:

User-agent: *

Disallow: /search

Disallow: [the page url to wanna hide from Google Bots (without your domain. Like, /p/contact-us)]

Allow: /

Sitemap: http://[[YOUR_BLOG_URL]].blogspot.com/sitemap.xml

-Shivansh Verma Skv


#4

You can use robots.txt like this:

User-agent: Mediapartners-Google

Disallow: /search?updated-max

Allow:/

User-agent: Googlebot-News

Disallow: /search?updated-max

Allow:/

User-agent: Googlebot-Image

Disallow: /search?updated-max

Allow:/

User-agent: Googlebot-Video

Disallow: /search?updated-max

Allow:/

User-agent: Googlebot-Mobile

Disallow: /search?updated-max

Allow:/

User-agent: Googlebot

Disallow: /search?updated-max

Allow:/

User-agent: *

Disallow:

Allow:/