Why in Webmaster Tools The Index Status Shows "Blocked by Robots" for Blogger Sites?


#1

Today, we would like to discuss here, that why Google webmaster tools show blocked by robots for most of blogger sites. Why it has been happening with blogspot sites, Or even it has happened with WP sites too. So if you are also in one of them who detects the error. Lets ask from our Pro Admin @Mohammad for the Issue.

The Question is further divided into 3 Parts:

1: In Google webmaster the index status under Advance show blocked by Robots for my site. Even we do not have yet setup the Custom robots.txt file on blogger dashboard. It is still disabled. Then why Webmaster show blocked by robots. ?

2: Should we Enable Custom robots.txt on Blogger Dashboard. ?

Custom robots.txt Disabled.

We hope, we will find out the solution soon, that why it is being happening with blogspot sites. What is the issue, and how we can debug the error. Stay tuned and hope for the best.


#5

Thanks for this bro. It is really helpful for bloggers.


#6

By default Blogger creates the following robots.txt files for every blog:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.mybloggertricks.com/sitemap.xml

Here the search pages ( /search) are blocked. Which means all label pages in your blog will be auto blocked and they will appear in the “Index Status” page under the Blocked by robots tab. The number would look greater than your labels count because it may also include the mobile URLs which has a different url parameter. You don’t need to worry about these urls if you have disabled the robots.txt option in “Search preferences”


URLs are blocked by robots? how can i resolve this index issue
#7

Thanks @Mohammad For your kindly reply.


#8

@Mohammad we would like to request for a robot.txt file, which you setup for MBT. will you reveal that codes to us you installed for MBT.

Thanks and hope to get your file soon.


#9

I already shared the code in my last comment! didn’t I? :smile: Ok again sharing it with you, it is the one which blogger auto created for MBT and we use this currently

 User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.mybloggertricks.com/sitemap.xml

#10

dear sir i am already use this cod but sir i have same problem


#11

Brother have you find and solution of this problem, I have too many blocked url in my webmaster tool.


#12

Bro Tell me your Web address… I will try to help you at my level best…!!! Don’t forget Dear to send a copy of your issue on my mail: anshik1998@gmail.com .


#13

Pls am also facing same issue how do i fix this @Mohammad


#14

I have sent you a mail @ANSHIK_BANSAL pls i need help this is really eating me up


#15

Sorry Bro @pookaman … I had gone for a 3 day tour to Jammu and Kashmir these days and Today, I came to my sweetie home from there. I had seen your mail today and I think the best answer can be: Listen Bro, Mohammad Bro had written the following order correctly, and I feel that you may had forget to mention your Blog name instead of that you had copied that code same.

User-agent: Mediapartners-Google Disallow:

User-agent: * Disallow: /search Allow: /

Sitemap: http://www.YOUR-BLOG-NAME.blogspot.com/sitemap.xml

Here, I am writing the code again, Just replace YOUR-BLOG-NAME with your Blog Name. Hope, this will sort out your problem. Have a Nice Blogging Experience.


#16

Brother i have fixed the issue by allowing css, js file into google index before this in my robots text these means css and js file are disallow to google bot for crawling.


#17

Hi Bro Please provide the code how to fix this issue and one more help needed,i have posted 62 posts in my blog but still those are not appearing in google search in first page ,what should i do for that,please help me


#18

Oh man, There is not such an advanced science behind this. You need just to use below line into robots text and submit it.

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php

and Wait until you received a successfully msg. I hope all is going will. And also you need to resubmit your sitemap into Google. If you are a WordPress user then, you need first to generate sitemap including, post, pages, category, and image then upload those files on root direct of your domain in the file manager. After that, you also submit the sitemap and wait for some days. Your post and other things will be indexing into Google and other search engine.


#19

Thanks. for your tips . Still i couldn’t able to view the possibilities of the blocked robots.txt in Webmaster.

i have faced the issue in the following site. Please help me how to recover the issue asap.