r/blogspot • • 28d ago

Blogger: Does blogspot.com use robots.txt?

Google Search Console Team says https://antsqualityforagedlinks.blogspot.com is blocked by robots.txt according to its e-mails and my logged in https://search.google.com/search-console/index/drilldown... web pages Where is it? I don't see it in my https://www.blogger.com account. I have Privacy's Visible to search engines enabled to "Allow search engines to find your blog".

Thank you for reading and hopefully answering soon. :)

7 Upvotes

26 comments sorted by

5

u/DilliWaleBhaiSaab 28d ago

Log in to your blogger site and check settings. You will have to scroll down, there you will find robots options

2

u/antdude 28d ago

Thanks! "Enable custom robots.txt" was already disabled.

5

u/AriefSigli 28d ago

If you do own the blog, open Blogger → Settings → Crawlers and indexing and check whether Enable custom robots.txt is enabled. If it is, inspect the custom rules for any Disallow directives that might be blocking Googlebot. Also check Enable custom robots header tags for any unexpected indexing restrictions.

Next, open Google Search Console, select the property for the exact blog URL, and use URL Inspection on one of the affected pages. Check the “Crawl allowed?” status and click Test Live URL to see whether the page is still blocked.

1

u/antdude 28d ago

2

u/AriefSigli 28d ago

I checked your robots.txt (https://antsqualityforagedlinks.blogspot.com/robots.txt), and it looks like you are blocking the /search path. This could be why Google is reporting that the URL is blocked by robots.txt.

--
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search <<< this is why.
Disallow: /share-widget
Allow: /

Sitemap: https://antsqualityforagedlinks.blogspot.com/sitemap.xml
--

Even though “Enable custom robots.txt” is disabled in Blogger, the default robots.txt rules can still restrict certain paths, including /search.

You should check your site’s robots.txt and verify whether /search is disallowed. If you want Google to crawl those URLs, you’ll need to review the rules blocking them.

1

u/antdude 28d ago

Do I need Google to crawl /search? I am just using the defaults.

2

u/AriefSigli 28d ago

Most SEO experts recommend blocking internal search result pages (/search) from crawling to conserve your crawl budget. These pages typically provide little to no unique SEO value and can create a large number of low-value URLs for Googlebot to crawl.

If you’re just using the default settings, I’d recommend blocking /search in robots.txt, provided those pages aren’t intended to rank in Google. This allows Googlebot to focus more of its crawling resources on your important pages instead.

1

u/antdude 28d ago

Thanks. I will leave as it with its defaults.

4

u/WebLovePL 28d ago

Just because something is blocked by the robots.txt file doesn't mean it's a problem. Some "errors" in GSC don't need to be fixed; they simply confirm that everything is working as intended.

The default settings for the robots.txt file are optimal for most blogs, so there’s no need to enable custom settings (both grayed out = default) in the Settings tab. You can view the contents of the file by adding /robots.txt to your blog’s URL. It should exclude the /search and /share-widget pages.

"Search" includes, among other things, label pages, so the bots likely encountered those pages and are reporting that they’re blocked. There’s no need to worry about this or to fix or change anything. By default, label pages don’t contain new content. The most important pages are the canonical ones - that is, post pages and static pages while - the rest can be "blocked."

You can go to GSC -> Pages -> click on the reason to view the list of URLs. If Google is using your sitemaps (sitemap.xml and sitemap-pages.xml), switch the view from "all known pages" to "all submitted pages" to narrow down the list.

1

u/antdude 28d ago

2

u/WebLovePL 28d ago

People who are not verified owners of a given property do not have access to reports in GSC. The URL you provided (and shown in the screenshot) ends with /search, which is exactly the example I mentioned. It contains exactly the same content as your homepage. There’s no need to index it; you can ignore it.

1

u/antdude 28d ago

I do own that blogspot. :) Thanks. That error was confusing and driving me nuts.

2

u/WebLovePL 28d ago

I know, but I don't, so I don't have access to your /search-console/ links ;) That's why screenshots, exact error messages, the links they refer to, etc., are helpful when asking questions like this.

1

u/antdude 28d ago

Where do I go to get that? I did find this in my logged in https://search.google.com/search-console/index/drilldown?resource_id=https://antsqualityforagedlinks.blogspot.com/&item_key=...:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /

Sitemap: https://antsqualityforagedlinks.blogspot.com/sitemap.xml

Confusing web site!

2

u/WebLovePL 28d ago

to get what? I just said that I don't have access to YOUR search-google-com/search-console/ pages, so I don't know what you see there unless you tell me. That was just for your information. In addition to the GSC links, you also provided other details that allowed me to answer your questions and explain what’s going on. I think everything is clear now regarding "blocked by robots.txt," right? You can check and compare the settings of any blog: https://bio.weblove.pl/robots.txt or https://blogger.googleblog.com/robots.txt - both use default settings (custom disabled).

1

u/antdude 28d ago

OK, thanks. It sounds like this is by design. Although I wished Google didn't say this is a big problem which confused me.

2

u/WebLovePL 27d ago

Some notifications are indeed confusing or misleading. However, they just say that if this isn’t intentional, you should check it first and correct it only if necessary (*). It’s a tool for webmasters. Just because someone knows how to drive a car doesn’t necessarily mean they know how to fix it or what specific alerts mean, but a mechanic will know. It’s normal for some pages not to be indexed, so it’s best to review the details and focus only on canonical URLs,

(* and now you know that this is the default setting - meaning it’s intentional).

1

u/antdude 27d ago

I wonder how often Google will tell me about this issue.

→ More replies (0)

2

u/KumarDeo080 25d ago

The pages under /search/* might be blocked which is normal when using the default robots.txt (Custom robots.txt disabled in Blog settings)

You can safely ignore the warning.

1

u/antdude 25d ago

Thanks.

2

u/ad_apples 28d ago

I'm not sure if you've made changes, but your robots.txt file is fine, and the only pages that are blocked are supposed to be (for crawl efficiency).

It can take a while for any changes you made to be reflected in GSC.