Blog

Google: Disallowed URLs Do Not Affect Crawl Budget

Google’s Gary Illyes updated his original writeup on crawl budget with clarification about disallowed URLs.

The document now includes the following information:

“Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?

A: No, disallowed URLs do not affect the crawl budget.”

The question refers to the “User-agent: * Disallow: /” protocol in robots.txt that blocks web crawlers.

It can either be used to block an entire site from being crawled, or it can be used to block specific URLs from being crawled.

According to the update from Illyes, blocking specific URLs will not impact the crawl budget throughout the rest of the site.

Pages will not get crawled more frequently as a result of other pages on the site being disallowed from crawling.

There’s also no disadvantage to disallowing URLs when it comes to crawl budget.

The updated information appears at the bottom of this article, which is a Webmaster Central blog post from 2017.

Illyes said on Twitter that there are plans to turn the blog post into an official help center article.

Related Posts


June 11, 2020

How to Choose a Virginia SEO Company

Owning a business is more challenging nowadays.  Along with the many things to consider and focus on, choosing the best Virginia SEO company that will work for your branding is another vital component to think about. There will be a significant consequence not just in your bank account, but also for the business. With the

June 8, 2020

How to Leverage Ecommerce Customer Reviews To Drive Conversions

Do you know your ecommerce website’s conversion rate? You absolutely should. It’s the number that you live and die by. It is the measure of your business’ success. Not only do you have to know your conversion rate, but you have to be continually taking steps to improve it. The average conversion rate for an