Blog

Google: Disallowed URLs Do Not Affect Crawl Budget

Google’s Gary Illyes updated his original writeup on crawl budget with clarification about disallowed URLs.

The document now includes the following information:

“Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?

A: No, disallowed URLs do not affect the crawl budget.”

The question refers to the “User-agent: * Disallow: /” protocol in robots.txt that blocks web crawlers.

It can either be used to block an entire site from being crawled, or it can be used to block specific URLs from being crawled.

According to the update from Illyes, blocking specific URLs will not impact the crawl budget throughout the rest of the site.

Pages will not get crawled more frequently as a result of other pages on the site being disallowed from crawling.

There’s also no disadvantage to disallowing URLs when it comes to crawl budget.

The updated information appears at the bottom of this article, which is a Webmaster Central blog post from 2017.

Illyes said on Twitter that there are plans to turn the blog post into an official help center article.

Related Posts


Seo-vs-PPC

September 21, 2022

Why You Need to Offset PPC with SEO & Email Marketing

If you’re going to invest in paid ads, it’s important to offset them with email marketing campaigns. Here are four reasons why: You need to build trust before asking for money It’s easy to get caught up in the excitement of a new campaign and reach out to customers right away. But that can lead

marketing-cannabis-billboard

April 15, 2022

How to do cannabis marketing

How to do Cannabis Marketing the right way?  This is a question we get asked a lot.  The cannabis industry has grown by leaps and bounds in the past few years. With legalization increasing across the country, the industry is booming. According to some estimates, it is set to double by 2025!  If you’re