Blog

Google: Disallowed URLs Do Not Affect Crawl Budget

Shutterstock 1159828186

Google’s Gary Illyes updated his original writeup on crawl budget with clarification about disallowed URLs.

The document now includes the following information:

“Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?

A: No, disallowed URLs do not affect the crawl budget.”

The question refers to the “User-agent: * Disallow: /” protocol in robots.txt that blocks web crawlers.

It can either be used to block an entire site from being crawled, or it can be used to block specific URLs from being crawled.

According to the update from Illyes, blocking specific URLs will not impact the crawl budget throughout the rest of the site.

Pages will not get crawled more frequently as a result of other pages on the site being disallowed from crawling.

There’s also no disadvantage to disallowing URLs when it comes to crawl budget.

The updated information appears at the bottom of this article, which is a Webmaster Central blog post from 2017.

Illyes said on Twitter that there are plans to turn the blog post into an official help center article.

Related Posts


Search Engine Optimization Process

January 10, 2023

SEO Services in Virginia Beach: Boosting Local Businesses Online

As the world becomes increasingly digital, inflation is up businesses of all sizes are turning to the internet to connect with customers and grow their reach. This is especially true for small and local businesses in Virginia Beach, where competition can be fierce. One of the most effective ways to stand out online is through

Seo-vs-PPC

September 21, 2022

Why You Need to Offset PPC with SEO & Email Marketing

If you’re going to invest in paid ads, it’s important to offset them with email marketing campaigns. Here are four reasons why: You need to build trust before asking for money It’s easy to get caught up in the excitement of a new campaign and reach out to customers right away. But that can lead