Why You Should Turn On “Crawler Hints” For Your Site

If you’re a Cloudflare user, you need to turn on “Crawler Hints” for your websites at the first opportunity. Here’s why it’s a good thing for you, for the planet, and even for website crawlers.

Crawling is Wasteful

If your website is typical., then you know that the vast majority of hits to your site are bots. And most of these bots are not malicious. Ordinary search engine crawlers from Google, Bing, and Yandex are probably requesting thousands of your site’s pages every day, just to keep their index up-to-date. And this is for a small to medium website. Larger sites have even more hits.

This is because search engines don’t have access to three pieces of information that they need:

1. Has a page changed?
2. When did it change?
3. What changed?

Without knowing all this, search engines are forced to crawl your site, again and again, to find out what changed. Even worse, they might not realize that some content has changed until a much later date. Potentially weeks. There’s got to be a better solution.

IndexNow – A new Protocol Developed by Microsoft and Yandex

While Google chooses to sit this one out, for now, Bing and Yandex collaborated to create a new “IndexNow” protocol, which allows websites to signal to search engines about changes on their site. The idea is that whenever some content on your site changes, or when you add new content, you can use the protocol to inform search engines about the exact changes. This way, crawlers know in advance before crawling your site, what content needs to be re-indexed.

This has several benefits.

First, search engines will crawl your site less intensely. Since they don’t need to crawl your entire site, they can just update the pages that have actually changed or that you’ve added. This means less strain on your origin server, and faster load times for your visitors, as your server can now dedicate more resources to serving them.

Second, crawlers will have access to updated content much faster than before. This means that the gap between content changing on your site and the changes showing up in the search results can be much lower.

Third, it’s good for the environment. Wasteful crawling is estimated to generate an equivalent of cutting down over 30 million acres of forest according to this calculator from the EPA.

So it’s a win-win for everyone.

Enable IndexNow from Cloudflare with “Crawler Hints”

Luckily, Cloudflare users have an easy way to enable the IndexNow protocol. Under the “Cache” section, you can enable crawler hints immediately as shown here:

With this change, Cloudflare will immediately know when your content has changed and will send a ping to the search engines informing them of the update through the IndexNow protocol.

Another improvement is that Cloudflare will zone in on important pages and get the changes on them indexed before those that are less important, further improving the priorities of crawling.

Will Google Follow?

While Microsoft and Yandex have committed to IndexNow, Google still hasn’t decided what it’s going to do. The initial responses are promising, though, since they said that they’re looking into the protocol and will decide the path forward. Hopefully, they don’t create an entirely new standard, necessitating the maintenance of two separate tracks for development.

But even if Google doesn’t sign on you’ll be spared excessive bot craws from both Bing and Yandex. Remember that the popularity of Bing doesn’t affect the rate at which it crawls your site. So this is a big deal even if you don’t get much traffic from Bing.