How To Add An X-Robots-Tag Header To Affiliate Links

In mid-2019, I had written a post about Google’s decision to stop supporting “noindex” directives in robots.txt. My feelings on the subject haven’t changed since then – the robots.txt should be the right place for this tag, as this directive is exclusive to robots. But Google powered ahead anyway, so I have no choice but to find a way to implement the x-robots-tag in the response header as recommended by the almighty tyrant of the web.

Unfortunately, there’s no easy way to add response headers. All methods require some level of fiddling around either with the .htaccess file or with PHP code. None of which appealed to me. I recently started looking into Cloudflare workers as a solution, and this seemed to fit the bill.

So here’s how I chose to add the “X-Robots-Tag” response header to my affiliate links on my site WP-Tweaks. For this purpose, the free tier of Cloudflare is more than enough.

Requirement:

1. A Cloudflare account with full DNS integration

Step 1: Note the URL “Base” of your Affiliate Links

I use the handy “Thirsty Affiliates” WordPress plugin, which creates affiliate links with a base of “/recommends/” by default. If you use another structure for your affiliate links, make sure that you can identify a part of the URL that is unique in this regard. You want to ensure that you add this tag only where you want to and nowhere else!

For my site WP-tweaks, I will assume that the unique part of the URL is

https://www.wp-tweaks.com/recommends/

Step 2: Create a Worker with the Following Code

When you log into Cloudflare, click the “Workers” tab on top and go through the steps to create a new worker. You’ll be asked if you want to set up a new subdomain – just click yes. Proceed to the area where you need to add the worker’s code and paste in the following:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  let response = await fetch(request)
  let newHeaders = new Headers(response.headers)
  newHeaders.set("X-Robots-Tag", "noindex, nofollow")

  return new Response(response.body, {
    status: response.status,
    statusText: response.statusText,
    headers: newHeaders
  })
}

Here’s what it looks like:

This code will apply the “noindex, nofollow” directive to the URLs that pass through Cloudflare. So far we haven’t yet assigned a route for this worker, so it’s not affecting anything on your site yet! This is extremely important to check.

Give your worker a name, and save your changes.

Step 3: Assign a “Route” to Your Worker

Go back to the initial worker screen, and click “Add Route”:

Here, add the route path we found in Step 1, and select the worker you just created from the dropdown box:

Save your changes, and you’re done! The Worker should start processing all requests immediately.

Step 4: Verify that it’s Working

Fire up Chrome, and open the Dev Tools via F12, or “ctrl+shift+c”. Go to the “Network” tab and check the box labeled “Preserve log”. We need to do this because affiliate links redirect immediately, and we don’t want the log to be cleared before we can see the response header.

Enter an affiliate link in the URL bar and load the page. It will, of course, redirect, but in the list of resources loaded, you should see the original request as shown here:

Check the “Headers” tab, and you should now see the recently added “x-robots-tag” in the response header as seen in this screenshot. Once you’re satisfied that all your affiliate links have the same thing, you should uncheck the “Preserve log” box.

Now load normal URLs on your site and ensure that there isn’t an x-robots-tag! You don’t want to de-index your entire site by mistake!

And there you have it. A free solution for adding an x-robots-tag to your affiliate links that doesn’t involve changing your server in any way!

Bhagwad Park

I’m a NameHero team member, and an expert on WordPress and web hosting. I’ve been in this industry since 2008. I’ve also developed apps on Android and have written extensive tutorials on managing Linux servers. You can contact me on my website WP-Tweaks.com!