• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Startup Hero

The Official Blog Of Name Hero

Learn To Fly Above The Competition Get Started
  • HomeWelcome
  • CoursesVideo Training Center
  • About UsWhat is StartupHero?
  • BlogGet the latest
  • Start HereStartup 101
  • SpeakingPodcast & Media
  • ResourcesTools to help You
    • Reselling WordPress Hosting
    • Resell Hero
    • How To Start A Blog
  • NameHeroCloud Web Hosting

How To Add An X-Robots-Tag Header To Affiliate Links

By Bhagwad Park on May 26, 2021 1

How To Add An X-Robots-Tag Header To Affiliate Links

In mid-2019, I had written a post about Google’s decision to stop supporting “noindex” directives in robots.txt. My feelings on the subject haven’t changed since then – the robots.txt should be the right place for this tag, as this directive is exclusive to robots. But Google powered ahead anyway, so I have no choice but to find a way to implement the x-robots-tag in the response header as recommended by the almighty tyrant of the web.

Unfortunately, there’s no easy way to add response headers. All methods require some level of fiddling around either with the .htaccess file or with PHP code. None of which appealed to me. I recently started looking into Cloudflare workers as a solution, and this seemed to fit the bill.

So here’s how I chose to add the “X-Robots-Tag” response header to my affiliate links on my site WP-Tweaks. For this purpose, the free tier of Cloudflare is more than enough.

Requirement:

  1. A Cloudflare account with full DNS integration

Step 1: Note the URL “Base” of your Affiliate Links

I use the handy “Thirsty Affiliates” WordPress plugin, which creates affiliate links with a base of “/recommends/” by default. If you use another structure for your affiliate links, make sure that you can identify a part of the URL that is unique in this regard. You want to ensure that you add this tag only where you want to and nowhere else!

For my site WP-tweaks, I will assume that the unique part of the URL is

https://www.wp-tweaks.com/recommends/

Step 2: Create a Worker with the Following Code

When you log into Cloudflare, click the “Workers” tab on top and go through the steps to create a new worker. You’ll be asked if you want to set up a new subdomain – just click yes. Proceed to the area where you need to add the worker’s code and paste in the following:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  let response = await fetch(request)
  let newHeaders = new Headers(response.headers)
  newHeaders.set("X-Robots-Tag", "noindex, nofollow")

  return new Response(response.body, {
    status: response.status,
    statusText: response.statusText,
    headers: newHeaders
  })
}

Here’s what it looks like:

Add the Code to the Cloudflare Worker
Add the Code to the Cloudflare Worker

This code will apply the “noindex, nofollow” directive to the URLs that pass through Cloudflare. So far we haven’t yet assigned a route for this worker, so it’s not affecting anything on your site yet! This is extremely important to check.

Give your worker a name, and save your changes.

Step 3: Assign a “Route” to Your Worker

Go back to the initial worker screen, and click “Add Route”:

Add Route to the Cloudflare Worker
Add Route to the Cloudflare Worker

Here, add the route path we found in Step 1, and select the worker you just created from the dropdown box:

Configure the Cloudflare Route with the Worker
Configure the Cloudflare Route with the Worker

Save your changes, and you’re done! The Worker should start processing all requests immediately.

Step 4: Verify that it’s Working

Fire up Chrome, and open the Dev Tools via F12, or “ctrl+shift+c”. Go to the “Network” tab and check the box labeled “Preserve log”. We need to do this because affiliate links redirect immediately, and we don’t want the log to be cleared before we can see the response header.

Enter an affiliate link in the URL bar and load the page. It will, of course, redirect, but in the list of resources loaded, you should see the original request as shown here:

Verify the X-Robots-Tag
Verify the X-Robots-Tag

Check the “Headers” tab, and you should now see the recently added “x-robots-tag” in the response header as seen in this screenshot. Once you’re satisfied that all your affiliate links have the same thing, you should uncheck the “Preserve log” box.

Now load normal URLs on your site and ensure that there isn’t an x-robots-tag! You don’t want to de-index your entire site by mistake!

And there you have it. A free solution for adding an x-robots-tag to your affiliate links that doesn’t involve changing your server in any way!

Bhagwad Park Profile Picture
Bhagwad Park

I’m a NameHero team member, and an expert on WordPress and web hosting. I’ve been in this industry since 2008. I’ve also developed apps on Android and have written extensive tutorials on managing Linux servers. You can contact me on my website WP-Tweaks.com!

Reader Interactions

Comments

  1. Tom Tolkien says

    September 9, 2021 at 5:43 pm

    Thank you so much – this advice was incredibly useful and solved a long-standing issue with Google Search Console. Much appreciated.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

Connect With Us!

Superhero Resources

Fix Common Issues

  • How To Setup Free And Automatic SSL Certificates
  • How To Setup Cloudflare With Railgun
  • How To Fix Memory Exhausted Errors In WordPress
  • How To Edit PHP Version/Upload Limit/Add Extensions
  • How To Move/Migrate Your Business To Name Hero

Free Guides

  • How To Setup NameHero Hosting
  • How To Create A Web Hosting Business With WordPress
  • How To Start A WordPress Blog
  • How To Migrate WordPress To A VPS
  • How To Speed Test And Optimize Your WordPress Website
  • Magento 2.X Installation Guide
  • How To Clean Up A WordPress Hack

Training

Recent Posts

  • The Best WordPress Plugin To Combat Plagiarism
  • Is It Time to Bring Back Blog Comments?
  • Two Filters That No Longer Work In Gutenberg (And Workarounds)
  • Migrating To The New Cloudflare Managed Rules Interface
  • How To Sort Posts By The “Last Modified” Date
  • How To Make WordPress Send E-mail From Another Address
Subscribe in a reader
  • Web Hosting
  • WordPress Hosting
  • Reseller Hosting
  • VPS Hosting
  • Twitter
  • Facebook
  • LinkedIn
  • YouTube
  • FTC Disclosure
  • Earnings Disclosure
  • Privacy Policy

Copyright © 2023 · Smart Passive Income Pro on Genesis Framework · WordPress · Log in