• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
NameHero Blog

NameHero Blog

Web Hosting Tips & Resources From NameHero

  • Hosting
    • Web Hosting
    • VPS Hosting
    • WordPress Hosting
    • WooCommerce Hosting
  • Reseller
  • Enterprise
  • Domains
  • Account
  • Blog Home
  • Categories

Is It Wise to Block (Almost) All Bots?

Bhagwad Park

Published on: February 9, 2022

Categories: Website Security 0

Your write and maintain your website for human visitors. And yet when you peer into your analytics logs, you see that the vast majority of hits to your server come from bots. These don’t show up on client-side measuring tools like Google Analytics, but they appear on the server logs. Most traffic these days is from automated bots. You might be tempted to block them all. But that would be disastrous. What you need is a policy that understands which bots to let through, and which bots to block.

Not All Bots Are Bad – Search Engine Bots Need to be Let Through

Probably the most important bots that visit your site are those linked to the search engines. You don’t want to block Google from crawling your site! Annoying as it may seem, search engine bots will bombard your site with requests for static and dynamic content, and there’s nothing you can do about it. If you block them, you’re toast.

Blocking Other Crawlers Might be Inconvenient

The second category of bots are those that might be useful. They’re usually tools you use to analyze your site automatically. For example, let’s say you want to use Screaming Frog to crawl your site for a complete SEO audit. They use a crawler that needs access to your site’s contents. Another example is SEO tools like SEMRush, whose bots crawl your site to generate backlink reports, keyword optimization strategies, and more.

Another example is if you want an online tool to crawl your site to extract the critical CSS so that you can inline it. The tool won’t be able to carry out its function if you block bots.

However, these tools are used rarely enough, that you can consider disabling your bots firewall just long enough for them to complete their task, and then re-enable it.

All Other Bots can be Safely Blocked

If a bot isn’t crawling your site to index it for search, or an online tool (used by you), you can safely block it without consequence. Chances are that it’s trying to scrape your content, or that hack/spam your site.

Using Cloudflare to Block Bots

Cloudflare has a very useful tool to block bots. You can find it in the “Firewall” section under bots. It has a whitelist of “verified bots” that include all the important search engines, and popular tools like Ahrefs so that your site doesn’t block important stuff.

If you’re using their free plan, Cloudflare will use JavaScript detections to check if a visitor is a bot, and if so, will block them when you enable the option. As a “Pro” user, you can go one step further and block automated bots. Here’s a screenshot of the breakup of the visitors to my site:

Breakup of Visitors to Blog
Breakup of Visitors to Blog

You can see that over 50% of all visits to my site are from “verified bots”. But around seven thousand visits are either “Automated” or “Likely Automated”. I’m comfortable blocking the “Definitely Automated” visitors.

When Cloudflare blocks a bot, it logs a report in the firewall. Here’s a screenshot of all the traffic that Cloudflare has blocked on my behalf:

Automated Bots Blocked
Automated Bots Blocked

The “Firewall: Managed” rules in this care are almost entirely automated bots. In addition to this, Cloudflare also blocks a whole bunch of traffic based on its WAF rules, but since these are always automated bots in the first place, they get blocked immediately before Cloudflare has a chance to evaluate their behavior and requests.

Blocking bots is a great way to keep your site’s resources free for real human visitors. Particularly when these bots are trying to access non-existent PHP files in an attempt to probe your application for weaknesses. Doing this with Cloudflare is easy, and even those using the free tier can benefit greatly. I see no reason why most people shouldn’t turn it on.

Bhagwad Park Profile Picture
Bhagwad Park

I’m a NameHero team member, and an expert on WordPress and web hosting. I’ve been in this industry since 2008. I’ve also developed apps on Android and have written extensive tutorials on managing Linux servers. You can contact me on my website WP-Tweaks.com!

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

Follow & Subscribe

Exclusive promos, content and more!

Most Popular Posts

Speed up your site: solving the WordPress slowdown

NameHero’s Recommended WordPress Plugin and Theme Setup (2023)

How To Increase The InnoDB Buffer Pool Size

How To Fix A Stuck All-in-One WP Migration Import

How To Add A Subdomain In Cloudflare

Top Categories

  • WordPress
  • Website Performance
  • Web Hosting
  • Resellers
  • Website Development
  • Website Security
  • VPS Hosting
  • SEO Tips
  • Announcements
  • Domain Registration
NameHero

NameHero proudly provides web hosting to over 40,000 customers with 99.9% uptime to over 750,000 websites.

  • Master Card
  • Visa
  • American Express
  • Discover
  • Paypal
Products
  • Web Hosting
  • VPS Hosting
  • WordPress Hosting
  • WooCommerce Hosting
  • Reseller Hosting
  • Enterprise Hosting
  • Domains
Help & Support
  • NameHero Blog
  • Knowledgebase
  • Announcements
  • Affiliates
Company
  • About Us
  • Contact Sales
  • Reviews
  • Uptime
  • We're Hiring

Copyright © 2023 NameHero, LLC. All rights reserved.

  • Privacy Policy
  • Terms of Use
  • Acceptable Use Policy
  • Payment Policy
  • DMCA