How to Stop Bots and Crawlers Causing High CPU and Slowdowns on WordPress

Your host warns that the account is hitting its CPU limit, top shows a wall of PHP processes, or the site slows to a crawl at random hours. Yet analytics shows a normal number of visitors. That gap is the clue: analytics only counts browsers that run JavaScript. Bots and crawlers do not, so they are invisible there, but every request they make still costs your server work.

Underneath, a bot is just a program requesting pages. Some are welcome (search engines), some are aggressive (SEO tools, content scrapers, AI training crawlers), and some are hostile (password guessing against wp-login.php and xmlrpc.php). When they request pages that bypass your cache, each request starts PHP and hits the database. This guide shows you how to identify them from your logs and slow them down or shut them out, using defensive measures only.

What is happening

Bot type	Typical pattern in logs	Response
Search engines (Googlebot, Bingbot)	Steady crawling of real URLs, verifiable by DNS	Allow. Reduce wasted crawling instead.
SEO crawlers (AhrefsBot, SemrushBot, MJ12bot, DotBot)	Bursts of many URLs, honest user agent	Slow down or block in robots.txt; enforce at the server if ignored.
AI and dataset crawlers (for example CCBot, Bytespider)	Fast, wide crawls including old and parameter URLs	Disallow in robots.txt, enforce at the edge.
Login brute force	Many POSTs to /wp-login.php or /xmlrpc.php	Block XML-RPC, rate limit and challenge login.
Fake crawlers	Claims to be Googlebot, IP does not belong to Google	Block.
Vulnerability scanners	Requests for .env, old plugin paths, many 404s	WAF rules; usually cheap if 404s are fast.

The expensive pattern is always the same: requests that cannot be served from cache. That includes logins, search (?s=), WooCommerce filter and sort parameters, ?add-to-cart= links, and random query strings.

Before you start

• Access: SSH to read logs, or your hosting panel’s raw access log download. Cloudflare dashboard access if you use it.
• Log locations: commonly /var/log/nginx/access.log, /var/log/apache2/access.log (Debian and Ubuntu), /var/log/httpd/access_log (RHEL family), or a logs folder in your hosting account. Older logs are rotated and compressed as .gz; read those with zcat instead of cat.
• Running commands: the log commands run on the server over SSH. DNS checks (host, dig) run on your own computer or the server; on Windows use WSL, or nslookup in PowerShell.
• Backup: copy any config file before editing it (cp FILE FILE.bak), and test Nginx changes with nginx -t before reloading.
• Behind Cloudflare or another proxy? Your logs may show the proxy’s IPs instead of visitors’. Your server needs real IP restoration (Nginx real_ip_header CF-Connecting-IP with set_real_ip_from for Cloudflare’s ranges, or Apache mod_remoteip). Without it, never block by IP at the server, or you will block Cloudflare itself.

Step 1: Read the access log to find the bots

The default “combined” log format looks like this:

IP_ADDRESS - - [07/Nov/2026:14:03:11 +0000] "GET /shop/?filter_color=red HTTP/1.1" 200 51234 "-" "Mozilla/5.0 (compatible; ExampleBot/1.0)"

Split on spaces, field 1 is the IP, field 4 the time, field 7 the URL and field 9 the status code. Split on double quotes, field 6 is the user agent. All commands below run on the server; replace ACCESS_LOG with your log path.

Requests per minute, to find when the spike happened

awk '{print substr($4,2,17)}' ACCESS_LOG | uniq -c | tail -30

substr($4,2,17) trims the timestamp to day and minute, and uniq -c counts consecutive identical lines, so you get requests per minute. Good: fairly even numbers. Bad: a jump from 40 to 900 per minute marks the attack window.

Top IPs

awk '{print $1}' ACCESS_LOG | sort | uniq -c | sort -rn | head -20

Counts requests per IP and sorts them highest first. head -20 keeps the top twenty. Reading it: one IP with thousands of requests in a log that covers a day is a bot. Many IPs in the same range (same first two or three numbers) suggest a crawler farm or a cloud provider.

Top user agents

awk -F'"' '{print $6}' ACCESS_LOG | sort | uniq -c | sort -rn | head -20

-F'"' splits each line on double quotes, so $6 is the user agent. Bot names are usually obvious here: anything with “bot”, “crawler” or “spider”, or empty and generic agents such as python-requests or Go-http-client.

Top URLs, and what one bot is requesting

awk '{print $7}' ACCESS_LOG | sort | uniq -c | sort -rn | head -20
grep "BOT_NAME" ACCESS_LOG | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Placeholder: BOT_NAME is part of a user agent from the previous list, such as AhrefsBot. Reading it: thousands of distinct filter, sort or search URLs mean the bot is lost in a crawl trap that bypasses cache. That is where your CPU goes.

Narrow to the spike window

grep "DD/Mon/YYYY:HH:" ACCESS_LOG | awk '{print $1}' | sort | uniq -c | sort -rn | head

Placeholder: DD/Mon/YYYY:HH is the day and hour of the spike as it appears in the log, for example 07/Nov/2026:14. Log times are usually in the server’s timezone or UTC; check the offset at the end of the timestamp.

Step 2: Rule out xmlrpc.php and wp-login.php brute force

grep -E '"POST /(wp-login|xmlrpc)\.php' ACCESS_LOG | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

This matches POST requests to either file and counts them per IP. Good: a handful from your own team. Bad: hundreds or thousands, often from many IPs. Each login attempt runs PHP and a password hash check, which is deliberately slow, so floods are expensive. XML-RPC is worse: its system.multicall method can try many passwords in a single request.

Block XML-RPC at the server

Most sites do not need XML-RPC. Jetpack and some mobile and publishing apps do; if you use them, restrict it instead of blocking it. Disabling it with a WordPress filter still loads PHP for every request, so block it before PHP. For Apache, in the .htaccess file in your site folder:

<Files "xmlrpc.php">
    Require all denied
</Files>

This is Apache 2.4 syntax and also works on LiteSpeed. For Nginx, in your site’s server block:

location = /xmlrpc.php {
    deny all;
}

Confirm: from your computer run curl -sI https://example.com/xmlrpc.php (replace example.com). Good output is 403 Forbidden. A 405 or 200 means requests still reach PHP.

For the login page, rate limit it (Step 5), add a challenge at Cloudflare (Step 6), and use a security plugin’s login protection or a tool such as fail2ban that bans IPs after repeated failures. Cleaning up a site where a login attempt succeeded is a separate job covered in the hacked site guide (FG-05).

Step 3: Spot fake Googlebots with reverse and forward DNS

Anyone can put “Googlebot” in a user agent. Blocking the real one hurts your search visibility; letting fakes through wastes CPU. Verify with two DNS lookups, from your computer or the server:

host IP_ADDRESS
host HOSTNAME_FROM_FIRST_RESULT

Placeholders: IP_ADDRESS is the IP from your log claiming to be Googlebot. HOSTNAME_FROM_FIRST_RESULT is the name the first command returns. Genuine example: the first lookup returns a name ending in googlebot.com, google.com or googleusercontent.com (the last is used by some user-triggered Google fetchers), and the second lookup returns the same IP you started with. Fake: the first lookup returns nothing or an unrelated hosting provider’s name, or the second returns a different IP. Both steps matter: anyone controlling an IP’s reverse DNS could make it claim a Google name, but they cannot make Google’s forward DNS point back to it.

Bingbot verifies the same way, with names ending in search.msn.com. Google and Bing also publish their crawler IP ranges as downloadable lists, useful for allow rules. Cloudflare does this verification for you: its “known bots” check only passes crawlers it has verified.

To check many IPs claiming to be Googlebot at once:

grep "Googlebot" ACCESS_LOG | awk '{print $1}' | sort -u | while read ip; do echo "$ip $(host $ip | awk '{print $NF}')"; done

This prints each unique IP with its reverse DNS name. Any name not ending in a Google domain, or showing 3(NXDOMAIN), is a fake to block.

Step 4: robots.txt for honest crawlers

robots.txt is a request, not a lock. Honest crawlers read it; hostile bots ignore it. It is still the right first move for SEO and dataset crawlers that identify themselves. A WordPress example, in the robots.txt file at your site root (or your SEO plugin’s robots editor):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?add-to-cart=
Disallow: /*?*filter_
Disallow: /*?s=
User-agent: AhrefsBot
Crawl-delay: 10
User-agent: CCBot
Disallow: /

What it does: the first group keeps all crawlers out of admin, add-to-cart links and filter and search URLs, which are the crawl traps that bypass cache. Crawl-delay: 10 asks a crawler to wait ten seconds between requests. The last group shuts out one crawler entirely. Adapt the patterns to your own URL structure; check the “Top URLs” output from Step 1 first.

Know the limits. Google ignores Crawl-delay completely, and Search Console no longer has a crawl rate setting. Googlebot slows down by itself when your server returns 500, 503 or 429 responses; Google advises returning those only briefly, since prolonged errors can lead to pages being dropped. Bing and several SEO crawlers do honor Crawl-delay. Changes can take a day or so to be picked up, because crawlers cache robots.txt.

Step 5: Rate limit at the web server

Nginx limit_req

Nginx has built-in request rate limiting. Define a zone in the http block (often /etc/nginx/nginx.conf or a file in conf.d):

limit_req_zone $binary_remote_addr zone=wplogin:10m rate=1r/s;
limit_req_status 429;

Then apply it in your site’s server block:

location = /wp-login.php {
    limit_req zone=wplogin burst=5 nodelay;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_pass PHP_UPSTREAM;
}

What each part does: $binary_remote_addr tracks each client IP compactly. zone=wplogin:10m names the zone and gives it 10 MB of memory, enough for many thousands of IPs. rate=1r/s allows one request per second on average. burst=5 lets a client make five extra requests in a quick burst, and nodelay serves those immediately instead of spacing them out. Requests beyond that receive 429 Too Many Requests because of limit_req_status (the default is 503). Placeholder: PHP_UPSTREAM is whatever your existing PHP location uses after fastcgi_pass, such as a socket path; copy it, and any other lines, from your current location ~ \.php$ block, because an exact-match location replaces that block for this URL.

Confirm: run nginx -t (good output ends with “test is successful”), reload Nginx, then watch the log: floods now show status 429 instead of 200. For a crawler that hammers every page, a second, more generous zone (for example rate=5r/s with a larger burst) can be applied in the PHP location, but test carefully so real visitors behind one shared office IP are not limited.

To refuse specific bad user agents in Nginx:

map $http_user_agent $blocked_agent {
    default 0;
    ~*(MJ12bot|DotBot|Bytespider) 1;
}
# inside the server block:
if ($blocked_agent) { return 403; }

The map goes in the http block. ~* means a case-insensitive regular expression. Edit the list to match the agents you found in Step 1.

Apache

Apache has no built-in per-IP request rate limiter (mod_ratelimit limits bandwidth, not request counts). Your options are the mod_evasive module, ModSecurity rules, fail2ban watching the access log, LiteSpeed’s own per-client throttling if your server runs LiteSpeed, or a security plugin. You can refuse user agents in .htaccess with mod_rewrite:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (MJ12bot|DotBot|Bytespider) [NC]
RewriteRule .* - [F,L]

[NC] makes the match case-insensitive, [F] returns 403 Forbidden and [L] stops processing further rules. Place it above the WordPress block in .htaccess. Test with curl -sI -A "DotBot" https://example.com/, where -A sets the user agent; good output is 403.

Step 6: Cloudflare WAF rules and bot settings

If your site is behind Cloudflare, blocking there is best: bad requests never reach your server at all. Cloudflare’s dashboard has been reorganized more than once, so menu names may differ slightly in your account. At the time of writing, custom rules and rate limiting rules are under Security, Security rules, and bot settings under Security, Settings (in older layouts, Security, WAF and Security, Bots).

Custom rules

Create a custom rule with “Edit expression” and paste one of these:

(http.request.uri.path eq "/xmlrpc.php")
(http.request.uri.path eq "/wp-login.php" and not ip.src in {YOUR_IP})
(http.user_agent contains "Googlebot" and not cf.client.bot)

Each is a separate rule. The first, with action Block, stops XML-RPC at the edge. The second, with action Managed Challenge, makes visitors to the login page pass a browser check; YOUR_IP is your own public IP address (search “what is my IP” to find it), so you are not challenged. The third, with action Block, stops anything claiming to be Googlebot that Cloudflare has not verified as a known bot. Check Security Events (the Events tab under Security, Analytics) after a few hours: good results show the rules matching bot traffic and not your real visitors.

Rate limiting rules

Rate limiting rules count requests per IP over a period and block or challenge above a threshold. A typical login rule matches the path /wp-login.php with method POST, counts by IP, and blocks for a period once the count is exceeded. The number of rules and the periods available depend on your plan.

Bot settings

• Bot Fight Mode (the free bot setting) challenges traffic Cloudflare identifies as automated. It cannot be skipped or adjusted with custom rules, so if it interferes with an API or a monitoring service you must turn it off.
• Super Bot Fight Mode (Pro and higher) lets you choose actions for “definitely automated” traffic, and on higher plans “likely automated”, while allowing verified bots. It has an option to optimize for WordPress so known WordPress paths keep working.
• AI crawler settings: the bot settings include AI crawler access policies (earlier labeled “Block AI bots”, then “Configure AI bot policies”; labels change, so check your dashboard), and AI Crawl Control (formerly AI Audit) lets you allow or block individual crawlers by name.
• Under Attack mode challenges every visitor; use it only during an active flood, then turn it off.

Also cache your HTML at the edge with a cache rule, bypassing it for logged-in and WooCommerce cookies. Crawlers that hit a cached page cost your server nothing.

Advanced: Correlating CPU with requests

To prove which traffic burns CPU, match the spike times from your monitoring against request types. Check the load on the server:

uptime
ps -eo pid,pcpu,etime,args --sort=-pcpu | head -15

uptime shows load averages for the last 1, 5 and 15 minutes; a load well above your CPU core count (see nproc) means requests are queueing. ps lists the busiest processes. Many php-fpm or lsphp processes near the top confirm PHP is the bottleneck. Then compare uncached and cached requests in the spike window. If your log format includes request time (Nginx $request_time), sort by it to find the URLs that are both frequent and slow. A crawler requesting 50,000 filter URLs at half a second each is far more expensive than a million cached page views. Fix the crawl trap (robots.txt, rel="nofollow" on filter links, or a WAF rule for crawlers on parameter URLs) and the CPU follows.

When to contact your host

Open a ticket if you cannot read raw logs, cannot edit server config, or the flood continues after blocking. Send the spike window in UTC, the top IPs, user agents and URLs from Step 1, the login and XML-RPC counts from Step 2, any DNS verification results, and what you have already blocked. Hosts can often block at a network firewall before traffic reaches your account. Include Cloudflare ray IDs if relevant.

For DNS and header checks, use our diagnostic tools. If the site is completely down under load, go straight to the Emergency Room.