robots.txt or noindex Blocking Your Whole Site From Google: How to Find and Fix It

Search Console shows a wall of Blocked by robots.txt or Excluded by ‘noindex’ tag, and your site is fading from Google. Somewhere, one line is telling search engines to stay away. It might be Disallow: / in a robots.txt file uploaded from staging, a “discourage search engines” box left checked, or an X-Robots-Tag header added by a server rule nobody remembers.

These two controls do different jobs, and mixing them up causes most of the damage. robots.txt controls crawling: whether Googlebot may fetch a URL. noindex controls indexing: whether a page it fetched may appear in results. This guide explains how each one is read, how to find which one is hurting you, how to protect a staging site the right way, and how to recover. If your whole site vanished and you have not yet narrowed down the cause, start with FG-77.

What is happening

Control	Where it lives	What it stops	What it does not stop
robots.txt Disallow	A plain text file at /robots.txt on each host	Crawling (fetching the URL)	Indexing of the bare URL if other sites link to it
Robots meta tag noindex	<meta name="robots" content="noindex"> in the page HTML	Indexing of that page	Crawling; Google must fetch the page to see the tag
X-Robots-Tag: noindex	An HTTP response header, set by the server, a plugin or a CDN	Indexing, and works for PDFs and images too	Crawling; also invisible in the page source
Password (HTTP basic auth)	Web server config	Everyone without the password, crawlers included	Nothing relevant; this is the reliable option for staging

The key consequence: a noindex behind a robots.txt Disallow is never seen. Google cannot fetch the page, so it cannot read the tag. If other pages link to the blocked URL, Google can still index the address with no description, which shows up as Indexed, though blocked by robots.txt. To remove a page from Google, let it be crawled and give it noindex. To save crawling on pages that do not matter, use robots.txt. Do not use both on the same URL.

Before you start

• Access you need: SFTP or a file manager to see whether a physical robots.txt exists in the site folder, the WordPress dashboard, Search Console, and SSH if you can get it. Server-level changes (Nginx config, some headers) need root or a ticket to your host.
• Backup: download the current robots.txt, .htaccess and any Nginx config file before editing.
• How to run commands: curl runs on your own computer. macOS and Linux include it; on Windows use curl.exe in PowerShell or install WSL. wp and htpasswd run in an SSH terminal on the server; run wp from the folder containing wp-config.php (some hosts name it wp-cli). Blocks that start with User-agent, Auth, Header, location or <RequireAny> are file contents you edit, not commands.

Step 1: Find which control is blocking you

Look at the Page indexing report under Indexing, Pages. A jump in Blocked by robots.txt points at robots.txt. A jump in Excluded by ‘noindex’ tag points at a meta tag or header. Then confirm on a key URL with the URL Inspection tool and Test live URL: Crawl allowed? answers the robots.txt question and Indexing allowed? answers the noindex question. Note that when crawling is blocked, the indexing answer cannot reflect a noindex Google never saw.

Check both from your computer in one go:

curl -s https://example.com/robots.txt
curl -sI https://example.com/ | grep -i -E '^HTTP|x-robots-tag'
curl -s https://example.com/ | grep -io '<meta[^>]*robots[^>]*>'

Replace example.com with your domain. -s hides progress output and -I requests headers only; grep -E lets one pattern match both the status line and the header. Good output: a robots.txt without a bare Disallow: /, an HTTP/2 200 status with no x-robots-tag line (or one without noindex), and a robots meta tag that is absent or free of noindex. Bad output is any of those blocks present on a page you want indexed.

Step 2: Understand how robots.txt is parsed

Google follows the Robots Exclusion Protocol, published as RFC 9309, with a few documented details of its own. Misreading the rules is how sites block themselves.

Groups and user agents

A file is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. A crawler picks the group whose user agent matches it most specifically and obeys only that group. If no group names it, it uses the User-agent: * group. If two groups name the same crawler, their rules are combined.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot
Disallow: /

This is robots.txt content, not a command. Here Googlebot ignores the * group entirely and follows its own group, which blocks everything. Every other crawler can crawl the site. This trap appears when someone adds a Googlebot-specific group to “tune” Google and accidentally writes /. The reverse trap also exists: a Googlebot group with a harmless rule means Googlebot no longer obeys any of the restrictions you put under *.

Disallow: / versus Disallow:

Line	Meaning
Disallow: /	Block every URL on this host for this group
Disallow: (empty)	Block nothing; same as no rule
Disallow: /shop	Block every path starting with /shop, including /shopping-guide/
Disallow: /shop/	Block only paths inside the /shop/ folder

Precedence between Allow and Disallow

Order in the file does not matter. When several rules match a URL, the one with the longest matching path wins. If an Allow and a Disallow match with equal length, Google uses the less restrictive one, the Allow. That is why Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file. Paths are case-sensitive: Disallow: /Private/ does not block /private/.

Wildcards

Google supports two special characters in paths. * matches any sequence of characters, and $ anchors the end of the URL.

• Disallow: /*? blocks every URL containing a query string. On WordPress that includes /?s= search and ?add-to-cart= links, and also any URL that carries a ? for any reason, so use it with care.
• Disallow: /*.pdf$ blocks URLs ending in .pdf, but not /file.pdf?download=1, because that does not end in .pdf.
• Disallow: /*/feed/ blocks feed URLs anywhere in the path.

A single stray *, for example Disallow: /*, is the same as Disallow: /.

Status codes, size and caching

• robots.txt applies per host and protocol. https://www.example.com/robots.txt and https://example.com/robots.txt are different files. Redirects are followed for a limited number of hops.
• If robots.txt returns a 4xx error (other than 429), Google treats the site as having no restrictions.
• If it returns a 5xx error or times out, Google pauses crawling and falls back to its last good copy for a while. A broken robots.txt can therefore stall crawling even though it contains no Disallow at all.
• Google usually caches robots.txt for up to 24 hours, so a fix takes time to be noticed unless you request a recrawl.
• Google stops reading after 500 KiB; anything beyond that is ignored.

Step 3: Check WordPress’s virtual robots.txt versus a physical file

WordPress does not ship a robots.txt file. When a request for /robots.txt finds no real file, the server passes it to WordPress, which generates one. In current versions the default output is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

This is the generated file content, not a command. The Sitemap line appears when core sitemaps are active and the site is public; SEO plugins usually replace it with their own sitemap URL. If a physical robots.txt exists in the site folder, the web server serves it directly and WordPress, along with every plugin’s robots.txt editor, is bypassed. Staging copies often bring a physical file with Disallow: / along.

ls -la robots.txt
wp eval 'do_robots();'

Run these over SSH in the site folder. ls -la lists the file with its size and date if it exists; “No such file or directory” means WordPress generates it. wp eval runs PHP inside WordPress, and do_robots() prints what WordPress would generate, including changes from plugin filters. If a physical file exists and differs from the generated version, the physical file is what Google sees. Good: either no physical file and a sensible generated one, or a physical file you wrote on purpose. Bad: a physical file with Disallow: /. Rename it to robots.txt.bak to fall back to WordPress’s version, or edit it.

The Search engine visibility setting

Settings, Reading, Discourage search engines from indexing this site stores blog_public = 0. Since WordPress 5.3 this outputs a noindex, nofollow robots meta tag on every page rather than a robots.txt block (older versions changed robots.txt instead). Confirm with wp option get blog_public: 1 is public, 0 is discouraged.

SEO plugins and server rules

Yoast SEO, Rank Math, All in One SEO and SEOPress can each set noindex per post type, taxonomy or individual post, and can edit robots.txt. A staging push that copies their settings tables brings any staging noindex with it. Also search for headers set outside WordPress:

grep -rni 'x-robots-tag' .htaccess wp-content/mu-plugins/ 2>/dev/null
grep -rni 'x-robots-tag' /etc/nginx/ 2>/dev/null

Run these over SSH, the first in the site folder and the second if you have root on your own server. -r searches recursively, -n shows line numbers and -i ignores case; 2>/dev/null hides “permission denied” noise. Good output is nothing, or rules you placed on purpose for specific files. A line like Header set X-Robots-Tag "noindex" applying to the whole site is your culprit. If you find nothing but curl still shows the header, it is coming from the host’s server config or a CDN rule, so check your CDN dashboard (for example Cloudflare Transform Rules) or open a ticket.

Step 4: Protect staging the right way

robots.txt is a request, not a lock. A blocked staging site can still be indexed from links, and the file is easy to carry over to live by mistake. A password stops everyone, Google included, and cannot leak a block onto production because it lives in server config that you do not migrate. Many hosts offer password protection for staging in the control panel; use that if available. Otherwise, create a password file outside the public folder:

htpasswd -c /home/ACCOUNT_USER/.htpasswd STAGING_USER

Run this over SSH. Replace ACCOUNT_USER with your hosting account’s home folder name and STAGING_USER with a login name you choose; it prompts for the password. -c creates the file and overwrites any existing one, so leave it off when adding a second user. Good output is “Adding password for user STAGING_USER”. “command not found” means the Apache utilities are not installed; your host’s panel may offer the same feature.

For Apache or LiteSpeed, add this to the staging site’s .htaccess:

AuthType Basic
AuthName "Staging"
AuthUserFile /home/ACCOUNT_USER/.htpasswd
<RequireAny>
  Require expr %{REQUEST_URI} =~ m#^/\.well-known/acme-challenge/#
  Require valid-user
</RequireAny>
Header set X-Robots-Tag "noindex, nofollow"

This is file content. AuthUserFile must be the full path you used with htpasswd. The RequireAny block lets Let’s Encrypt reach its validation path without a password so the staging certificate can renew, and requires a login for everything else. The Header line (needs mod_headers) adds a noindex for anything that slips through. Good result: the browser shows a login prompt and curl -sI returns 401. A 500 error means a module or directive is not allowed; remove the RequireAny block and test again, or ask your host. For Nginx, inside the staging server block:

auth_basic "Staging";
auth_basic_user_file /home/ACCOUNT_USER/.htpasswd;
add_header X-Robots-Tag "noindex, nofollow" always;
location ^~ /.well-known/acme-challenge/ { auth_basic off; }

This is Nginx config. always makes the header appear on error responses too, and the location block exempts certificate validation. Test with nginx -t before reloading; “syntax is ok” and “test is successful” are good output. Expect WordPress Site Health to report failed loopback requests on a password-protected site; that is normal on staging.

Step 5: Fix and recover

1. Remove the block: edit or rename the physical robots.txt, uncheck Search engine visibility, clear SEO plugin noindex settings, remove stray headers.
2. Purge every cache layer (plugin, server, CDN) so the old robots.txt and HTML stop being served. FG-50 covers cache layers.
3. Verify with curl, as in Step 1, from your own computer.
4. In Search Console open Settings, then the robots.txt report. It lists the robots.txt files Google found for your top hosts, when each was last fetched, and any problems. Use the menu next to a file to Request a recrawl after an urgent fix.
5. Run Test live URL in URL Inspection on the homepage and key pages; Crawl allowed and Indexing allowed should both say Yes. Then click Request indexing.
6. Resubmit your sitemap and use Validate fix on the affected reason in the Page indexing report.

Recovery time depends on how often Google crawls your site. A short block on a well-crawled site often recovers quickly once the fix is seen; a long block can take weeks to fully undo, and positions are not guaranteed to come back exactly as they were.

Advanced: test rules against real URLs before you deploy

The Search Console robots.txt report shows what Google fetched, but it does not test individual URLs against a draft file. For that, use Google’s open source robots.txt parser, published on GitHub as google/robotstxt, which Google describes as a lightly modified copy of the production code Googlebot uses. Clone it and build it with CMake as its README describes (cmake .. then make in a c-build folder), then test a draft file:

./robots ROBOTS_FILE Googlebot https://example.com/PAGE_PATH/

Run this on your computer (macOS, Linux or WSL with a C++ compiler and CMake installed) in the c-build folder. Replace ROBOTS_FILE with the path to your draft robots.txt and PAGE_PATH with a URL you care about. The second argument is the user agent token to evaluate. Good output ends with ALLOWED, for example user-agent 'Googlebot' with URI 'https://example.com/PAGE_PATH/': ALLOWED. DISALLOWED for a page you want in Google means the draft has a problem. The exit code is 0 for allowed and 1 for disallowed, which makes it easy to script a list of URLs. Run it for your homepage, a post, a product, a category, and a URL with a query string, so you see exactly how wildcards and precedence play out. For a quick check without building anything, the URL Inspection live test tells you whether the current live file allows a URL.

A minimal, safe WordPress robots.txt for most sites is the default shown in Step 3. Every extra line should have a reason you can explain. Blocking CSS, JavaScript or /wp-content/ stops Google rendering your pages properly and should not be done.

When to contact your host

Open a ticket when curl shows an X-Robots-Tag header or a robots.txt you cannot find in your files, when robots.txt returns 5xx or times out, or when the Search Console robots.txt report shows fetch errors. Include:

• The exact URLs tested and the full curl -sI output, including any CDN ray or request ID.
• The date and time, in UTC, when the block began or the errors appeared.
• A screenshot of the robots.txt report and the URL Inspection live test.
• Whether a physical robots.txt exists and what your .htaccess contains.
• Any recent staging push, migration, or security and CDN changes.

Our diagnostic tools can fetch robots.txt and response headers from outside your network. If the site itself is unreachable, use the Emergency Room checklist first.