Why AI can't find your WordPress site

Most advice about getting cited by AI opens with the same diagnosis: your content is built in the browser and crawlers cannot run JavaScript. On WordPress that is almost never the problem. WordPress assembles a page on the server and sends finished HTML — classic themes, block themes and full site editing alike. Which is good news and awkward news at once: the easy explanation is off the table, and the real reason an engine cannot find you is something you cannot see by looking at your site.

There are four places it usually hides. One of them is a single checkbox whose behaviour changed in WordPress 5.3 — and almost every guide still describes the old behaviour.

First, rule out the setting that quietly deletes you

Settings → Reading → Discourage search engines from indexing this site. It sets the blog_public option to off. It exists so you can build in peace, it gets ticked on staging, and it survives the clone to production more often than anyone admits.

What every write-up tells you it does is out of date. They say it adds Disallow: / to your robots.txt. WordPress removed exactly that in 5.3 — the changelog entry on do_robots() reads "Remove the 'Disallow: /' output if search engine visibility is discouraged in favor of robots meta HTML tag." Since 5.7, wp_robots_noindex() runs on the wp_robots filter and, when blog_public is off, puts noindex, nofollow in the robots meta tag on every page of the site.

# The whole diagnosis, in one line. Any noindex here and nothing else matters yet.
curl -s https://yourdomain.com/ | grep -i '<meta name="robots"'

# Same check on a page you actually want cited, not just the homepage.
curl -s https://yourdomain.com/your-best-page/ | grep -i '<meta name="robots"'

If a noindex comes back, stop and fix that before touching anything else. And note that the Reading checkbox is not the only thing that can produce one: an SEO plugin can set noindex per post type or per archive, a theme can inject it, and a three-line wp_robots filter left in a child theme by whoever built the site will do it silently and forever.

Your robots.txt is not a file — until somebody makes it one

WordPress generates robots.txt at request time. do_robots() builds it, and the default body is small: a User-agent: *, a Disallow on the admin path, an Allow for admin-ajax.php, and a Sitemap: line added by core's sitemap class. Everything else you see in yours was appended through the robots_txt filter — that is the hook your SEO plugin's robots.txt panel writes into.

# What the internet actually gets. Compare this against your plugin's panel;
# if they disagree, a physical file is shadowing WordPress.
curl -s https://yourdomain.com/robots.txt

Allowing the right crawlers — and the layer above WordPress

Vendors split their crawlers along two independent axes, and the distinction decides whether a robots.txt line costs you anything. One token governs whether your content may be used to train future models — a content-rights decision. A different token governs whether the retrieval crawler may fetch your pages so you can be cited in an answer right now — a visibility decision. Separate strings, separate published IP ranges. Blocking one says nothing about the other.

The common mistake is to add GPTBot to the allow list and call it done. GPTBot is the training token; it does not gate a single ChatGPT citation. OAI-SearchBot does, and it is the one most copied snippets never mention. Match strings exactly — a User-agent line written for a token no crawler sends is a rule that reads like protection and does nothing.

You almost certainly have two sitemaps, and one of them is wrong

WordPress core has shipped XML sitemaps since 5.5. WP_Sitemaps serves an index at /wp-sitemap.xml, registers three providers by default — posts, taxonomies and users — and appends the Sitemap: line to the generated robots.txt itself.

Install an SEO plugin and it will usually switch core's off through the wp_sitemaps_enabled filter and serve its own index at a different URL. There is a detail in how core handles that which bites people: when sitemaps are disabled by filter, the rewrite rules stay registered specifically so the URL returns a 404. So /wp-sitemap.xml does not vanish — it becomes a dead address that used to work, and anything still pointing at it now points at nothing.

What WordPress renders, and the few places it stops

Content lives in post_content and comes back inside the HTML response. That is why the rendering panic does not transfer here — and why the test is genuinely simple. WordPress serves the same markup to everybody, so curl is a valid stand-in for a crawler in a way it is not on a JavaScript-rendered site.

# Pick a sentence you would want quoted, and prove it is in the served HTML.
curl -s https://yourdomain.com/your-best-page/ | grep -c "the exact sentence you want cited"

# 0 means it is being added after load, and no non-Google crawler will see it.

The exceptions are worth knowing because they are the only rendering failures WordPress actually produces:

• Load-more and infinite scroll. Tabs and accordions usually ship their content in the HTML and merely hide it with CSS, which is fine. Anything fetched on click or on scroll is not there at all.
• A headless setup. Point a JavaScript front end at the WordPress REST API and you have traded away the one thing WordPress was giving you for free. That is a rendering problem again, and it needs a rendering answer.
• Facts locked in images. A number that exists only inside an infographic is invisible to text extraction. Put it in a sentence as well.
• A stale full-page cache. Caching is not the enemy, but a crawler can be handed a copy generated before your last edit. Purge after publishing, and re-fetch the live URL rather than trusting the editor preview.

Schema: the usual problem is too much of it, not too little

WordPress core emits no structured data at all. Every bit of it on your site comes from a plugin or a theme, which is why the failure mode here is duplication rather than absence. Two SEO plugins left active produce two entity graphs making overlapping claims; a theme that adds its own Organization block on top produces a third. An engine resolving who you are then has several same-named nodes to choose between, which is the exact ambiguity structured data exists to remove. Run one source of schema and turn the others off.

The one rule that matters more than any type choice: schema must describe the page, not aspire to describe it. Marking up a review score, a price or an FAQ that a reader cannot find on the page is a claim that every system reading your site can check.

After the settings, the settings stop mattering

Everything above buys candidacy: the engine can reach your pages, read them, and index them. None of it buys selection, which is the separate question of whether the engine picks you over everything else it retrieved for that question. Conflating the two is why people fix all four things and see nothing move. We pull them apart in discoverability vs selectability.

WordPress creates its own version of the selection problem, and it comes from its greatest strength: publishing is nearly free, so sites accumulate. Forty near-identical posts circling one topic compete with each other and give an engine no reason to choose any of them. Eight pages carrying real numbers, a named process and a stated professional judgment give it a reason. The thing a model cannot synthesise from everywhere else is the thing that gets quoted.

Structure the writing so a passage can survive being lifted. Lead each section with the claim and follow with the evidence, keep one idea per paragraph, and make sure a paragraph still means something when it is quoted without the three above it. Most citations in AI answers also point at pages you do not own — roundups, review sites, community threads — which is slower work on a longer clock. Which sources AI engines cite maps that ground.

How to tell whether any of it worked

Not by asking an assistant once. Across 2,961 brand-recommendation prompts run against ChatGPT, Claude and Google's AI surfaces, the engines returned the same ordered brand list in under one percent of runs for an identical prompt. A single query measures the noise. It will tell you the fix failed on Tuesday and worked on Thursday, and both readings will feel convincing.

1. Freeze a prompt set first — fifteen to thirty questions a buyer would really type, written down before you change anything.
2. Run each one several times per cycle and keep every raw answer, with its date and the model version that produced it.
3. Tag each result with the engine and the surface. A model API answer and a Google AI Overview are different products with different citation behaviour; averaging them hides half your exposure.
4. Report ranges, not points. With a handful of runs per prompt the interval around any percentage is wide, and hiding that is how teams celebrate noise.
5. Wait a full refresh window before judging a change. Crawl and index refresh runs on each engine's clock — days to weeks, not hours.
6. Watch impressions, not just sessions. If you only count clicks, a channel that answers the question without one will always look like pure loss.

The short version for WordPress: read the live meta robots tag before you believe any setting, read the live robots.txt before you believe any plugin panel, pick one sitemap and one source of schema, and then accept that the technical layer was only ever the entry fee. For the baseline to measure the rest against, run the free audit.