1. The crawlers cannot read your page at all
GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot do not execute JavaScript. They fetch your HTML and read whatever is in it, and nothing more.
If your site builds its content in the browser, which most React, Vue and Angular sites do by default, those crawlers receive an effectively empty page. Googlebot renders JavaScript and sees a complete site, so your rankings stay healthy while you are invisible to every assistant. Nothing in your analytics reports this, because a crawler that finds nothing does not send you traffic to be missing.
This is an engineering fix rather than a content one: server-side rendering, static generation or prerendering. There is no way to write your way out of it.
The short versionView source, or disable JavaScript and reload. If the page is empty, stop here and fix this first.
2. Your robots.txt is blocking the wrong bot
The most common version of this mistake is blocking OAI-SearchBot while meaning to opt out of model training. Those are different crawlers with different jobs: GPTBot collects training data, OAI-SearchBot powers ChatGPT search. Blocking the second removes you from ChatGPT's answers entirely while doing nothing about training.
The same trap exists on the Anthropic side, where ClaudeBot trains and Claude-SearchBot answers, and with Google-Extended, which is a robots.txt token governing Gemini training and not a crawler you can block from AI Overviews.
Check the file rather than trusting your memory of it. These rules are often added years ago by somebody who has left, or inherited from a template.
The short versionRead your robots.txt line by line. Training bots and answering bots are different, and blocking the answering ones is self-inflicted.
3. Nothing on your site says plainly what you do and where
An assistant answering "who does this near me" has to resolve your business to an entity: a name, a category, a service area, a way to be contacted. If that information only exists implicitly, in a hero image or a tagline about passion and excellence, there is nothing to extract.
The fix is unglamorous. Say what you do, for whom, in which places, in plain sentences, on a page that exists to answer that. Add the structured data that states the same facts unambiguously for a machine.
This is the single most common content-side failure, and it is usually caused by a site written to impress visitors who already know what the company does.
The short versionA machine cannot infer your category from your tone. State it.
4. Your pages do not contain liftable answers
Answer engines retrieve passages, not pages: typically a heading and the text beneath it. A page that only makes sense read top to bottom performs badly, because it is never read top to bottom.
Practically, that means headings phrased as the question a buyer would actually ask, followed by a complete answer that survives being read in isolation. "Pricing" is a heading for a human who is already scrolling. "How much does an AI answering service cost?" is a heading that gets retrieved.
If every section of your page needs the section above it to make sense, you have written a brochure. Brochures do not get quoted.
The short versionWrite headings as questions and make each section stand alone, because it will be read alone.
5. You have no evidence, and evidence is a ranking factor here
The foundational peer-reviewed work in this field, the GEO paper presented at KDD 2024 by researchers from Princeton and Georgia Tech, tested which page changes lift visibility in generative engines. Three tactics each produced gains in the region of 30 to 40%: adding citations to sources, adding quotations from credible sources, and adding statistics. Keyword stuffing performed at or below baseline.
The same study found the gains were largest for lower-ranked sites, which makes this the rare channel where a small, well-evidenced site can be cited ahead of a vague page from a large one.
The catch is that the evidence has to be real. A statistic you cannot trace to whoever measured it is a liability, not an asset, and it sits in the exact place you are claiming authority.
The short versionCite, quote and quantify. And check that every number you use traces to a named study.
6. The plumbing underneath is broken in a boring way
Soft 404s, where a missing page returns a 200 status with a "not found" message, teach a crawler that every URL on your site is valid. Single-page applications produce these by default.
Then the ordinary discovery problems: no sitemap, or one that was never submitted; pages nothing links to, which crawlers therefore never reach; canonical tags pointing somewhere unexpected; a staging copy competing with production.
None of this is interesting and all of it is common. It is also the cheapest category to fix, which is a good reason to check it before commissioning content.
The short versionCheck status codes, sitemap submission and internal links. Boring failures are still failures.
