A 2026 guide to getting your company website into AI search results — how GEO, structure, and content help AI engines recommend your brand.
Quick answer: To get a B2B website into AI search, audit five things: robots.txt and CDN access for AI search crawlers such as OAI-SearchBot and PerplexityBot, content in the server-rendered HTML, accurate Organization structured data, answer-first page formats and measurement. GPTBot and Google-Extended govern AI training, not search visibility. llms.txt is optional.
When a B2B buyer asks an AI assistant to shortlist suppliers, your website is either readable to that assistant or it is not. Most of the reasons a site is invisible are technical and fixable in days: a crawler blocked by habit, content that only appears after JavaScript runs, a CDN rule nobody remembers switching on. This is the audit we run on company websites, with each check tied to what the platforms themselves document.
Allow the crawlers that power AI search answers, and make a separate, deliberate decision about the ones that collect training data. The two groups are controlled by different user agents, so you can say yes to one and no to the other.
| User agent | Operator | What it actually controls (per official docs) | Follows robots.txt? |
|---|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode; Google says meeting its technical requirements "covers you for search generally, including AI formats" (Google) | Yes | |
| Google-Extended | Use of content for training future Gemini models and for grounding in the Gemini app and Vertex AI; it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google) | Yes (it is a robots.txt token, not a separate crawler) | |
| OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT's search features; changes take "~24 hours" to apply (OpenAI) | Yes |
| GPTBot | OpenAI | Content that "may be used in training" OpenAI's foundation models | Yes |
| ChatGPT-User | OpenAI | Pages fetched when a user asks ChatGPT a question | "robots.txt rules may not apply" |
| PerplexityBot | Perplexity | Surfacing and linking websites in Perplexity results; "not used to crawl content for AI foundation models" (Perplexity) | Yes |
| Perplexity-User | Perplexity | Pages fetched for a user's question | "Generally ignores robots.txt rules" |
| ClaudeBot, Claude-SearchBot, Claude-User | Anthropic | Model training, search result quality and user-requested fetches respectively (Anthropic) | Yes, all three |
Two details catch people out. First, blocking GPTBot does not remove you from ChatGPT search; OAI-SearchBot does that. Second, Google-Extended is not purely a training switch: because it also covers grounding in the Gemini app, a B2B firm that wants to be named in Gemini answers should think carefully before blocking it. Bing is the other crawler not to forget, since Microsoft's AI Performance report tracks citations in Copilot and Bing's AI summaries.
A sensible default for most B2B sites that want AI visibility:
No. Google's own robots.txt guide says it is "not a mechanism for keeping a web page out of Google" and that its rules "may not be supported by all search engines". In August 2025 Cloudflare reported that Perplexity used undeclared crawlers impersonating Chrome when its declared crawler was blocked, while ChatGPT-User "fetched the robots file and stopped crawling when it was disallowed". The reverse problem is just as common: on 1 July 2025 Cloudflare announced it was "changing the default to block AI crawlers unless they pay creators". If your site sits behind Cloudflare or another CDN, check its bot settings as well as robots.txt. A permissive robots.txt cannot help if the firewall refuses the request.
If a fact only appears after JavaScript runs, most AI crawlers will never see it. Vercel's analysis of crawler traffic (December 2024) found that "none of the major AI crawlers currently render JavaScript", naming OpenAI's three agents, Anthropic's ClaudeBot and PerplexityBot among others. OpenAI's crawlers fetched JavaScript files in 11.50% of their requests but did not execute them. Google's Gemini is the exception, because it uses Googlebot's rendering.
Google itself still recommends that "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript". The test is simple: open the page source (not the inspector) or fetch the URL with JavaScript disabled, and check that headings, body copy, prices, FAQs and contact details are all there.
Speed belongs in the same audit. Our own homepage, with a 3D WebGL hero, scored 27 on mobile Lighthouse. Code that re-measured element positions on every animation frame, a LinkedIn feed loading on page load and the full 3D engine loading on phones were the main causes. After fixing them it scores about 74, service pages went from 66 to 89, and blog posts score about 98.
Google lists page experience, including "latency of your experience", among its recommendations for AI search, so a slow site is a handicap in both classic and AI results.
It helps machines understand your pages, but it is not a ticket into AI answers. Google says there is "no special schema.org structured data that you need to add" for AI Overviews or AI Mode, and that any markup you use must match the visible content of the page (AI features documentation).
The most valuable markup for a B2B company is usually Organization structured data on the home or About page, which Google says helps it "disambiguate your organization in search results", including a sameAs list of your official profiles. Add Article markup with real authors and dates on insight pages, and Service markup on service pages. Keep expectations realistic for FAQ markup: since August 2023 Google shows FAQ rich results only for "well-known, authoritative government and health websites". Google also says unused structured data "does not cause problems", so there is no need to remove it.
It is optional and, today, low impact. The llms.txt proposal by Jeremy Howard dates from September 2024 and suggests a Markdown file at the site root that gives AI agents a clean summary and a list of key pages. It is a proposal, not an adopted standard.
In June 2025 Google's John Mueller wrote that "no AI system currently uses llms.txt", adding that server logs show AI systems fetching pages but not that file. Google's documentation says you do not need "AI text files" to appear in its AI features. Our view: add one if your CMS makes it trivial, but never ahead of crawler access, server-rendered content and accurate pages.
Pages that answer a specific question in the first sentence or two under a clear heading, then back it with specifics. Google notes that AI Overviews and AI Mode may use "query fan-out", searching several related subtopics before answering. That is why an Ahrefs study in March 2026 found only 38% of AI Overview citations came from pages ranking in the top 10 for the original query.
For a B2B site, the format checklist is practical:
The research on which content changes increase citations is covered in our GEO strategy guide, and how each platform selects sources in our guide to getting cited by ChatGPT, Gemini and Perplexity.
Use this table as the working checklist. Every item can be tested in minutes with a browser, a terminal and your webmaster tools.
| Check | How to test | Pass condition |
|---|---|---|
| AI search crawlers allowed | Read /robots.txt | No Disallow for Googlebot, Bingbot, OAI-SearchBot, PerplexityBot or Claude-SearchBot on public pages |
| Training crawlers decided | Read /robots.txt | GPTBot, Google-Extended and ClaudeBot rules reflect a documented decision |
| CDN and firewall | Check bot-management settings and server logs | AI search crawlers receive HTTP 200, not 403 or a challenge page |
| One hostname | Request both www and non-www versions | One returns 200; the other 301-redirects to it |
| Server-rendered content | View source or fetch without JavaScript | All headings, copy, prices and FAQs present in raw HTML |
| Speed | Mobile Lighthouse on each page type | Homepage, service, case study and blog templates all measured |
| Titles and descriptions | Crawl the site and export metadata | Titles within about 60 characters; descriptions neither truncated nor thin |
| Clean text | Search content for zero-width characters | None inside words |
| Organization markup | Rich Results Test on the home or About page | Valid, with sameAs links and details that match the page |
| Answer-first format | Read the first two sentences under each H2 | Each one answers its heading on its own |
| Sourced facts | Spot-check statistics | Every figure links to the page that states it |
| llms.txt | Request /llms.txt | Optional; if present, accurate and current |
| Measurement | Search Console and Bing Webmaster Tools | Both verified; Bing AI Performance and AI referral traffic reviewed monthly |
Run it in this order, because each step depends on the one before:
Every item in the checklist above is there because we found it broken on a real site, including our own, during audits in September and October 2026.
Our FAQ page loaded its 12 questions with JavaScript after the page rendered, so the raw HTML contained about 55 words. To a crawler that does not run JavaScript, the page was close to empty. We moved the questions into the HTML, taking it to 595 words, and added FAQ schema.
We also found that www.dxdasia.com served a full copy of the site with an HTTP 200 instead of redirecting. Two identical sites split signals and confuse any engine trying to work out which is canonical. We added a 301 to the main domain, and now check both hostnames on every site we audit.
The strangest find was invisible. Three of our own blog posts contained 695 hidden zero-width characters inside words, pasted in from another editor, so "data" was not a real match for search. A Singapore exhibition-display client had 2,447 of them across 9 posts. Nothing looked wrong on screen.
Verify every fix on the live page. On shared hosting an upload to a wrong-but-plausible folder succeeds silently, and re-downloading the file proves nothing. We check each change over the public HTTPS URL before ticking it off the checklist.
For a B2B website, AI search visibility is mostly an engineering and editorial hygiene problem before it is a marketing one. The platforms document what they need: allowed crawlers, readable HTML, accurate markup and clear content. None of them asks for special files or tricks, and Google says so explicitly. Fix access and readability first, because no content strategy can overcome a blocked crawler or an empty HTML page. Then put effort into content that answers buyers' real questions with specifics. If you would like us to run this audit on your site, our SEO team does it as a fixed-scope engagement, and our performance optimisation specialists handle the speed side. You can also contact us directly.
No. OpenAI documents GPTBot as the crawler for content that may be used to train its models, while OAI-SearchBot is the one used to surface websites in ChatGPT's search features. You can disallow GPTBot and allow OAI-SearchBot. OpenAI says robots.txt changes for OAI-SearchBot take about 24 hours to apply, so allow a day before re-testing.
Check your server or CDN access logs for user agents such as GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Claude-SearchBot, and note the status codes they receive. Repeated 403 responses or challenge pages usually mean a firewall rule is blocking them. Your CDN or hosting dashboard may also break traffic down by bot.
It can, badly, if content is rendered only in the browser. Vercel found that major AI crawlers, including OpenAI's and Anthropic's, do not execute JavaScript, and Google recommends server-side or pre-rendering because not all bots can run it. Frameworks such as Next.js, Nuxt and Astro can render pages on the server while keeping an app-like experience for visitors.
Usually not. Blocking search-facing crawlers such as OAI-SearchBot and PerplexityBot removes you from answers buyers increasingly use to shortlist suppliers. A middle path is to allow search crawlers, decide separately on training crawlers like GPTBot and Google-Extended, and keep genuinely confidential material behind a login, since robots.txt is a request rather than security.
Quarterly, and after every redesign, CMS migration, CDN change or new plugin. Most of the problems in this checklist are introduced by changes nobody connects to search: a new security rule, a JavaScript-heavy template or pasted content. A quick monthly look at Bing's AI Performance report and Search Console will flag sudden drops between full audits.