You paste your website link into an AI assistant. The page opens normally in your browser, but the assistant says it cannot access it—or returns a description that ignores most of the page.
That does not automatically mean your website is broken. It also does not prove that the assistant has read it.
ChatGPT, Gemini or Claude may fail to read a website because web access is unavailable in that conversation, the request is blocked, the URL fails, or the retrieved version is missing the relevant content. Sometimes the assistant can read the page but does not find it independently in search. These are different problems with different fixes.
The fastest useful response is to identify which failure you have before changing your website.
This guide starts with checks any business owner can perform, then explains what to ask a developer or hosting provider to investigate. You do not need to install an “AI SEO” plugin to begin.
Start here: what exactly is failing?
Choose the description closest to what you see.
| What happens | First thing to check | What it does not establish |
|---|---|---|
| The assistant says it cannot browse any website | Search tools, account limits and workspace settings | That your website blocks it |
| Other public sites work, but your URL fails | The exact URL, public access and request logs | Which security rule is responsible |
| It reads headings but misses prices or product details | Whether those details are present in the retrieved content | That the entire website is unreadable |
| It reads your supplied URL but cannot find your business in search | Discovery, identity and indexing evidence | A direct-access failure |
| It reads the page but gives the wrong business facts | The claim and its supporting sources | A crawler problem |
If the last row describes your situation, use the companion guide to tracing and correcting inaccurate business details in AI answers. This article focuses on access and retrieval.
First, separate reading, finding and recommending
An assistant can successfully open a page without choosing it as a source for a broader question. It can also find a search result without successfully retrieving the complete destination page.
Think of three separate questions:
- Access: Can the relevant tool retrieve this exact public URL?
- Discovery: Can the assistant find the correct business or page without being given its URL?
- Selection: Does it choose that business or page when answering a particular question?
“ChatGPT does not recommend my company” is not evidence that ChatGPT cannot read the website. Likewise, a successful summary of your homepage is not proof that every product, price or policy page is accessible.
For a broader accuracy check after retrieval works, use the website facts test across ChatGPT, Gemini and Claude.
Step 1: check whether the conversation has web access
Before asking your developer to change anything, verify that the assistant can use the web in the current experience.
In ChatGPT
OpenAI documents automatic and manually requested web search. If search is missing, its troubleshooting advice includes starting a new conversation, checking the tools menu, confirming the intended account and reviewing usage or workspace restrictions. Interface labels can change.[1]
Ask it to search a known public reference page as a control, then test your exact URL. Keep the questions comparable.
If both fail, investigate the tool or account first. If the control succeeds and your page fails, you have a more specific website-access question.
In Claude
Anthropic documents web search and direct URL retrieval through web fetch. Search availability can depend on workspace controls and the current interface; some experiences search automatically rather than showing a separate toggle.[2]
Do not assume that pasting a link proves a successful fetch. Look for observable search or fetch activity where the interface provides it, then check the content actually returned.
In Gemini
Use a supported experience with web information available, request the specific page and inspect the answer's links. Google notes that Gemini responses do not always include sources or related links. Their absence alone does not identify the cause of a failure.[3]
Keep Gemini Apps distinct from AI Overviews and AI Mode in Google Search. Testing one does not establish how the others treat your website.
Use this diagnostic prompt
Use web access to retrieve this exact public URL: [URL].
From this page only, report:
1. Its title or main heading
2. The main product or service
3. One short excerpt supporting that description
4. The URL used as evidence
Do not substitute another website or answer from general knowledge.
If retrieval fails, state that it failed. Report an error or status
only if the tool actually exposed it; otherwise write “Cause unknown.”The prompt asks for evidence, but the assistant's explanation is still not a server diagnostic. “Your firewall blocked me” remains an unverified hypothesis unless logs or tool output support it.
Step 2: test the exact URL as a public visitor
Open the same URL in a private browser window while logged out of your CMS and other relevant accounts.
Check whether it shows the intended content without requiring:
- A website login or member session.
- A staging password.
- A CAPTCHA or browser challenge.
- A location selection that hides the main content.
- A consent interaction before the relevant information appears.
Your normal browser may have cookies, an authenticated session or a previously passed challenge. A server-side fetch may have none of those.
Also check the address itself. A copied link might point to a preview, expired share URL, account page or temporary redirect rather than the public page you intended.
What to do: use the permanent public URL. If the content should remain private, provide an authorised excerpt or file for that conversation instead. That helps with the immediate task; it does not make the website publicly discoverable.
Step 3: check the response, not just whether the browser looks normal
A developer should inspect the HTTP response for the failing URL. The status code is the server's basic description of what happened.
| Response | Plain-English meaning | Next investigation |
|---|---|---|
200 | The server returned a response marked successful | Confirm the body contains the intended page |
301 or 308 | A permanent redirect | Follow it and inspect the final destination |
302 or 307 | A temporary redirect | Check whether it leads to login, region selection or another unexpected page |
401 | Authentication is required | Confirm whether the URL should be public |
403 | The request was refused | Inspect security or access rules |
404 or 410 | The URL is missing or intentionally removed | Correct the URL or use a relevant replacement |
429 | The request was rate-limited | Inspect request limits and retry behaviour |
5xx | The server or an upstream service failed | Investigate hosting, application or gateway errors |
A 200 response is only a starting point. Its body could contain “Please enable JavaScript,” a challenge screen, an error message or an empty application shell.
Google documents a similar distinction in its crawling guidance: a page can return a success status while behaving like an error page, known as a soft 404. That is Google-specific indexing terminology, but inspecting the response body is also a useful retrieval diagnostic.[4]
Do not use a HEAD request alone to conclude that the content is available. Inspect an ordinary GET request as well, because servers can treat the methods differently.
Optional developer check
If curl is available, this command follows redirects and saves response headers and HTML for inspection:
curl --location --max-redirs 10 --connect-timeout 10 --max-time 30 \
--dump-header response-headers.txt \
--output response-body.html \
"https://example.com/service/"Replace the example with the public URL you own or are authorised to test. The filenames are diagnostic outputs, not website files to upload. On Windows, use curl.exe if curl resolves to another command.
This tests your machine's request. It does not reproduce an assistant's network, identity, location or browser capabilities.
Step 4: inspect firewall, CDN and security-plugin events
If your page works for visitors but fails for a particular retrieval path, check the systems in front of your website.
A CDN, web application firewall or WordPress security plugin may challenge unfamiliar requests, limit automated traffic or block particular networks. That can be an intended protection or a false positive. The logs should help distinguish the two.
Ask the hosting or security team to inspect the time of the failed attempt. Provide the exact URL, timestamp and timezone, assistant used, and any request or error identifier shown.
Look for the matched rule, action taken, response code and available request details. A request reaching the CDN may never reach the origin server, so check both layers when relevant.
Do not disable all bot protection to test a theory. If a legitimate request is being refused, make the smallest justified change using the provider's verification guidance and your security requirements.
Changing a local user-agent string to resemble an AI crawler is not proof that the real crawler can access the page. User-agent strings can be copied, while security decisions may depend on network verification and other signals.
Step 5: review crawler permissions by purpose
“Allow AI bots” is too vague to be a useful instruction. Different agents serve different purposes.
| Provider | Agent or control | Documented purpose |
|---|---|---|
| OpenAI | OAI-SearchBot | Surfacing websites in ChatGPT search |
| OpenAI | GPTBot | Crawling content that may be used for model training |
| OpenAI | ChatGPT-User | Certain user-initiated website visits |
| Anthropic | Claude-SearchBot | Search-related crawling |
| Anthropic | ClaudeBot | Collecting content that may contribute to model training |
| Anthropic | Claude-User | Retrieval at a user's direction |
OpenAI and Anthropic document these distinctions. Training preferences and search-related access should be deliberate, separate decisions. OpenAI also notes that robots.txt rules may not apply to some user-initiated ChatGPT-User actions.[5][6]
Google documents Googlebot for Search and a separate Google-Extended control covering specified Gemini training and grounding uses. Google-Extended does not control inclusion or ranking in Google Search. Do not treat it as a substitute for Googlebot access.[7]
Have a developer review the effective rules for the exact hostname and path. A staging subdomain and your main domain can have different configurations.
Avoid copying a generic robots.txt example over the existing file. It may remove rules that your site needs.
Step 6: check whether the important content is actually retrieved
Sometimes access works, but the answer ignores the details you care about.
For example, a page may deliver its heading immediately while loading the price table later through JavaScript. A human sees the completed page; a basic fetch may receive only the initial shell.
Compare three views where you can:
- The content visible to a logged-out visitor.
- The initial HTML returned by the server.
- The content exposed by the assistant's retrieval tool, if available.
Search for one distinctive sentence or product specification. Is it present in each view?
Google documents crawling and rendering as separate stages in its JavaScript guidance. Its ability to render a page should not be generalised to every AI fetch tool. Equally, avoid claiming that every AI-assisted browser is unable to execute JavaScript.[8]
Possible action: serve essential public information in meaningful initial HTML through server rendering or static generation, where appropriate for the site. Keep the same facts visible to visitors. Do not create misleading bot-only offers.
Check information hidden behind interactions
Inspect prices, specifications and policies that appear only after a tab click, variant selection, calculator input or API response.
An accordion is not automatically a problem. Its text may already be in the HTML. Determine whether the content is present before deciding that the design needs changing.
For product variants, use an explicit variant URL or a clear public specification table when the platform supports it. Do not expect a generic product-page summary to establish the price and stock of every option.
Step 7: inspect images, PDFs and embedded content
If key business facts exist only inside an image, scanned brochure, embedded viewer or external iframe, a text retrieval path may not expose them.
That does not mean AI can never interpret images or PDFs. It means you need to test the particular file, tool and mode rather than assuming that every website visit includes visual analysis.
For information buyers need before enquiring, publish a readable on-page version where appropriate:
- List services as text rather than only inside a banner.
- Provide important product specifications in a table.
- Explain delivery coverage and conditions on the relevant page.
- Summarise a downloadable catalogue and link to the full document.
Google's guidance for its Search AI features explicitly includes making important content available in textual form. That recommendation does not require removing useful images or downloads.[9]
Step 8: check discovery only after direct access is understood
If the assistant can read your supplied page but cannot independently find the business, move to discovery checks.
For Google, inspect the URL in Search Console. Review its index status, canonical URL and any unintended noindex rule. Google documents noindex as an indexing control; it is not the same thing as refusing an HTTP request.[10]
The distinction matters. A page may be technically retrievable while intentionally excluded from a search index. Conversely, a discovered URL may lead to content a particular fetch tool cannot access.
Review internal links, current sitemaps, consistent business identity and live canonical destinations. These are sensible discovery checks, not a guarantee of selection by an assistant.
Search Console describes Google's systems. It does not certify that ChatGPT or Claude has retrieved the same page.
A WordPress checklist before installing another plugin
If you use WordPress, begin with the implementation you already have:
| Check | What to inspect |
|---|---|
| Public visibility | Maintenance mode, password protection and membership restrictions |
| Indexing settings | Reading settings and SEO-plugin robots output on the live page |
| Security | Firewall events, bot challenges and rate-limit rules |
| Rendering | Whether service copy and specifications appear in initial HTML |
| Cache | Whether logged-out visitors receive an old or incomplete version |
| Redirects | Old domains, permalink changes and HTTP-to-HTTPS behaviour |
| Mobile or regional output | Whether different visitors receive materially different content |
Check actual responses after changes. A setting that looks correct in the admin area does not establish what your CDN or cached page serves publicly.
For quick on-page checks, Marketer Companion can help inspect metadata, canonicals and structured data. Those checks do not reproduce an AI assistant's fetch or replace server-log investigation.
How to tell whether a fix worked
Retest the original failure with the same URL and prompt. Use a fresh conversation, record the visible model or mode and preserve the full output.
I would repeat the test three times as a practical consistency check. That is an operating choice, not a statistical benchmark.
Keep three outcomes separate:
- Access: did the retrieval path return the intended content?
- Extraction: did the assistant identify facts actually present on that page?
- Discovery: did it find the correct page without a supplied URL?
If a firewall correction restores direct retrieval, that is a useful result. It does not prove that the business will now appear in recommendation lists or that an index has refreshed.
What to send your developer or hosting provider
Use this brief to avoid a vague request to “make the site AI-friendly”:
Public URL:
Assistant and visible mode:
Test date, time and timezone:
Exact prompt and full error/answer:
Does a known public control URL work in the same session?
Does this URL open while logged out?
Any visible request/error ID:
Please investigate the response status and body, redirect destination,
CDN/firewall events, origin errors and whether the essential text
is available to the relevant retrieval path.
Please identify the cause before changing crawler permissions or
security rules, and document the specific change and retest result.Common fixes that do not prove the problem is solved
Adding schema: useful when it accurately describes the page, but it cannot make an unavailable server respond or pass a login screen.
Adding llms.txt: does not establish that a specific assistant fetched the page. Google says special AI text files or markup are not required for its Search AI features.[9]
Submitting a sitemap: can support discovery in systems that use it. It does not override authentication, repair errors or guarantee inclusion.
Pasting the entire page into chat: helps the assistant analyse supplied text. It is a separate test from retrieving the live website.
Correcting the assistant in the same conversation: may improve the next answer in that chat. Retest independently before calling the public issue resolved.
Frequently asked questions
Why can ChatGPT read other websites but not mine?
The problem may involve your exact URL, request restrictions, response content or the retrieval path. Compare a control page, inspect the failing request and check the body returned. Do not infer a specific firewall cause from that symptom alone.
Why does Claude say it cannot access a link?
The link may fail, search or fetch may be unavailable, or the site may restrict the request. Anthropic documents web fetch, but that does not guarantee every URL can be retrieved. Check the current experience and the specific attempt.[2]
Does ranking in Google mean AI can read my site?
No. Ranking is evidence about a page's presence in a particular search system. It does not establish that a separate assistant's current fetch succeeds or that it receives the same content.
Does blocking GPTBot stop ChatGPT from finding my site?
OpenAI documents GPTBot and OAI-SearchBot as separate controls for potential training use and search. Review the current documentation and your effective configuration rather than treating all OpenAI user agents as one permission.[5]
Can AI read JavaScript websites?
It depends on the tool and mode. A non-rendering fetch, search index and interactive browser can expose different content. Test the essential information in the relevant retrieval path instead of applying one rule to every AI product.
Should I remove my firewall or CAPTCHA?
Investigate the specific block first. If access is unintentionally restricted, make a narrow, verified correction consistent with your security needs. There is no reason to remove site-wide protection merely because an assistant reports a failed visit.
How long until an assistant sees the fixed page?
There is no universal refresh deadline. A direct retrieval and search-discovery result may update differently. Confirm the corrected response, then monitor each separately without promising a fixed recovery period.
Your next step
Test one important public page, a known control URL and a prompt that asks for verifiable page content. Identify whether the failure involves tools, access, missing content or discovery.
Once retrieval works, move to the business-facts audit. If the assistant retrieves the page but misstates the offer, follow the source-tracing and correction guide.
For help investigating the website itself, request an AI-ready SEO audit. I will focus on the evidence needed to identify the problem and prioritise the next action.
- Can AI understand your business? – the website facts test: reference sheet, three test tracks and scoring
- Why can't ChatGPT read my website? (this guide) – diagnose access, redirects, firewalls and missing content
- AI gets your business details wrong – trace the source and correct the facts
- Why does ChatGPT show old information? – find the outdated source and track the correction
- Domain changes, rebrands and AI search – preserve sources, test old links and identity after a move
- Can AI read your product catalogue? – variants, prices, units and availability, checked
- Does AI know where you operate? – offices, service areas, delivery and remote markets
Can't tell why AI can't read your site?
I'm Jayant Solanki, an SEO, GEO and automation strategist. I investigate the response, redirects, CDN and firewall events and rendering behind a failed AI fetch, then tell you the specific fix.
Ranked #1 for "metal buildings" · +30% YoY organic traffic · Evidence-labelled research
Sources
Documentation checked on 17 October 2026. The troubleshooting order, prompts and retesting suggestions are recommendations for this guide, not platform-endorsed standards. Illustrative examples are not client results.
- OpenAI: Searching the web with ChatGPT.
- Anthropic: Enable and use web search.
- Google: View related sources from Gemini Apps.
- Google Search Central: Troubleshoot crawling errors.
- OpenAI: Overview of OpenAI crawlers.
- Anthropic: Crawler purposes and site-owner controls.
- Google: Common crawlers, including Google-Extended.
- Google Search Central: JavaScript SEO basics.
- Google Search Central: AI features and your website.
- Google Search Central: Block indexing with noindex.
Jayant Solanki
AI-Ready SEO, GEO & AIO strategist based in Indore, India. He works across technical SEO, analytics, website development and marketing automation for eCommerce, local-service and global businesses.
More about Jayant →