The prompt asks for evidence, but the assistant's explanation is still not a server diagnostic. “Your firewall blocked me” remains an unverified hypothesis unless logs or tool output support it.
``` ## Step 2: test the exact URL as a public visitor Open the same URL in a private browser window while logged out of your CMS and other relevant accounts. Check whether it shows the intended content without requiring: - A website login or member session. - A staging password. - A CAPTCHA or browser challenge. - A location selection that hides the main content. - A consent interaction before the relevant information appears. Your normal browser may have cookies, an authenticated session or a previously passed challenge. A server-side fetch may have none of those. Also check the address itself. A copied link might point to a preview, expired share URL, account page or temporary redirect rather than the public page you intended. **What to do:** use the permanent public URL. If the content should remain private, provide an authorised excerpt or file for that conversation instead. That helps with the immediate task; it does not make the website publicly discoverable. ## Step 3: check the response, not just whether the browser looks normal A developer should inspect the HTTP response for the failing URL. The status code is the server's basic description of what happened. | Response | Plain-English meaning | Next investigation | | --- | --- | --- | | `200` | The server returned a response marked successful | Confirm the body contains the intended page | | `301` or `308` | A permanent redirect | Follow it and inspect the final destination | | `302` or `307` | A temporary redirect | Check whether it leads to login, region selection or another unexpected page | | `401` | Authentication is required | Confirm whether the URL should be public | | `403` | The request was refused | Inspect security or access rules | | `404` or `410` | The URL is missing or intentionally removed | Correct the URL or use a relevant replacement | | `429` | The request was rate-limited | Inspect request limits and retry behaviour | | `5xx` | The server or an upstream service failed | Investigate hosting, application or gateway errors | A `200` response is only a starting point. Its body could contain “Please enable JavaScript,” a challenge screen, an error message or an empty application shell. Google documents a similar distinction in its crawling guidance: a page can return a success status while behaving like an error page, known as a soft 404. That is Google-specific indexing terminology, but inspecting the response body is also a useful retrieval diagnostic.[[4]](#src-4) Do not use a `HEAD` request alone to conclude that the content is available. Inspect an ordinary `GET` request as well, because servers can treat the methods differently. ### Optional developer check If curl is available, this command follows redirects and saves response headers and HTML for inspection: ``` curl --location --max-redirs 10 --connect-timeout 10 --max-time 30 \ --dump-header response-headers.txt \ --output response-body.html \ "https://example.com/service/"Replace the example with the public URL you own or are authorised to test. The filenames are diagnostic outputs, not website files to upload. On Windows, use curl.exe if curl resolves to another command.
``` This tests your machine's request. It does **not** reproduce an assistant's network, identity, location or browser capabilities. ## Step 4: inspect firewall, CDN and security-plugin events If your page works for visitors but fails for a particular retrieval path, check the systems in front of your website. A CDN, web application firewall or WordPress security plugin may challenge unfamiliar requests, limit automated traffic or block particular networks. That can be an intended protection or a false positive. The logs should help distinguish the two. Ask the hosting or security team to inspect the time of the failed attempt. Provide the exact URL, timestamp and timezone, assistant used, and any request or error identifier shown. Look for the matched rule, action taken, response code and available request details. A request reaching the CDN may never reach the origin server, so check both layers when relevant. **Do not disable all bot protection to test a theory.** If a legitimate request is being refused, make the smallest justified change using the provider's verification guidance and your security requirements. Changing a local user-agent string to resemble an AI crawler is not proof that the real crawler can access the page. User-agent strings can be copied, while security decisions may depend on network verification and other signals. ## Step 5: review crawler permissions by purpose “Allow AI bots” is too vague to be a useful instruction. Different agents serve different purposes. | Provider | Agent or control | Documented purpose | | --- | --- | --- | | OpenAI | `OAI-SearchBot` | Surfacing websites in ChatGPT search | | OpenAI | `GPTBot` | Crawling content that may be used for model training | | OpenAI | `ChatGPT-User` | Certain user-initiated website visits | | Anthropic | `Claude-SearchBot` | Search-related crawling | | Anthropic | `ClaudeBot` | Collecting content that may contribute to model training | | Anthropic | `Claude-User` | Retrieval at a user's direction | OpenAI and Anthropic document these distinctions. Training preferences and search-related access should be deliberate, separate decisions. OpenAI also notes that robots.txt rules may not apply to some user-initiated ChatGPT-User actions.[[5]](#src-5)[[6]](#src-6) Google documents Googlebot for Search and a separate Google-Extended control covering specified Gemini training and grounding uses. Google-Extended does not control inclusion or ranking in Google Search. Do not treat it as a substitute for Googlebot access.[[7]](#src-7) Have a developer review the effective rules for the exact hostname and path. A staging subdomain and your main domain can have different configurations. Avoid copying a generic robots.txt example over the existing file. It may remove rules that your site needs. ## Step 6: check whether the important content is actually retrieved Sometimes access works, but the answer ignores the details you care about. For example, a page may deliver its heading immediately while loading the price table later through JavaScript. A human sees the completed page; a basic fetch may receive only the initial shell. Compare three views where you can: 1. The content visible to a logged-out visitor. 2. The initial HTML returned by the server. 3. The content exposed by the assistant's retrieval tool, if available. Search for one distinctive sentence or product specification. Is it present in each view? Google documents crawling and rendering as separate stages in its JavaScript guidance. Its ability to render a page should not be generalised to every AI fetch tool. Equally, avoid claiming that every AI-assisted browser is unable to execute JavaScript.[[8]](#src-8) **Possible action:** serve essential public information in meaningful initial HTML through server rendering or static generation, where appropriate for the site. Keep the same facts visible to visitors. Do not create misleading bot-only offers. ### Check information hidden behind interactions Inspect prices, specifications and policies that appear only after a tab click, variant selection, calculator input or API response. An accordion is not automatically a problem. Its text may already be in the HTML. Determine whether the content is present before deciding that the design needs changing. For product variants, use an explicit variant URL or a clear public specification table when the platform supports it. Do not expect a generic product-page summary to establish the price and stock of every option. ## Step 7: inspect images, PDFs and embedded content If key business facts exist only inside an image, scanned brochure, embedded viewer or external iframe, a text retrieval path may not expose them. That does not mean AI can never interpret images or PDFs. It means you need to test the particular file, tool and mode rather than assuming that every website visit includes visual analysis. For information buyers need before enquiring, publish a readable on-page version where appropriate: - List services as text rather than only inside a banner. - Provide important product specifications in a table. - Explain delivery coverage and conditions on the relevant page. - Summarise a downloadable catalogue and link to the full document. Google's guidance for its Search AI features explicitly includes making important content available in textual form. That recommendation does not require removing useful images or downloads.[[9]](#src-9) ## Step 8: check discovery only after direct access is understood If the assistant can read your supplied page but cannot independently find the business, move to discovery checks. For Google, inspect the URL in Search Console. Review its index status, canonical URL and any unintended `noindex` rule. Google documents `noindex` as an indexing control; it is not the same thing as refusing an HTTP request.[[10]](#src-10) The distinction matters. A page may be technically retrievable while intentionally excluded from a search index. Conversely, a discovered URL may lead to content a particular fetch tool cannot access. Review internal links, current sitemaps, consistent business identity and live canonical destinations. These are sensible discovery checks, not a guarantee of selection by an assistant. Search Console describes Google's systems. It does not certify that ChatGPT or Claude has retrieved the same page. ## A WordPress checklist before installing another plugin If you use WordPress, begin with the implementation you already have: | Check | What to inspect | | --- | --- | | Public visibility | Maintenance mode, password protection and membership restrictions | | Indexing settings | Reading settings and SEO-plugin robots output on the live page | | Security | Firewall events, bot challenges and rate-limit rules | | Rendering | Whether service copy and specifications appear in initial HTML | | Cache | Whether logged-out visitors receive an old or incomplete version | | Redirects | Old domains, permalink changes and HTTP-to-HTTPS behaviour | | Mobile or regional output | Whether different visitors receive materially different content | Check actual responses after changes. A setting that looks correct in the admin area does not establish what your CDN or cached page serves publicly. For quick on-page checks, [Marketer Companion](https://thejayant.in/marketer-companion) can help inspect metadata, canonicals and structured data. Those checks do not reproduce an AI assistant's fetch or replace server-log investigation. ## How to tell whether a fix worked Retest the original failure with the same URL and prompt. Use a fresh conversation, record the visible model or mode and preserve the full output. I would repeat the test three times as a practical consistency check. That is an operating choice, not a statistical benchmark. Keep three outcomes separate: - **Access:** did the retrieval path return the intended content? - **Extraction:** did the assistant identify facts actually present on that page? - **Discovery:** did it find the correct page without a supplied URL? If a firewall correction restores direct retrieval, that is a useful result. It does not prove that the business will now appear in recommendation lists or that an index has refreshed. ## What to send your developer or hosting provider Use this brief to avoid a vague request to “make the site AI-friendly”: ``` Public URL: Assistant and visible mode: Test date, time and timezone: Exact prompt and full error/answer: Does a known public control URL work in the same session? Does this URL open while logged out? Any visible request/error ID: Please investigate the response status and body, redirect destination, CDN/firewall events, origin errors and whether the essential text is available to the relevant retrieval path. Please identify the cause before changing crawler permissions or security rules, and document the specific change and retest result.Adding schema: useful when it accurately describes the page, but it cannot make an unavailable server respond or pass a login screen.
``` **Adding llms.txt:** does not establish that a specific assistant fetched the page. Google says special AI text files or markup are not required for its Search AI features.[[9]](#src-9) **Submitting a sitemap:** can support discovery in systems that use it. It does not override authentication, repair errors or guarantee inclusion. **Pasting the entire page into chat:** helps the assistant analyse supplied text. It is a separate test from retrieving the live website. **Correcting the assistant in the same conversation:** may improve the next answer in that chat. Retest independently before calling the public issue resolved. ## Frequently asked questions ### Why can ChatGPT read other websites but not mine? The problem may involve your exact URL, request restrictions, response content or the retrieval path. Compare a control page, inspect the failing request and check the body returned. Do not infer a specific firewall cause from that symptom alone. ### Why does Claude say it cannot access a link? The link may fail, search or fetch may be unavailable, or the site may restrict the request. Anthropic documents web fetch, but that does not guarantee every URL can be retrieved. Check the current experience and the specific attempt.[[2]](#src-2) ### Does ranking in Google mean AI can read my site? No. Ranking is evidence about a page's presence in a particular search system. It does not establish that a separate assistant's current fetch succeeds or that it receives the same content. ### Does blocking GPTBot stop ChatGPT from finding my site? OpenAI documents GPTBot and OAI-SearchBot as separate controls for potential training use and search. Review the current documentation and your effective configuration rather than treating all OpenAI user agents as one permission.[[5]](#src-5) ### Can AI read JavaScript websites? It depends on the tool and mode. A non-rendering fetch, search index and interactive browser can expose different content. Test the essential information in the relevant retrieval path instead of applying one rule to every AI product. ### Should I remove my firewall or CAPTCHA? Investigate the specific block first. If access is unintentionally restricted, make a narrow, verified correction consistent with your security needs. There is no reason to remove site-wide protection merely because an assistant reports a failed visit. ### How long until an assistant sees the fixed page? There is no universal refresh deadline. A direct retrieval and search-discovery result may update differently. Confirm the corrected response, then monitor each separately without promising a fixed recovery period. ## Your next step Test one important public page, a known control URL and a prompt that asks for verifiable page content. Identify whether the failure involves tools, access, missing content or discovery. Once retrieval works, move to the [business-facts audit](https://thejayant.in/blog/can-ai-understand-your-business). If the assistant retrieves the page but misstates the offer, follow the [source-tracing and correction guide](https://thejayant.in/blog/fix-wrong-business-information-ai). For help investigating the website itself, [request an AI-ready SEO audit](https://thejayant.in/#contact). I will focus on the evidence needed to identify the problem and prioritise the next action. 1. [Can AI understand your business?](https://thejayant.in/blog/can-ai-understand-your-business) – the website facts test: reference sheet, three test tracks and scoring 2. **Why can't ChatGPT read my website?** (this guide) – diagnose access, redirects, firewalls and missing content 3. [AI gets your business details wrong](https://thejayant.in/blog/fix-wrong-business-information-ai) – trace the source and correct the facts 4. [Why does ChatGPT show old information?](https://thejayant.in/blog/why-chatgpt-shows-old-business-information) – find the outdated source and track the correction 5. [Domain changes, rebrands and AI search](https://thejayant.in/blog/ai-search-visibility-domain-change-rebrand) – preserve sources, test old links and identity after a move 6. [Can AI read your product catalogue?](https://thejayant.in/blog/can-ai-read-product-catalogue-correctly) – variants, prices, units and availability, checked 7. [Does AI know where you operate?](https://thejayant.in/blog/ai-business-location-service-area) – offices, service areas, delivery and remote markets ## Sources Documentation checked on 17 October 2026. The troubleshooting order, prompts and retesting suggestions are recommendations for this guide, not platform-endorsed standards. Illustrative examples are not client results. 1. OpenAI: [Searching the web with ChatGPT](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt). 2. Anthropic: [Enable and use web search](https://support.claude.com/en/articles/10684626-enable-and-use-web-search). 3. Google: [View related sources from Gemini Apps](https://support.google.com/gemini/answer/14143489?hl=en). 4. Google Search Central: [Troubleshoot crawling errors](https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors). 5. OpenAI: [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots). 6. Anthropic: [Crawler purposes and site-owner controls](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). 7. Google: [Common crawlers, including Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers). 8. Google Search Central: [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics). 9. Google Search Central: [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features). 10. Google Search Central: [Block indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing).