Verify that important content can be requested, understood and discovered. Technical access is a prerequisite for retrieval, not a guarantee of citation.
Test the actual response
Request key URLs and inspect status codes, redirects and the final document. A page that looks correct in your browser can still return an error to another client. Check production rather than assuming the local build represents the deployed response.
Prefer stable canonical URLs and redirect duplicate or retired routes deliberately. Avoid redirect chains that make access unnecessarily fragile. Confirm that assets needed to interpret essential content are available too.
Review crawler controls separately
Inspect robots.txt and page-level indexing directives. Controls differ across products and purposes; a search crawler and a training crawler are not interchangeable. Consult current operator documentation before deciding which agents to allow.
Robots rules express crawl preferences, not access control for private data. Keep confidential content behind real authentication. Do not block an entire site because one crawler policy needs refinement.
Keep important content in the document
Verify that titles, headings, article text and links exist in the delivered HTML where practical. A static or server-rendered page reduces dependence on client execution for those essentials.
Check internal navigation, descriptive anchors and sitemap entries. A sitemap is a discovery aid; it does not guarantee indexing or override an inaccessible page. Avoid adding URLs that redirect, return errors or intentionally remain unindexed.
Make the checklist repeatable
Run these checks after major deployments and route changes. Keep a small set of representative pages: home, a category page and a full article. Test both desktop and mobile usability so accessibility is not sacrificed for retrieval.
If an AI product still omits a page after access checks pass, continue with relevance, evidence and source analysis. Repeatedly changing robots rules is not a substitute for diagnosing the actual gap.
Your next steps
- Check final HTTP responses and canonical URLs.
- Review search and training crawler policies independently.
- Inspect delivered article text without JavaScript.
- Validate sitemap entries and internal links.
Source notes & further reading
These primary references support the technical and editorial context. The practical recommendations above are Arrow Signal’s editorial interpretation, not platform guarantees.