Which Tools Do AI Agents Actually Fetch?
A per-page view of AI-crawler behavior across our 971-tool catalogue — first-party measured, downloadable under CC BY 4.0.
The headline finding: sweepers, not searchers
Across 30 days, AI-class agents fetched 3,681 tool and content pages — but the distribution tells the real story. 500 distinct pages were each fetched 7-10 times, like clockwork: a scheduled catalogue sweep every 3-4 days, dominated by Amazonbot and masked Android user-agents. By agent class, tool-page fetches split as:
- other-ai — 3,609 tool-page fetches
- chatgpt — 40 tool-page fetches
- google-ai — 32 tool-page fetches
ChatGPT's tool-page fetches? Dozens, not thousands — concentrated on the catalogue index and a handful of finance pages. The popular narrative of "ChatGPT sending traffic to tools" describes grounding fetches that are real but rare; the daily reality of AI crawler traffic is scheduled infrastructure sweeps.
What gets fetched repeatedly
Category hubs (/calculators/finance/, /state-taxes/), conversion pairs (/convert/), and guides recur at the top of the per-path table. Pages with visible methodology and tabular data are re-fetched; thin pages are fetched once and dropped. Extraction-ready structure is what earns repeat reads.
Downloads (CC BY 4.0)
- ai-crawler-tool-paths.csv — fetches + active days per agent per path
- ai-crawler-daily.csv — daily fetches per agent class
FAQ
Which AI agents actually fetch tool pages?
The surprise: the bulk of 'AI' tool-page traffic comes from scheduled catalogue-sweeping crawlers (Amazonbot and masked Android user-agents), not from ChatGPT grounding. ChatGPT's tool-page fetches in this window number in the dozens, concentrated on /all-tools/ and a handful of finance pages.
How often do AI crawlers re-fetch the same page?
The dominant pattern is 7-10 fetches per path per month — a scheduled sweep of the entire catalogue every 3-4 days. Content pages are infrastructure to these agents, not one-time reads.
What does this mean for site owners?
Crawl budget matters differently than for search: these sweepers will read your whole catalogue regardless. Differentiation comes from what they extract — visible methodology, tables, and answer-ready structure — not from begging for one more fetch.