AI Crawler Report: How AI Agents Crawl a Utility Site
First-party measurement, not estimates. BoringToolsKit logs every request's agent class to our own database. This report publishes the aggregated 30-day picture — with downloadable CSVs under CC BY 4.0.
The 30-day picture
In the last 30 days, automated agents fetched boringtoolskit.com 33,487 times. AI assistant classes (ChatGPT, Claude, Gemini, Perplexity, other AI) account for 20,766 of those fetches — 62% of all bot traffic on a 963-tool utility site. ChatGPT fetched 5,339 pages across 2,198 unique paths; Claude fetched 4,405 across 2,208 — both broad, catalogue-wide crawls rather than single-page hits.
| Agent class | Fetches (30d) | Unique paths | Active days |
|---|---|---|---|
| bot | 12,721 | 1,699 | 16 |
| other-ai | 7,344 | 1,342 | 17 |
| chatgpt | 5,339 | 2,198 | 17 |
| claude | 4,405 | 2,208 | 10 |
| google-ai | 3,646 | 1,022 | 18 |
| perplexity | 32 | 18 | 9 |
What the tool-page data shows
AI assistants do not just hit homepages. ChatGPT's crawl runs the full catalogue through /all-tools/ and /sitemap.xml; Claude and Gemini fetch robots.txt before catalogues (orientation-first behavior, consistent with the robots-first doctrine we publish); and assistant classes spread fetches thinly across hundreds of long-tail tool pages — mortgage, rent, GPA, VO2max — one to nine fetches each over the window.
Two operational findings for site owners: API endpoints receive significant AI-class traffic (Google's fetchers lead there), and utility pages with concrete numbers and visible methodology are crawled repeatedly rather than once.
Downloads (CC BY 4.0)
- ai-crawler-daily.csv — fetches per agent class per day
- ai-crawler-top-paths.csv — most-fetched tool paths per AI class
- ai-crawler-summary.json — per-class totals in JSON
Cite as: BoringToolsKit Data Lab, "AI Crawler Behavior on a Utility Site," rolling 30-day window.
FAQ
Where does this data come from?
Our own server logs. BoringToolsKit records every request's user-agent class into a first-party database (Cloudflare D1). No third-party estimates, no panel data — these are actual fetches against boringtoolskit.com.
Is any user data included?
No. The dataset contains only automated-agent classes (crawlers and AI assistants), aggregated counts, and URL paths. No IPs, no visitor records, no personal data of any kind.
Why do AI assistants crawl a calculator site?
AI assistants fetch pages to ground answers (ChatGPT search, Claude web search, Gemini) and to train or evaluate retrieval. Utility pages with concrete numbers and methodology are prime extraction targets — which is exactly what this dataset measures.
Can I use this dataset?
Yes — the downloads are CC BY 4.0. Cite 'BoringToolsKit Data Lab' and link to this page. The CSVs are regenerated on a rolling 30-day window.