The AI Crawler Reality Report
769 training/ingest-class requests for every retrieval-class request
A seven-day audit of 513,534 requests shows the difference between being crawled and being distributed. A separate ChatGPT Search baseline found Nice Pick in zero of ten source lists.
Executive verdict
Training crawl is vanity. Retrieval crawl and citations are distribution.
Nice Pick received 254,595 requests from crawlers we classify as training or bulk-ingest oriented. Retrieval/search/user-fetch-class crawlers made 331. OpenAI's OAI-SearchBot accounted for 256 requests and encountered zero 5xx responses. Access was not the obvious problem. In a separate manual baseline on July 3, Nice Pick appeared in zero of ten ChatGPT Search source lists for a fixed set of software-comparison prompts.
The numbers
The audit window ran from July 8 at 17:41 UTC through July 15 at 17:41 UTC. Training-classified traffic made up 49.58% of all observed requests. Retrieval-classified traffic represented 0.06%.
- Cloudflare-reported requests
- 513,534
- Training/ingest class
- 254,595
- Retrieval class
- 331
- Training to retrieval
- 769:1
| Measure | Requests | Share of all traffic |
|---|---|---|
| All Cloudflare-reported requests | 513,534 | 100% |
| Training/ingest-class crawlers | 254,595 | 49.58% |
| Retrieval-class crawlers | 331 | 0.06% |
| OAI-SearchBot | 256 | 0.05% |
| OAI-SearchBot 5xx responses | 0 | — |
Half the traffic was from training/ingest-class user agents. Six-hundredths of one percent was from retrieval-class user agents. Quite the audience. Shame about the distribution.
What the data does—and does not—show
The audit shows that crawler attention is profoundly uneven. Systems associated with bulk collection requested Nice Pick hundreds of times more often than systems associated with live retrieval.
It does not show that training crawl suppresses retrieval, that a crawler used a fetched page in a model, or that more retrieval requests would automatically produce citations. A crawler request proves a request. It does not prove recommendation, attribution, or traffic.
The zero-error OAI-SearchBot result matters for a narrower reason: during this window, the crawler reached us without a server-side failure. That removes one convenient excuse. It does not establish indexing, ranking, selection, or citation.
Crawl is the top of the funnel, not the win
Raw bot traffic belongs in an infrastructure report, not a victory speech. Each stage below can fail independently.
- 01AccessibleThe crawler is allowed and the server responds.
- 02FetchedThe crawler requests the page.
- 03RetainedThe system keeps or indexes the page.
- 04SelectedThe page becomes evidence for an answer.
- 05CitedThe answer attributes the page.
- 06VisitedA person follows the citation.
Methodology and limits
We queried Nice Pick's Cloudflare edge request aggregates in one-day windows, then grouped requests by user-agent identity and HTTP response status. We classified known user agents by their apparent function: training or bulk ingest, live search and retrieval, search engine, SEO tool, other bot, or unclassified browser-like traffic.
- User-agent classification is descriptive, not forensic. User agents can be spoofed, and a crawler's identity does not prove how its operator ultimately used a page.
- Cloudflare's daily grouped query returned the 300 highest-volume user agents. Those rows covered 513,365 of 513,534 requests; 169 long-tail requests were outside the classifier result set.
- Cloudflare Adaptive Groups are analytics estimates, not raw access logs. Requests are not unique pages, people, referrals, or citations, and repeated asset fetches may count separately.
- The 0/10 baseline was one manual ChatGPT Search run on July 3, separate from the July 8–15 request window. Ten prompts establish “not yet”; they do not estimate the market.
- Server logs measure requests from retrieval-class user agents. They cannot reveal whether an answer engine cited the page in its final response. This one-site, one-week ratio should not be generalized to the web.
The aggregate figures, window, definitions, and limitations are available in the machine-readable data file.
Crawler-role definitions: OpenAI crawler documentation and Perplexity crawler documentation.
What publishers should measure instead
Crawler logs are useful diagnostic evidence. They are simply not the outcome. The useful measures are:
- Retrieval visits to specific, citable pages
- Access failures and server errors by crawler
- Repeat retrieval after substantive updates
- Citation rate across a stable query set
- Referral traffic from answer products
- Time from publication to retrieval, citation, and visit
What we will measure next
Nice Pick will preserve the July 3 baseline and repeat an expanded citation test after the planned observation window. We will compare retrieval to original research and selected comparisons, citation rate across a stable query set, attributable visits, and the gap between training crawl, retrieval crawl, and actual citation.
Future updates will retain dated methodology and intervention notes. Without stable queries and dates, an increase is merely a pleasant anecdote.
The takeaway
Cloudflare reported 254,595 training/ingest-class requests and 331 retrieval-class requests in seven days: approximately 769:1. OAI-SearchBot reached the site 256 times without a 5xx response. In a separate July 3 run, Nice Pick appeared in zero of ten ChatGPT Search source lists.
Training crawl is consumption. Retrieval is opportunity. Citation is the result.
Suggested citation
Nice Pick. “The AI Crawler Reality Report.” July 15, 2026. https://nicepick.dev/research/ai-crawler-reality-report