Independent web experiment
AI Internet Observatory
A small research space for the machine-readable web: what is published for agents, what gets requested, and what we can actually infer from those requests.
Latest daily measurement · 2026-09-17 UTC
165 verified AI-category requests
This is a complete UTC-day Cloudflare adaptive estimate, grouped by response status. The dated JSON record includes exact-path request groups for the isolated experiments.
A zero requires a successful complete query. These counts do not identify unique agents, prove discovery or show that an agent completed a task.
Browse all daily records
- 2026-09-17 UTC — 165 verified AI-category request estimates
- 2026-09-16 UTC — 191 verified AI-category request estimates
Latest measured week · 9–15 September 2026 UTC
1,447 verified AI-category requests
Cloudflare classified 1,386 requests as AI Crawler and 61 as the legacy AI Search category. 1,416 received HTTP 200. The previous measured week had 1,596 such requests.
These are adaptive request estimates from one site, last queried 2026-09-16. They are not unique agents, conversations, task completions, referrals or evidence of product demand. The classification scheme can change.
See daily counts and method
| UTC date | AI Crawler | AI Search | Total |
|---|---|---|---|
| 2026-09-09 | 250 | 14 | 264 |
| 2026-09-10 | 229 | 3 | 232 |
| 2026-09-11 | 245 | 5 | 250 |
| 2026-09-12 | 183 | 12 | 195 |
| 2026-09-13 | 156 | 4 | 160 |
| 2026-09-14 | 167 | 14 | 181 |
| 2026-09-15 | 156 | 9 | 165 |
Each day was queried separately from Cloudflare's HTTP adaptive analytics, filtering to its AI Crawler and legacy AI Search categories. Counts include every response status; they may be sampled estimates.
Permanent research record
Experiments
Each experiment has a stable page with its hypothesis, setup, observations, evidence and limits. Planned work is listed as planned; a request to a test page is not proof that an agent completed a task.
| No. | Experiment | Status | Observation |
|---|---|---|---|
| 001 | Agent trail | Running | Not yet measured |
| 002 | llms.txt discovery | Running | Not yet measured |
| 003 | HTML vs structured data | Planned | — |
| 004 | Machine-readable API discovery | Planned | — |
For site owners
Check what you publish for agents
Paste the public text of your robots.txt, llms.txt and one HTML page. The free readability check points out discoverability and structured-data issues and generates a report you can keep. It runs in your browser; your pasted content is not sent to this site.
Automated monitoring experiment
Monitor a domain automatically
Testing whether site owners want automated recurring checks of machine-readability: changes to robots.txt, llms.txt drift, and broken machine-readable resources. Proposed as a self-service weekly report ($19/month). No advisory calls, bespoke consulting, or manual reviews.