When an AI crawler fetches a page from an ordinary website, no API is involved anywhere in the exchange. An API is a contract between two programs. GPTBot, ClaudeBot and PerplexityBot request HTML pages over HTTPS the way a browser does, and the site owner has to build nothing for it to work. Every component a developer would recognize in an API call is already present in that plain fetch. Standard hosting controls act on every one of them.
What Is an API in Plain Terms
The definition covers the agreed rules by which one program asks another for something and gets a predictable answer back. The word gets used loosely, most often as a shorthand for a URL that returns JSON, and that shorthand is what confuses site owners about AI access. JSON is only one format the contract can be written in, and HTML satisfies the same structure.
Salesforce shipped the first commercial web API on February 7, 2000, and it returned XML. eBay followed nine months later with a program open to a small set of partners. Because most of the modern definition is convention and not requirement, neither program looks anything like what the term suggests today.
The Parts of an API Request
Every request over HTTP has the same parts, no matter what the response body contains, and a crawler asking for a page sends all of them:
A method. GET to read something, POST to create, PUT to replace, DELETE to remove.
An endpoint. The URL the request goes to.
Headers. Metadata about the request, including who is asking and what formats they accept.
An optional body. The data being sent, for methods that send data.
A status code in reply. 200 for success, 404 for missing, 429 for too many requests, and so on.
A response body. JSON, XML, HTML, an image, a zip file.
What a REST API Is
REST was defined by Roy Fielding in a 2000 doctoral dissertation at UC Irvine, several years after he had helped write the HTTP specifications themselves. The core property is statelessness, so the server treats each request as brand new and remembers nothing about the client between calls, so identification has to travel with every single request.
Fielding was describing the web that existed in 1999, where HTML documents moved over HTTP between servers and people reading them. The dissertation barely mentions web APIs, and he has objected in public to the way the industry later repurposed the word. The web he described was the website itself. A site with no JSON layer on top of it is already what Fielding meant by the term.
Do AI Tools Use an API to Read Your Site?
For the overwhelming majority of sites, the answer is no. An AI crawler opens a connection to port 443 and sends GET with a path, then reads whatever HTML comes back. Agent-facing surfaces do exist, but only on sites where somebody deliberately built one.
A site owner controls four parts of that exchange:
The endpoint. The URL of the page being requested.
The identity header. The user-agent claim, visible in the access log.
The robots.txt file. Read by a compliant crawler from the root directory.
The status code. Set by the server config.
Your URL Is the Endpoint
The address of any page on a site is the endpoint an AI system requests, using GET as its method. The HTML comes back as the response body, the response headers are the metadata, and the ETag and Last-Modified values are the versioning part of the contract that tells a well-behaved client it already has the current copy. Those two headers are there so a conditional request can be answered with 304 Not Modified and no bytes of the page move at all.
In July 2024 a curl trace on Read the Docs caught a crawler following a 302 chain into a 5.8 MB zip file whose Last-Modified date was 2021 and whose ETag had not changed, and it re-downloaded that identical file hundreds of times. One crawler moved 73 TB in a single month, almost 10 TB of it in one day, and the bandwidth bill came to more than $5,000. Both headers were correct throughout, and the crawler read neither of them.
How AI Crawlers Identify Themselves
Most requests contain exactly one identity claim, the User-Agent header. OpenAI’s search robot announces itself as OAI-SearchBot/1.4 with a link to its documentation page appended, Anthropic sends ClaudeBot and Perplexity sends PerplexityBot/1.0. Since a header string is trivially forged, operators back it with a second signal a site owner can verify independently.
Web Bot Auth supplies the verification a user-agent string cannot, building on RFC 9421 HTTP Message Signatures, with the operator generating an Ed25519 keypair, publishing the public half at /.well-known/http-message-signatures-directory, and signing every outbound request with Signature and Signature-Input headers. The receiving server tests the signature against the published key and rejects anything that fails.
What robots.txt Does and Does Not Do
The robots.txt file has no enforcement behind it, so compliance is entirely at the discretion of the operator reading it. Per-agent Disallow rules work on the declared training crawlers from the major operators, and they do nothing about anything undeclared.
A documented carve-out covers user-initiated fetches. Perplexity-User generally ignores robots.txt rules because a person asked for the page, and the same rules may not apply to ChatGPT-User. Anthropic honors the non-standard Crawl-delay directive for ClaudeBot while Google has never supported it, so the same two lines in the same file produce different behavior from different operators.
The Status Codes an AI Crawler Gets Back
The server’s response codes are the other half of the contract, and they are the half a site owner controls completely. 200 serves the content and 403 refuses it at the firewall. 429 Too Many Requests, defined in RFC 6585, tells a client it has exceeded a rate limit and pairs with Retry-After to say for how long. 503 means the server could not take the request at all, and a rate limiter refusing a client returns it too.
Cloudflare added a new code to the vocabulary in July 2025 by reviving 402 Payment Required for pay-per-crawl. An unpaid crawler gets a 402 with pricing attached and can retry with signed payment credentials, over the same plain HTTP request the site was already answering.
AI Crawler User Agents You Will See in Your Logs
The robots below do different jobs and answer to different rules, so a single rule applied to all of them blocks the ones a site meant to keep and misses the ones it meant to stop. OpenAI alone operates four, each controllable separately:
GPTBot collects training data for foundation models.
OAI-SearchBot retrieves and cites pages inside ChatGPT search.
ChatGPT-User fetches a page on demand when someone in ChatGPT asks for it.
OAI-AdsBot validates landing pages submitted as ChatGPT ads.
Anthropic maintains ClaudeBot for training, Claude-User for on-demand fetches and Claude-SearchBot for indexing. Perplexity operates PerplexityBot and Perplexity-User. Google splits its AI controls across Google-Extended, GoogleOther and Google-CloudVertexBot, though one structural catch makes all three less useful than they look, since Googlebot feeds both Search and AI Overviews and no token separates them. Access logs will also show Meta-ExternalAgent, Amazonbot, Bytespider, Applebot-Extended and CCBot.
Agentic browsers are the harder category. ChatGPT Atlas and Perplexity’s Comet are both Chromium-based, and naive fingerprinting classifies them as generic Chrome. Atlas exposes a user agent containing “ChatGPT Atlas” with a CFNetwork signature on macOS. Beyond that, the practical tell is behavioral, meaning rapid multi-page navigation with near-zero dwell time and no scroll depth.
Can AI Crawlers Read JavaScript Content?
None of the major AI crawlers execute JavaScript, and no file added to the root directory works around that. They download .js files and read them as text without running them, which makes JavaScript SEO a hosting-layer question rather than a plugin one. Anything a site renders client-side is invisible to the systems most likely to cite it.
What AI Crawler Traffic Does to a Server
The cost of AI traffic comes from the share of requests that miss cache and reach origin, and the raw request count is a poor proxy for it. Crawlers generated 65% of Wikimedia’s most expensive, cache-missing traffic while accounting for 35% of its pageviews. Human traffic spikes around news events and sustained scraper load behave nothing alike, so infrastructure built to absorb the first handles the second badly.
The same pattern appears on unrelated projects run by unrelated people, so it cannot be written off as one badly configured server. SourceHut’s maintainer spent between 20% and 100% of his working time in a given week mitigating crawlers, with dozens of brief outages weekly. At GNOME, 81,000 requests arrived at the GitLab instance in 2.5 hours, and 3% of them passed a proof-of-work challenge. Duke University Libraries lost three separate platforms to extended outages in mid-2025, on infrastructure with staff and budget behind it.
Does Rate Limiting by IP Work?
The crawlers arrived from a large number of addresses, so per-endpoint IP rate limiting on Read the Docs had no effect. Limiting by user agent was closed off too, because much of the abusive traffic sent the same browser strings human visitors send. Wherever large botnets run headless Chrome on cloud providers, blocking one range only moves the traffic to another.
Why Your Site Throws 503 Errors Under Crawler Load
On shared hosting, the failure signature is specific enough to diagnose from the error alone. 503 Service Unavailable with no obvious cause usually means the PHP handler is saturated and has no worker free to take the next request, while RAM ceilings appear as 500 or 503 with a “Cannot allocate memory” entry in the log. Nothing on the site has to have been touched for either symptom to appear, since the load arrived from outside it.
What to Do at the Hosting Layer
Caching removes the most load while refusing the fewest legitimate requests, so it belongs ahead of anything that blocks or challenges a request.
Cache Everything You Can Serve From the Edge
A crawler hit answered by a CDN or a server-level cache never reaches PHP or opens a database connection, so it costs the server close to zero. The expensive requests are the rarely visited URLs whose cache has expired and which reach origin every time. On a 14,000-page WordPress site, origin requests from AI bots fell from 47,000 a day to 380 after full-page edge caching was switched on.
LiteSpeed with LSCache holds cached pages at the web server rather than inside a plugin, so cached URLs are served without touching PHP and only the uncached ones consume a worker. GreenGeeks runs that combination with a free CDN on its shared plans, which puts the cache in front of PHP by default.
Write Per-Agent Rules in robots.txt
Per-agent rules cost almost nothing to write, and they bind only the operators that already honor them. OpenAI takes 24 hours to propagate a robots.txt change to its search systems, so nothing written there takes effect immediately. A March 2026 sample of robots.txt files found GPTBot named in 13.8% of them and fully disallowed by fewer than one site in twenty. Uptake that thin is a factor in its own right. A modest Crawl-delay belongs in the file for ClaudeBot, since Anthropic is the one major operator that reads the directive at all.
Match the User Agent and the IP Together
Blocking on user-agent alone produces two failures at once. Human visitors whose browser string resembles a bot get refused, and bots spoofing a browser string get through. Both failures are answered by the operator-published IP ranges, issued as JSON at documented URLs by OpenAI, Anthropic and Perplexity, so a block rule can match the agent name against the source IP. Anthropic warns specifically against blocking its IP addresses in place of robots.txt, since a crawler that cannot reach the server cannot read the opt-out either.
Server-Level Rate Limiting and Bot Controls
nginx implements rate limiting through limit_req and a leaky bucket. Its default rejection status is 503, and setting limit_req_status to 429 makes the response name what happened. Misconfigured limits are a documented cause of self-inflicted outages, so anything aggressive needs testing against real traffic first. Above that layer are bot-management products, with Cloudflare’s AI Crawl Control offering allow, charge or block per crawler. Proof-of-work interstitials such as Anubis are the last resort. They charge every visitor including the humans, and users on slow phones have reported waits of 30 seconds and longer.
MCP, Agentic Commerce and llms.txt
These three surfaces deserve the word API in its strict sense, all of them are opt-in, and two of the three are irrelevant to most sites.
What Is MCP, and Is It Replacing REST APIs?
The Model Context Protocol was open-sourced by Anthropic in November 2024 and has since been adopted across the industry, with governance now handled by a joint steering group. Its authorization spec makes an MCP server an OAuth 2.1 resource server, requiring bearer tokens, PKCE and proper token expiration. A plain HTML site hands out nothing to authenticate against, so a crawler fetch has no equivalent step. The structural difference from REST is discovery, since a REST client is written against fixed endpoints documented in advance while an MCP client asks the server at runtime which tools it exposes and calls them from that answer.
None of it applies to a typical website, only to one whose owner deliberately stands up an MCP server. Most production MCP servers wrap an existing REST API that the team goes on running alongside it.
How Agent Checkout Works
The Agentic Commerce Protocol, released by OpenAI and Stripe in September 2025, is the most fully specified agent-facing API a site owner can choose to expose. It defines machine-readable product feeds, a stateful checkout session and delegate payment tokens that keep the merchant as merchant of record. Etsy was live on launch day and a set of Shopify brands followed within weeks. For a store that wants agents completing purchases, it is the working specification.
Does llms.txt Do Anything?
On the available evidence, not yet. No AI system currently uses llms.txt, and Google has said it has no plans to support it. Across a sample of 137,000 sites, 97% of llms.txt files received zero traffic in May 2026, even as adoption grew almost ninefold in a year. Writing one takes an hour and has no downside, so adding the file costs very little against the chance that support arrives later.
Should You Block AI Crawlers?
The people running servers and the people running publications have reached opposite conclusions, and both are arguing from evidence that holds up. Sysadmins see sustained load at denial-of-service volume without the intent behind it, and they block. Publishers who blocked saw total traffic fall 23% and human traffic fall 14% across the top 30 that tried it, and some reversed the decision. The Googlebot problem removes the tidy middle option. Blocking the crawler that feeds AI Overviews removes the site from Search as well.
The reading that holds up under both sets of evidence begins with caching. Most crawler requests then never reach PHP, and the load argument for blocking weakens on its own. Per-agent rules follow, for the operators that honor them. Verify agents by IP before blocking anything. Server-rendered HTML matters more than any of it, because the systems reading a site cannot run the code that would otherwise produce the page. Once the caching work is done, the server-load case for blocking disappears and what remains is a revenue and licensing decision.
Frequently Asked Questions
What is an API key, and what is it used for?
An API key is a shared secret sent with a request, usually in a header, that identifies which account is calling. The provider uses it to authorize the call and count it against a quota.
What is an API token?
A token is a credential issued after authentication, typically sent as “Authorization: Bearer” followed by the token value. Unlike a static key, a token expires and can be scoped or revoked on its own.
What is the difference between an SDK and an API?
The API is the contract the server exposes. An SDK is a language-specific library that wraps that contract so a developer does not hand-build the HTTP requests.
What is an API endpoint?
An endpoint is the address where an API receives requests, which for most services is a URL. Each one pairs a resource path with an HTTP method such as GET or POST to describe the action requested.
What is an API call?
An API call is a single request-and-response exchange, made up of one method, one endpoint, optional headers and body, and one response that returns a status code. Rate limits are counted in calls. A rejected 429 still consumes a slot on most platforms.
What is an API gateway?
An API gateway is a layer in front of backend services that handles routing, authentication and rate limiting centrally instead of inside each service. A CDN paired with a web application firewall does the same job for a website.
How do I see AI crawler traffic in my server logs?
Grep the access log for the agent name. Apache logs are usually at /var/log/apache2/access.log and nginx at /var/log/nginx/access.log, with the bot appearing in the user-agent field as GPTBot/1.2 or ClaudeBot/0.1.
How do I verify that a bot is GPTBot or ClaudeBot?
Do not trust the user-agent string, which is trivially forged. Each operator publishes its IP ranges in a JSON file at a documented URL. The standard check matches the agent name against the source IP.
What is Web Bot Auth?
Web Bot Auth is a cryptographic bot-verification scheme built on RFC 9421 HTTP Message Signatures. The operator publishes an Ed25519 public key at /.well-known/http-message-signatures-directory and signs each request with Signature and Signature-Input headers.
What is pay per crawl?
Pay per crawl is a Cloudflare feature announced on July 1, 2025 that lets a site charge AI crawlers per request. It uses HTTP status 402 Payment Required, so an unpaid crawler gets a 402 with pricing attached and can retry with signed payment credentials.
How do agentic browsers like ChatGPT Atlas and Comet show up in analytics?
Sessions from both arrive in analytics as desktop Chrome on macOS, because the browsers are built on Chromium. Filtering the user agent for “Atlas” or “Comet” catches the declared cases. The rest arrive with dwell time near zero and scroll depth flat.
