What user-agent spoofing is
Every HTTP client can send a User-Agent header describing itself: a browser name and version, a crawler name, a library identifier. The header is self-reported. Nothing in HTTP checks that it is true, so any client can send any value. User-agent spoofing is the practice of sending a value that misrepresents the client, for example:
- an automation script presenting itself as a current desktop browser,
- a scraper claiming to be a well-known search crawler to get favorable treatment,
- an AI agent using a generic browser string so that it is treated like a person.
Why it matters
Many systems still make decisions from the user-agent string: allowlisting known crawlers, serving different content to bots, or excluding automation from analytics. Each of those decisions inherits the weakness of the header. If a policy says "allow anything that claims to be crawler X", it allows everyone who copies crawler X's string.
For applications receiving AI agent traffic, this matters more over time. Agents increasingly run inside real browsers or headless browser frameworks, so their user-agent strings look exactly like human browsers. Others identify honestly. The header alone cannot tell you which is which. Our post on how to tell if website traffic comes from AI agents looks at the signals that work better.
Better evidence than the header
- Cryptographic signatures. With HTTP Message Signatures and the Web Bot Auth work, a client proves it holds a key published by the operator it names. A copied user-agent string cannot reproduce a signature.
- Operator-published verification. Some crawler operators publish IP ranges or support reverse DNS checks, which make impersonation harder than the header alone. Google documents a reverse DNS and IP range check for its crawlers, Microsoft offers a Verify Bingbot tool, and OpenAI publishes IP address files for each of its bots. Anthropic's guidance for its hosted browser agent states plainly that anyone can send the same User-Agent header and recommends checking its published IP addresses.
- Consistency checks. Comparing the claimed client with protocol details and interaction signals can show when a claim does not fit, though this produces inference, not proof.
- Credentials. For APIs, an API key or token tells you which account is calling regardless of what the header says.
Common misconceptions
- "A browser user-agent means a human." Headless browsers and agent frameworks routinely send browser strings.
- "A crawler user-agent means the crawler." Without a signature or an operator verification check, it is only a claim.
- "Spoofing is always malicious." Some legitimate tools send generic strings to avoid being served broken pages. Spoofing is a reason to discount the header, not proof of bad intent.
How Proof of Actor treats the user-agent string
Proof of Actor records a user-agent claim as declared evidence: something the client stated, which may or may not be true. It never upgrades a declaration to verified on its own. A request whose only identifying evidence is a crawler user-agent string is labeled accordingly, and when evidence is insufficient the result is Unknown rather than a confident guess. Your policies in Route can then treat a verified signed agent and a self-declared one differently, for example allowing both on read-only routes but applying tighter limits to the declared one.
See how evidence bases work or apply for early access.
Sources
- RFC 9110: HTTP Semantics, User-Agent: the header is self-reported by the client
- Google: Verifying Google's crawlers and fetchers: reverse DNS and published IP range checks
- Microsoft Bing: Verify Bingbot tool: checking whether an IP address belongs to Bingbot
- OpenAI: Overview of OpenAI Crawlers: user agents and published IP address files
- Anthropic: how Anthropic-hosted browsers identify themselves: warning that the User-Agent header can be copied