How to tell if website traffic comes from AI agents

To tell whether website traffic comes from AI agents, combine evidence you can check, such as request signatures and verified crawler IP lookups, with weaker declared signals like the User-Agent header and inferred signals like request patterns and browser interaction. Record every result with its basis (verified, declared, inferred or unknown), keep Unknown as a real answer, and observe before you enforce anything.

The reason this takes care is that your website and API serve three kinds of visitors at once. People clicking through pages. People working alongside an AI assistant that drafts, summarizes or fills in forms. And autonomous agents that browse, call your API or invoke your MCP tools on someone's behalf, sometimes with no person watching in real time.

From the server's point of view, many of these interactions look alike. They arrive as HTTP requests with headers, a path and maybe a session cookie. The question "is this an AI agent?" sounds simple, but answering it well means understanding which signals you actually have, how much each one can be trusted, and what you plan to do with the answer.

This guide walks through the signals available to a typical web application (the product view is on AI agent detection for websites), how to combine them, and how to avoid the two most common mistakes: trusting claims that cost nothing to fake, and forcing a confident label onto traffic you cannot actually classify.

Start with the question you need answered

Before collecting signals, decide what decision the answer feeds into. "Is this an agent?" is rarely the real question. More often it is one of these:

• Should this caller get the structured API instead of the HTML page?
• Should this request count toward human engagement metrics?
• Does this account-changing action need a stronger check?
• Which automated clients are using my content, and how often?
• Is this MCP client one I have agreed to serve?

The decision determines how much certainty you need. Routing a documentation request to a Markdown version tolerates a wrong guess. Blocking a checkout does not. Keep the decision in mind as you read the signal categories below.

Category 1: What the client says about itself

User-agent strings

The User-Agent header is the first thing most teams look at, and many AI crawlers and assistants do identify themselves there. HTTP defines the header as information about the user agent originating the request (RFC 9110), and operators publish their tokens so site owners can recognize and manage their traffic. As of November 2026, for example, OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot.

The problem is that a user-agent string is a declaration. Any HTTP client can send any value. A scraper can claim to be a well-known crawler, and an automated browser can present a perfectly ordinary desktop browser string. Treat user-agent matches as useful context and as a way to group traffic, never as verification.

Self-identification in protocols

Some protocols carry richer self-description. MCP clients, for example, send a clientInfo name and version, which the MCP specification describes as self-reported and not verified by the protocol, and many API clients set descriptive headers. These declarations are valuable because cooperative clients use them honestly, and they help you understand who is calling. They remain declarations, though. Record them with that basis attached.

robots.txt behavior

Whether a client fetches and respects robots.txt, standardized as the Robots Exclusion Protocol in RFC 9309, says something about its intentions, and some operators, such as Anthropic, document that their crawlers follow it. It is a behavioral hint rather than an identity check, and it only applies to crawling-style traffic.

Category 2: Evidence you can check

Verified crawler checks

Several large operators publish ways to confirm that a request really comes from their infrastructure. The classic method is forward-confirmed reverse DNS: resolve the client IP to a hostname, check that the hostname belongs to the operator's documented domain, then resolve that hostname back and confirm it matches the original IP. Google and Microsoft Bing both document this method for their crawlers. Operators also publish IP ranges in machine-readable files: Google lists separate files for its crawlers and user-triggered fetchers, Bing publishes bingbot.json, and OpenAI and Anthropic publish JSON lists for their bots on the pages linked above.

These checks are much stronger than a user-agent match because the client cannot simply claim them. Their limits matter too. They confirm which operator's infrastructure sent the request, not which person or task triggered it, and they only cover operators that publish this information.

HTTP message signatures

The most promising development for agent identity is cryptographic request signing. HTTP Message Signatures, standardized in RFC 9421, let a client sign selected parts of a request with a private key. The IETF Web Bot Auth working group builds on this so automated clients can sign requests and point to a published directory of their public keys; as of November 2026 its protocol document is an Internet-Draft. A server that validates the signature knows the request was signed by a holder of that key, associated with a published identifier such as the operator's domain.

This is the closest thing the web has to a verifiable agent identity. It is also precise about what it proves, which we cover in depth in What signed AI agent requests prove, and what they don't.

Your own credentials

If a caller uses an API key you issued, you already know which customer or integration it belongs to. That is strong evidence about the account, and weak evidence about whether a person or an agent is driving a given request. Many API keys are shared between human-operated scripts, backend jobs and agent frameworks.

Category 3: Request context

Server-side integrations see a lot of context that does not require any cooperation from the client:

• Header patterns. Real browsers send a characteristic set of headers in typical combinations. Many HTTP libraries send far fewer, or send values that real browsers do not.
• Resource loading. A person viewing a page usually triggers requests for stylesheets, scripts, images and fonts. Many automated clients fetch only the HTML or only the API route.
• Timing and sequence. Request rates, intervals and navigation order can differ sharply between people and automation.
• Network characteristics. Traffic from cloud and hosting networks is more often automated than traffic from residential networks, although the reverse is not reliable, because automation can also run through residential proxies.
• TLS and protocol fingerprints. The way a client negotiates TLS and HTTP/2 often identifies the underlying library. Methods such as JA3 and JA4 fingerprints are widely used for this.

Each of these is an inference. Individually they are easy to misread: a privacy-focused browser can look unusual, a corporate proxy can make many people look like one client, and a headless browser can be configured to look ordinary. Together, and with consistent labeling, they become informative.

Category 4: Interaction signals in the browser

For traffic that runs JavaScript, a browser integration can observe how the page is actually used: whether input events occur, how they are distributed, whether the environment exposes automation markers such as navigator.webdriver, which the W3C WebDriver specification defines for browsers under automation control, and whether the browser behaves consistently with the browser it claims to be.

These signals are powerful for distinguishing human-directed sessions from scripted ones on the pages where the collector runs. Two boundaries are worth stating clearly to your team:

1. They see nothing from clients that do not execute JavaScript, which includes many API callers and simple crawlers. Those need server-side evidence instead.
2. They cannot see what happens outside the page. If a person asks a chatbot in another tab and pastes the answer into your form, the session is human-directed, because a person directed it. Detecting AI-assisted work reliably requires a declaration or an observable signal, such as your application's own AI feature or a disclosure control in your interface.

Putting signals together

Separate the class from the basis

The single most useful design decision is to record two things for every interaction: what you concluded and why. We cover this model in more depth in how to distinguish human and AI agent traffic. A practical scheme uses four classes:

• Human: observable signals indicate human-directed interaction.
• Human + AI: AI assistance was declared or directly observed.
• Agent: automated or agent-origin interaction.
• Unknown: the evidence does not support a reliable determination.

And four bases:

• Verified: a checkable assertion validated, such as a signature or a confirmed crawler check.
• Declared: the client or user said so.
• Inferred: concluded from behavioral or contextual signals.
• Unknown: not enough evidence.

An "Agent, verified" result from a valid signature and an "Agent, inferred" result from header patterns deserve different treatment. Keeping the basis next to the class lets your policies reflect that, and lets anyone reviewing a decision see what it rested on.

Keep Unknown visible

There is constant pressure to make every request land in a confident bucket. Resist it. Unknown tells you where your coverage ends: routes without a browser collector, clients that send nothing distinctive, sessions too short to say anything about. If you quietly fold Unknown into Human, your metrics look clean and your decisions get worse. If you fold it into Agent or treat it as suspicious, you start penalizing real users who happen to use privacy tools.

A visible unknown rate is also how you measure progress. As you add integrations and as the agents that reach you start signing their requests, the share of Unknown traffic should fall for the right reasons. Our guide on handling unknown traffic shows how to set fallbacks per route.

Account for time and scope

Evidence has a context. A browser signal applies to the page and moment it was collected. A signature applies to the request it was attached to. Reusing an early-session result for a sensitive action an hour later is a common mistake. For high-impact actions, evaluate the request in front of you.

From detection to decisions

Classification only matters because of what you do with it. Good policies share a few traits:

• They are route-specific. A public pricing page, a read-only catalog API and a password change have very different risk profiles.
• They match controls to impact. Allow low-risk automated reads, offer structured endpoints to known agents, and ask for stronger credentials before high-impact actions.
• They define explicit fallbacks. Decide in advance what Unknown, a timeout or missing telemetry means for each route. A timeout should never silently become a Human result.
• They do not treat agents as abuse by default. Many agents are legitimate participants acting for your customers. A structured path often serves them, and you, better than a block.

Start in observation mode

The safest way to introduce any of this is to observe first. Record classifications without changing how your application responds, then compare them with outcomes you already know: accounts you know are automated, internal test traffic, support tickets, conversion data. Look closely at false positives before anything is enforced, especially on flows that real customers depend on.

Observation also tells you which questions your data can answer today and which need another integration, such as a browser collector on a key flow or signature validation at the edge.

How Proof of Actor approaches this

Proof of Actor is built around exactly this model. Detect gathers the signals available to each integration and separates verified assertions from behavioral inferences. Understand shows classes, bases, routes and trends, with the evidence behind each result and the unknown rate kept in view. Route lets your application apply its own policies, starting in observation mode, with enforcement always in your hands.

If you operate a website, see how this applies on the websites use case. If you operate a SaaS application, website, API or MCP server and want to see who is really using it, apply for early access. To understand the evidence model in more depth, read Detection & Limitations.

Sources

• RFC 9421: HTTP Message Signatures, IETF.
• RFC 9110: HTTP Semantics, section 10.1.5 User-Agent, IETF.
• RFC 9309: Robots Exclusion Protocol, IETF.
• Web Bot Auth (webbotauth) working group charter, IETF Datatracker.
• HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol), IETF Internet-Draft, as of November 2026.
• Verifying Google's crawlers and fetchers, Google.
• How to verify Bingbot and bingbot.json IP ranges, Microsoft Bing.
• Overview of OpenAI crawlers, OpenAI.
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic.
• Model Context Protocol specification, base protocol (2026-07-28).
• WebDriver, W3C.