How to give AI agents a structured path through your app

Why do AI agents need a different path through your app?

A growing share of the requests to your application come from AI agents: assistants fetching documentation for a user, research tools reading product pages, autonomous workflows calling your API. Most of them are working through interfaces designed for people. They download full HTML pages, parse layouts, follow navigation meant for a mouse, and retry when a script-heavy page does not render the way they expect.

That is inefficient for everyone. Agents get noisy content and make mistakes. Your servers spend resources rendering pages nobody looks at. And the traffic is hard to reason about, because it looks like a browser that behaves strangely. If you are not yet sure how much of your traffic this is, start with how to tell if website traffic comes from AI agents.

A better approach is to give agents a structured path: content and actions in formats built for programs, discoverable in standard places, with limits that reward clients who identify themselves properly. This guide shows how to do that without building a second application.

Step 1: Decide what agents should be able to do

Start with intent, not technology. Look at the routes where you already see agent traffic and group them:

• Read content: documentation, product information, pricing pages, help articles, public profiles.
• Query data: search, catalog lookups, availability, status.
• Take actions: create, update or delete something on behalf of a user.

Reading and querying are good candidates for a structured path that is open to any well-behaved client. Actions need authentication and authorization regardless of how the client identifies itself, and they are covered in the guide on protecting sensitive actions.

Step 2: Offer machine-friendly formats

For content routes, offer the same information without the page chrome. Two formats cover most needs:

• Markdown for prose content such as documentation and articles, served with the text/markdown media type registered in RFC 7763. It keeps headings, lists, links and code blocks, and it is compact.
• JSON for structured records such as products, plans or status.

Use content negotiation so the URL stays the same and the client asks for the format it wants:

import express from "express";

const app = express();

app.get("/docs/:slug", async (req, res) => {
  const doc = await loadDoc(req.params.slug);
  if (!doc) return res.status(404).end();

  res.vary("Accept");
  const preferred = req.accepts(["text/html", "text/markdown", "application/json"]);

  if (preferred === "text/markdown") {
    return res.type("text/markdown; charset=utf-8").send(doc.markdown);
  }
  if (preferred === "application/json") {
    return res.json({ title: doc.title, updatedAt: doc.updatedAt, body: doc.markdown });
  }
  return res.render("doc", { doc });
});

Send a Vary: Accept header (RFC 9110 on Vary) so caches store each representation separately. Many teams also expose an explicit alternate URL, such as appending .md to a documentation path, because some clients find that easier than negotiation. Link it from the HTML page with <link rel="alternate" type="text/markdown" href="..."> so it can be discovered. Typed links like this follow the Web Linking model.

Step 3: How do agents discover your structured path?

Agents cannot use what they cannot find. Publish entry points in places automated clients already look:

1. /llms.txt at the site root: a short Markdown summary of what your site offers with links to the most useful pages and their structured versions, following the llms.txt proposal.
2. An API description such as an OpenAPI document, linked from your developer documentation and from /llms.txt.
3. Alternate links in HTML pages pointing to Markdown or JSON versions.
4. A sitemap listing your canonical content URLs, in the format defined by the Sitemaps protocol.

A minimal llms.txt:

# Example App

> Example App helps teams plan projects. This file lists the best entry points for automated clients.

## Docs
- [Getting started](https://example.com/docs/getting-started.md): setup in five steps
- [API reference](https://example.com/openapi.json): OpenAPI description of the public API

## Policies
- [Automated access](https://example.com/docs/automated-access.md): limits and how to sign requests

Keep a page that explains your rules for automated clients: which routes are open, what limits apply, and how a client can identify itself with signed requests to receive better treatment. Clear rules turn guesswork into cooperation.

Step 4: Route by actor class, not by user agent string

Once you have structured responses, decide who gets them by default. Use the actor class and its basis rather than parsing user agent strings ad hoc. AI operators do publish their user agents, as OpenAI and Anthropic do for their crawlers and fetchers, but a string match is still only a declaration.

type ActorClass = "human" | "human_ai" | "agent" | "unknown";
type Basis = "verified" | "declared" | "inferred" | "unknown";

function preferStructured(actor: ActorClass, basis: Basis, accept: string): boolean {
  if (accept.includes("text/markdown") || accept.includes("application/json")) return true;
  return actor === "agent" && (basis === "verified" || basis === "declared");
}

Two principles keep this safe:

• Explicit requests always win. If a client asks for Markdown or JSON, serve it, whatever the class.
• Defaults never take anything away. A Human or Unknown visitor always gets the full page. Choosing a structured default for agents changes the format, not what they are allowed to see.

Step 5: What rate limits should AI agents get?

Rate limits are where actor context pays off most visibly. Instead of one limit for everyone, give each combination of class and basis a sensible budget:

Key the limit by the most stable identifier available. For verified agents, that is the operator origin and key identifier. For others, use the credential or a network-level key.

function limitKey(actor: { actor: ActorClass; basis: Basis; operator?: string }, ip: string, apiKey?: string) {
  if (actor.actor === "agent" && actor.basis === "verified" && actor.operator) return `op:${actor.operator}`;
  if (apiKey) return `key:${apiKey}`;
  return `ip:${ip}`;
}

function limitFor(actor: { actor: ActorClass; basis: Basis }): number {
  if (actor.actor === "agent" && actor.basis === "verified") return 600;
  return 120;
}

The numbers are placeholders. Set yours from the baseline you observed. When a client exceeds a limit, return 429 Too Many Requests with a Retry-After header and a link to your automated access page. Well-built agents back off and read it.

Step 6: Keep actions behind authorization

A structured path for reading is not a path around your security model. Any action that changes data still requires:

1. authentication with a credential your application issued,
2. authorization for that specific action, and
3. whatever additional checks the route's risk tier calls for.

A verified signature tells you which client is calling. It does not tell you that the client may act for a given user. Keep those questions separate. Our post on what signed AI agent requests prove explains the difference.

Step 7: Measure the effect

Run the structured path in observation mode first, then compare:

• the share of agent requests that use structured formats,
• server cost and latency for agent traffic before and after,
• error and retry rates for agent clients,
• how many automated clients move from declared to verified once you publish your signing guidance.

These numbers show whether the structured path is doing its job and where to extend it next.

Step 8: Version and document the structured path

Treat the structured path as a public interface, because agents will build on it. A few habits keep it dependable:

1. Keep URLs stable. If an agent learned that /docs/getting-started.md exists, it should keep working. When content moves, return a 301 to the new location rather than a 404.
2. Include update times. Put an updatedAt field in JSON responses and a Last-Modified header on Markdown, so agents can cache sensibly and refresh only when something changed.
3. Support conditional requests. Respond to If-None-Match and If-Modified-Since with 304 Not Modified. Agents that poll your content cost far less when they can revalidate instead of downloading everything again.
4. Version breaking changes. If a JSON shape must change, add a new version path or media type and keep the old one for a published period.
5. Publish a changelog for the automated access page. Operators of well-behaved agents read it, and it gives you a place to announce new structured routes.

app.get("/api/plans", async (req, res) => {
  const plans = await loadPlans();
  const etag = `"${plans.version}"`;
  res.set("ETag", etag);
  res.set("Cache-Control", "public, max-age=300");
  if (req.get("if-none-match") === etag) return res.status(304).end();
  return res.json({ updatedAt: plans.updatedAt, plans: plans.items });
});

These practices also benefit your own frontend and partner integrations. A structured path that is stable, cacheable and documented quickly becomes the easiest way for any program to work with your application.

Common pitfalls

• Serving different facts to agents and people. The structured version should contain the same information as the page. Divergent content confuses users who ask an assistant about your product.
• Hiding the structured path. If it is not linked from /llms.txt, your API docs or the pages themselves, agents will keep scraping HTML.
• Rewarding claims. Higher limits belong to verified clients. A copied user agent string should not earn a bigger budget.
• Forgetting caches. Without Vary: Accept, a cache can serve Markdown to a browser.

Where Proof of Actor fits

Proof of Actor gives you the actor class and basis you need to route agents to structured content and set fair limits. It detects signed, declared and inferred agents, helps you understand how they use your routes, and leaves the routing decision in your code.

To design a structured path for your own application with our team, apply for early access. For API and MCP scenarios specifically, see APIs and MCP Servers, and read who is calling your MCP server? for how actor context applies to Model Context Protocol tool calls.

Sources

• RFC 9110: HTTP Semantics: content negotiation (Section 12), Vary (Section 12.5.5), conditional requests (Section 13) and Retry-After (Section 10.2.3)
• RFC 7763: The text/markdown Media Type: serving Markdown representations
• RFC 8288: Web Linking: typed alternate links
• RFC 6585: Additional HTTP Status Codes, Section 4: 429 Too Many Requests
• llms.txt proposal: the /llms.txt file format
• OpenAPI Specification: machine-readable API descriptions
• Sitemaps protocol: sitemap file format
• OpenAI: Overview of OpenAI Crawlers: OpenAI's published user agents and IP ranges
• Anthropic: crawler and fetcher documentation: ClaudeBot, Claude-User and Claude-SearchBot
• Model Context Protocol specification (2026-07-28): the current MCP revision as of November 2026