{"slug":"agent-seo-and-discovery","kind":"essay","title":"Agent SEO and Discovery","summary":"The next discovery problem is not only how humans find a page, but how agents discover, trust, and route to it.","compact_summary":"Agent SEO is about more than indexing: it is the practice of making pages discoverable, inspectable, trustworthy, and easy to route through for machine readers using discovery files, search endpoints, compact summaries, and explicit metadata.","key_claims":["Discovery for agents is partly a routing problem, not only an indexing problem.","`llms.txt`, sitemaps, manifests, and compact content routes form a practical discovery pack.","Trust signals matter because agents need to decide not just what exists, but what deserves attention."],"section_map":["Agent SEO and Discovery","Discovery Is Not Enough","The Practical Discovery Pack","Trust As A Discovery Signal","Already Happening","The Real Opportunity"],"confidence":"high","intended_use":["Use this page to understand why civ.build ships discovery files and compact retrieval."],"do_not_use_for":["Do not treat this page as a final standard for the whole ecosystem."],"updated_at":"2026-04-10T00:00:00.000Z","verified_at":"2026-04-10T00:00:00.000Z","version":"0.2.0","estimated_tokens":446,"word_count":330,"content_hash":"d7ab14e7a6cdf0f9e8076b97286b1cb8e7c73b657de339f2b1f359bdf772f2a2","change_summary":"Added the Cloudflare markdown-for-agents pattern as a real-world example alongside the original discovery essay.","requires_human_judgment":false,"tags":["discovery","seo","llms-txt","agents"],"_links":{"self":"/api/v1/content/agent-seo-and-discovery","compact":"/api/v1/content/agent-seo-and-discovery/compact","meta":"/api/v1/content/agent-seo-and-discovery/meta","raw":"/api/v1/content/agent-seo-and-discovery/raw","versions":"/api/v1/content/agent-seo-and-discovery/versions","related":["/api/v1/content/public-knowledge-contracts-for-agents/compact","/api/v1/content/for-agents/compact"],"canonical_human":"/p/agent-seo-and-discovery","capabilities":"/api/v1/capabilities"},"content":"# Agent SEO and Discovery\n\nTraditional SEO asks: how do humans find this page?\n\nThe next question is: how do agents discover it, trust it, and route other agents toward it?\n\n## Discovery Is Not Enough\n\nAn agent does not only need to know that a page exists. It needs to answer:\n\n- is this page relevant\n- is it fresh\n- is it trustworthy enough for the task\n- what is the shortest useful representation\n- where should I go next if I need more depth\n\nThat means discovery is partly a routing problem.\n\n## The Practical Discovery Pack\n\nThe first useful pack is boring in the best way:\n\n- `llms.txt`\n- `robots.txt`\n- `sitemap.xml`\n- a machine-readable manifest\n- search\n- compact content routes\n\nNone of this requires magical AI infrastructure. It requires explicit structure.\n\n## Trust As A Discovery Signal\n\nAgents should not only rank pages by relevance. They should also inspect trust signals:\n\n- confidence\n- updated date\n- verified date\n- sources\n- intended use\n- do not use for\n\nWithout those signals, a page is discoverable but not really usable.\n\n## Already Happening\n\nThis is not theoretical. Cloudflare has already shipped an agent-facing markdown mode: when an agent sends the right request header, the site returns clean markdown instead of HTML, with an estimated token count so the agent can budget its context window. Combined with `llms.txt` and AI index features that let site owners make their content discoverable to agents directly — bypassing traditional search — the infrastructure for an agent-readable web is being built right now.\n\nOther tools point the same direction. Firecrawl and Exa strip presentation noise and return structured content. Server-side compaction handles context-window constraints. The pieces exist. What is missing is sites that take the contract seriously on the content side, not just the transport side.\n\n## The Real Opportunity\n\nAgent SEO is not \"do SEO but mention AI.\" It is the design of public surfaces that agents can correctly route through with minimal ambiguity and minimal wasted context.","author":"civ.build","sources":[],"related_pages":["public-knowledge-contracts-for-agents","for-agents"],"canonical_url":null,"license":null,"contact":null,"status":null,"audience":["humans","agents"],"agent_takeaway":null}