--- title: "You Didn't Know GEO: AI Visibility Principles, Practices, and Trade-offs" source: "https://x.com/HiTw93/status/2050931710066565374" author: - "[[@HiTw93]]" published: 2026-05-03 created: 2026-05-04 description: "One Hour to Make AI Find Your ContentA few friends pinged me recently saying my open source tools were showing up when they asked AI questio..." tags: - "clippings" ---  ## One Hour to Make AI Find Your Content A few friends pinged me recently saying my open source tools were showing up when they asked AI questions. I hadn’t done anything deliberate, so I figured: why not spend an hour structuring things properly? After doing it, I fired off a quick tweet, but the notes were messy. People seemed genuinely interested, so I decided to write it up as a proper article for reference. I hate gaming rankings or generating junk content. This article won’t teach you shortcuts. It’s about helping AI better understand the content you already have. I looked into why it happened and found that AI search runs on entirely different logic. Traditional SEO is about fighting into the top 10, but 83% of AI Overview citations come from pages outside the top 10. AI rewards clear structure and reliable sourcing, not PageRank. My projects aren’t big, but the READMEs and docs are well-written enough that AI picks them up where bigger sites have thin content. That’s probably why friends were seeing Pake and MiaoYan in their AI results. AI search is growing fast: 527% year-over-year in the first half of 2025, ChatGPT hit 900 million weekly active users by February 2026, and referral traffic converts at roughly 5x the rate of traditional search. But it still accounts for less than 1% of total referral traffic. This is a brand visibility strategy, not a traffic strategy. Worth an hour of setup, not a week, because your product is your real competitive advantage, not this.  ## Use robots.txt to Sort Out Crawler Types Most people treat robots.txt as a switch: either block AI crawlers or allow them all. But AI crawlers come in several types, and they do different things. **Training crawlers** (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot) take your content to train models. Blocking them keeps your content out of training data but doesn’t affect current AI search results. **Search and retrieval crawlers** (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch content in real time to answer user queries. Block these and you vanish from AI search. **User-triggered fetchers** (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent) only fire when someone pastes your URL into a chat window. Block them and users asking “summarize this page” get nothing. **Opt-out tokens** (Google-Extended, Applebot-Extended) aren’t real crawlers. They’re signals you declare in robots.txt to opt out of AI training. **Undeclared crawlers** (Bytespider, xAI’s Grok bot) don’t identify themselves and don’t necessarily follow the rules.  My approach: allow search/retrieval and user-triggered crawlers, block training and undeclared ones. ```text # Search & retrieval: allow User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # User-triggered: allow User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / # Training: block User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / # Opt-out tokens User-agent: Google-Extended Disallow: / # Undeclared: block User-agent: Bytespider Disallow: / ``` ## Write Your llms.txt and Cross-Link Your Sites llms.txt is a new standard, similar to robots.txt but designed for AI consumption. You place a Markdown file at your site root describing what your site does, its key pages, and who’s behind it. AI systems prioritize this file when they crawl your content. BuiltWith tracks over 840,000 websites that have deployed llms.txt, including Anthropic, Cloudflare, Stripe, and Vercel. But in SE Ranking’s survey of 300,000 domains, adoption is only 10%. It’s still early, and being early is an advantage. The format is simple: ```markdown # Your Project Name > One-line description of what this is. ## Links - [Documentation](https://yoursite.com/docs) - [GitHub](https://github.com/you/project) - [Blog](https://yoursite.com/blog) ## About Short paragraph explaining the project, its purpose, key features, and what makes it different. ```  After creating yours, submit it to [directory.llmstxt.cloud](https://directory.llmstxt.cloud/), [llmstxt.site](https://llmstxt.site/), and the llms-txt-hub repository on GitHub via PR. I also did something interesting: I made each site’s llms.txt reference the others. I maintain [tw93.fun](https://tw93.fun/), [weekly.tw93.fun](https://weekly.tw93.fun/), and [yobi.tw93.fun](https://yobi.tw93.fun/). Each site’s llms.txt links to the rest, forming a mesh. An AI crawler entering through any one site can follow the links and discover everything else.  These changes take effect after crawlers revisit your site, usually within a few days. After that, try searching for your project name in ChatGPT. The citation sources and description accuracy should improve.  ## Provide Full Content and Markdown Routes llms.txt is the summary. llms-full.txt is the complete version, typically 30-60KB, containing project descriptions, FAQs, usage scenarios, competitive comparisons, and README excerpts. Mintlify’s CDN analysis shows llms-full.txt gets 3-4x more traffic than llms.txt. AI systems that find the summary want the full version. **Markdown routes** go further. Evil Martians recommend providing a .md version of every page on your site. A 15,000-token HTML page becomes a 3,000-token Markdown document, 80% less noise.  The simplest way to tell AI you have a Markdown version is adding this to your page’s
: ```text ``` Claude Code and Cursor already send Accept: text/markdown headers when fetching docs. This is standard HTTP/1.1 content negotiation, around since 1997. ## Register Your Site with Search Platforms The robots.txt and llms.txt work from the previous sections makes your content readable to AI, but AI has to find you first. ChatGPT’s search runs on Bing, Google AI Overview uses Google’s own index, and Perplexity also relies on search APIs. If your pages aren’t indexed by search engines, none of the structuring work above matters. So the first step is making sure Google and Bing have indexed your site. The setup is straightforward: go to [Google Search Console](https://search.google.com/search-console), verify your domain via DNS or HTML file upload, then submit your sitemap URL (usually [yoursite.com/sitemap.xml](https://yoursite.com/sitemap.xml)). Check the “Pages” indexing report to see which pages are indexed and which have issues. If an important page isn’t indexed, use the URL Inspection tool to manually request indexing. You might think Bing doesn’t matter, but Copilot, DuckDuckGo, and Yahoo all run on Bing’s index underneath. Register with Bing Webmaster Tools, submit your sitemap, and check the AI Performance panel to see how often AI cites your content. While you’re there, set up IndexNow so Bing gets notified immediately when you publish new content instead of waiting for a crawler to find it. Setting up IndexNow means placing an API key file at your site root, then sending a POST request to [api.indexnow.org/indexnow](https://api.indexnow.org/indexnow) with the list of changed URLs whenever you publish. Bing picks them up within minutes. Many static site generators and CMS platforms have IndexNow plugins available.  Google Search Console doesn’t have an AI-specific panel yet, but submitting your sitemap and monitoring indexing status is still worth doing. Google’s AI Overviews pull from a wider range than traditional results, so even pages that don’t rank in the top 10 can appear in AI-generated answers. Perplexity has more users than you’d expect. They run a publisher program at [pplx.ai/publisher-program](https://pplx.ai/publisher-program). Once approved, you get an 80/20 revenue share and access to citation analytics data. ## I Built a Dedicated Knowledge Site for AI Instead of waiting for AI to scrape information from scattered sources, give it a single entry point with everything organized. A knowledge site should provide three layers: an overview (llms.txt), a full version (llms-full.txt, 30-60KB), and standalone knowledge pages for each core project. Add structured JSON APIs so AI tools can fetch data programmatically. Pull data live from upstream sources like the GitHub API with caching that refreshes periodically, keeping maintenance cost near zero. One thing that’s easy to miss: give AI a narrative structure, not just a list of projects. If you have multiple projects, write a description that connects them, how they relate, your technical direction, the overall picture. When AI answers “who is this person” or “what does this team do,” a coherent narrative works much better than a flat list. My implementation is called Yobi (from the Japanese 呼び / よび, meaning “to call” or “to summon”). It serves an llms.txt overview, a 50KB llms-full.txt, per-project pages, and four JSON endpoints (/api/profile, /api/projects, /api/blog, /api/weekly) that pull live data from the GitHub API with ISR caching that refreshes hourly. Stack: Next.js + TypeScript on Vercel.  The JSON API returns structured project data with live GitHub stats:  ## Give Each Project Its Own Page Each project needs its own standalone page, not a row in a list, but a self-contained Markdown document with a citable summary, core features, competitive comparisons, use cases, and install commands. Ahrefs found that cited pages have titles with higher semantic similarity to user queries, and natural-language URL slugs (like /projects/pake) get cited more than opaque IDs (like /page?id=47).  URL structure matters. /projects/pake tells the model what the page is about before it reads a single line. /page?id=47 tells it nothing. ## Sync Structured Data to Your Main Domain Subdomains carry less authority than root domains. AI crawlers that discover [example.com](https://example.com/) won’t automatically find [docs.example.com](https://docs.example.com/) or [api.example.com](https://api.example.com/). If your llms.txt, project pages, and API data are spread across subdomains, AI may only see part of the picture. The fix is to mirror key structured data onto your main domain so that [example.com/llms.txt](https://example.com/llms.txt), [example.com/projects/xxx.md](https://example.com/projects/xxx.md), and [example.com/api/projects.json](https://example.com/api/projects.json) all live under one roof. AI crawlers discover your main site through search indexing and find everything without leaving. Implementation options include scheduled CI sync, build-time fetching, or reverse proxying. I use a GitHub Action that syncs subdomain data to the blog repo nightly.  When launching a new site, use a checklist to avoid gaps. Core items: robots.txt (categorized crawler permissions), llms.txt (site summary with cross-references), sitemap (submitted to search engines), Bing Webmaster Tools (enable IndexNow), Google Search Console (monitor indexing). Each site’s llms.txt should reference the others, forming a discovery mesh. The easiest trap when doing this work is getting carried away with every GEO technique you come across and trying to add them all, creating a mess and losing sight of what matters. ## These Didn’t Work ** and **: no spec, no adoption by any major AI system. **/.well-known/ai.txt**: multiple competing proposals, no real adoption. Wait for a winner. **HTML comments with AI hints**: parsers strip comments before AI sees the content. **User-Agent sniffing to serve Markdown**: returning different content to bots versus humans is cloaking. Google will penalize you. **Unofficial AI meta tags**: unless a major AI provider explicitly documents support, it’s just noise. ## JSON-LD Isn’t as Useful as You’d Think I initially thought JSON-LD would be powerful for AI visibility. Deeper research showed a more complicated picture. SearchVIU ran an experiment where they put data only in JSON-LD without showing it on the page. All five AI systems they tested failed to find it. Mark Williams-Cook’s follow-up experiment showed that LLMs treat