Google’s New AI Training Control in Robots.txt: What Businesses Should Set

Google just announced support for a shared AI training control in robots.txt — one signal that tells participating AI providers whether your content may be used to train their models. It sounds like a simple privacy switch. For businesses that want to be found in AI search, choosing the wrong setting can quietly cost visibility.

This guide explains what Google announced, how the new policy token relates to Google-Extended, what it does and does not affect, and how to decide on the right setting for a business website.

What Google Announced

Google said it plans to support a shared AI training control in robots.txt, based on work from the AIPREF working group at the Internet Engineering Task Force (IETF). Instead of writing separate rules for every AI company, a site owner can use a shared “policy token” to express broad training preferences across participating providers.[1]

The key rule: if your robots.txt uses the shared token to opt out of training and does not explicitly mention Google-Extended, Google plans to treat it as if you had disallowed Google-Extended.[1]

Existing company-specific controls continue to work. That means you can set one broad preference with the shared token, then tailor rules for individual providers where you want a different outcome. Google describes the announcement as a plan, so watch for its final documentation before relying on exact syntax.

Training Versus Search: The Distinction That Matters

Robots.txt now carries two very different kinds of decisions. One controls whether search crawlers can find and rank your pages. The other controls whether your content can be used to build or ground AI models. Mixing them up is the most common — and most expensive — mistake.

Google notes that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.[1] In other words, a blanket opt-out can remove a discovery channel, not just a training use. Read our guide to technical SEO for generative search and AI agents for how crawler access shapes AI visibility.

Should Your Business Opt Out?

Publishers whose content is the product — news sites, paid research, original courses — have real reasons to restrict training. A local service business or B2B company is usually in the opposite position: its pages exist to be found, quoted, and contacted. For those sites, being understood by AI systems supports the same goal as ranking in Google.

Google is also sending more traffic paths out of its AI results, with more inline links and a test section that highlights official websites for specific tasks.[2] Our breakdown of Google AI search links covers how to become one of those sources.

Practical Robots.txt Checklist

• Audit your current file: Open yourdomain.com/robots.txt and list every AI-related user agent and rule already in place.
• Never block Googlebot to stop AI: Blocking Googlebot removes you from search results, including AI Overviews, rather than just training.
• Decide training policy deliberately: Agree internally whether training use helps or hurts your business before adding the shared token.
• Set Google-Extended explicitly: If you opt out broadly but still want Gemini grounding, allow Google-Extended by name so Google does not infer a disallow.
• Review plugin defaults: Some CMS and security plugins add AI blocks automatically. Confirm they match your decision.
• Re-check after changes: Test the file in Search Console and record the date, so traffic shifts can be traced back.

Robots.txt mistakes are among the fastest ways to lose visibility. Our technical SEO services include crawler-access audits that cover both search and AI user agents.

Common Mistakes to Avoid

• Copying a “block all AI” template without reading it
• Disallowing Googlebot to keep content out of AI Overviews
• Assuming robots.txt removes content already used in training
• Forgetting that one opt-out token now affects Google-Extended

Where This Fits in Your AI Search Strategy

Access is only the first gate. Once AI systems can read your pages, they still need clear facts, strong entity signals, and trustworthy sources before they recommend you. Our AI SEO services handle that full path, and our AI search statistics page tracks how quickly these surfaces are growing.

To see whether your choices are paying off, connect AI visibility to leads with the methods in our AI search attribution guide.