LESS THAN IT SEEMS: Google and Microsoft among those signing up to OpenAI’s plan for a new website crawling standard — including opting-out of internet scraping.

Google and Microsoft have joined OpenAI and Anthropic in promising to honor a single line in robots.txt that tells AI developers whether a website’s content can be used to train their models.

Until now, a site that wanted to say no had to write a rule for each company’s crawler, and sometimes for several crawlers run by the same company. The shared signal is meant to replace that sprawl with a single entry that every participating provider reads the same way.

That is genuine progress for overworked webmasters, but the promise is narrower than much of the coverage suggests: The new line states a preference about how content may be used rather than stopping bots from fetching it; it is not an approved standard, does nothing about content already collected, and none of the four major players fully honor it yet.

So less of a new standard and more like something the companies involved can point to and claim they’re doing something.