Cloudflare: Stay discoverable in search while disallowing AI training

Cloudflare has introduced a new feature that allows website owners to allow or disallow crawlers from training AI models, while still being discoverable in search results. The feature is called 'Accountable Mixed Use Crawlers' and it uses a new HTTP header called 'CF-Do-Not-Train'. This header can be used to signal to crawlers that they should not be used for AI training, but still crawl the website for indexing purposes. Cloudflare's goal is to help website owners balance their desire for discoverability with their need to prevent AI training.