llms.txt and the Fight over AI Access to Your Website Content

Artificial intelligence has changed the relationship between websites and visitors faster than most organizations expected.

For years, websites were built around a fairly simple model. Search engines crawled content, indexed pages, and then sent visitors back to the original source. Publishers, associations, nonprofits, media organizations, and brands all benefited because visibility usually led to traffic.

AI systems are changing that equation.

Today, platforms like ChatGPT, Google Gemini, Claude, Perplexity, and other AI assistants increasingly answer questions directly instead of simply pointing users toward websites. In many cases, users receive summaries, recommendations, explanations, and synthesized answers without ever clicking through to the original source.

That shift has sparked an important conversation across the digital publishing world:

If AI systems are consuming website content to generate answers, what do website owners receive in return?

That is where concepts like llms.txt, pay-per-crawl systems, AI licensing models, and AI content restrictions are beginning to emerge.

The conversation is still early, and standards are still forming, but many organizations are starting to recognize that AI content access may eventually become one of the biggest digital governance issues of the next decade.

What Is llms.txt?

llms.txt is a Markdown file a website can publish to help large language models quickly understand its key pages, documentation, and preferred context. Rather than controlling crawler access, the file serves as an AI-friendly guide that points models toward the most useful version of a site’s content.

The simplest way to understand llms.txt is to think of it as a possible future equivalent to robots.txt, but specifically designed for AI systems.

For decades, robots.txt files have allowed websites to communicate with search engine crawlers. A website owner could indicate which sections of a site should or should not be crawled by search engines. The proposed llms.txt concept attempts to create a similar mechanism for large language models and AI crawlers.

In theory, an llms.txt file could allow organizations to define whether AI systems are permitted to train on content, summarize it, quote it, or access specific areas of a website. It could potentially establish attribution expectations, licensing requirements, approved content sources, or monetization rules tied to AI access.

At the moment, there is no universally adopted standard. Different AI companies use different crawlers, different policies, and different approaches to respecting publisher controls.

Current research suggests that major AI search and LLM platforms are not meaningfully using llms.txt as a standard input today, even though the file was proposed as a way to give LLMs a clean, Markdown-based guide to website content. Google’s own guidance says site owners can ignore llms.txt for Google’s generative AI search features, while OpenAI’s crawler documentation points site owners to robots.txt controls for GPTBot and OAI-SearchBot rather than llms.txt. The strongest evidence comes from log and field studies: OtterlyAI’s 90-day experiment saw only 84 /llms.txt visits out of 62,100+ AI-bot hits, SE Ranking’s analysis of nearly 300,000 domains found no correlation between having llms.txt and being cited by AI systems, and Search Engine Land’s 10-site before/after study found no measurable lift in AI crawl frequency or AI traffic attributable to the file.  The nuance is that some documentation-heavy ecosystems do publish llms.txt files, and tools or agents may fetch them when explicitly directed—Perplexity and Claude Code docs, for example, point developers to documentation indexes in this format—but that is different from evidence that ChatGPT, Claude, Gemini, or Perplexity automatically check arbitrary websites’ llms.txt files during normal answer generation. 

Still, the growing discussion around llms.txt reflects something important:

Website owners increasingly want clearer control over how AI systems access and use their content.

Why Publishers and Organizations Are Concerned

Many organizations invest heavily in creating authoritative content. Associations publish research. Nonprofits produce educational resources. Healthcare organizations maintain guidance libraries. Media companies employ editorial teams. Businesses invest in thought leadership, documentation, case studies, and technical resources.

Historically, search engines rewarded that investment with traffic.

AI changes the economics.

If a user asks an AI assistant a question and receives a synthesized answer drawn from multiple sources, the original publishers may receive little visibility, fewer clicks, and reduced advertising or lead-generation opportunities.

For some organizations, that creates a troubling possibility: their content may still be valuable enough to train or inform AI systems, while becoming less valuable as a direct traffic driver.

That is one reason publishers are beginning to explore licensing arrangements, restricted access systems, and controlled AI crawl policies.

Some organizations are already blocking certain AI crawlers entirely. Others are experimenting with partnerships and licensing agreements. And many are still trying to determine whether AI exposure ultimately helps or harms long-term visibility.

The Rise of Pay-Per-Crawl

One of the more interesting concepts emerging from this debate is the idea of pay-per-crawl.

The logic is relatively straightforward. If AI systems derive commercial value from accessing website content, should content owners be compensated when that content is crawled or used?

Some advocates envision systems where AI companies would pay publishers for access to premium content, license structured datasets, compensate websites based on crawl frequency, or negotiate agreements for access to high-value content libraries. Others imagine revenue-sharing systems tied to AI-generated responses themselves.

This idea is still highly experimental, and there are enormous technical, legal, and operational questions that remain unresolved. Nobody fully knows how usage would be measured, whether compensation would be tied to crawling or actual answer generation, how attribution would work, or what would qualify as fair use.

There are also broader questions surrounding smaller publishers. Would they need APIs instead of open websites? Would only large media organizations benefit from these agreements? Would AI systems favor content behind formal partnerships while ignoring independent publishers?

Despite the uncertainty, the concept itself reflects a growing realization that content may eventually become part of a negotiated AI economy rather than an openly crawlable public resource.

Cloudflare and the Emerging AI Access Layer

One reason this topic has gained momentum recently is that infrastructure providers are beginning to discuss AI access controls more openly.

Cloudflare, for example, has explored ways publishers might gain more visibility into AI crawler activity and more control over how content is accessed.

That matters because companies like Cloudflare sit at the infrastructure layer of the internet itself.

When infrastructure providers begin discussing AI crawl governance, the conversation moves beyond theory, and starts becoming operational.

Website owners increasingly want answers to practical questions:

• Which AI crawlers are visiting the site?
• How often are they crawling?
• Which content are they accessing?
• Are they respecting permissions?
• Can access be restricted selectively?
• Can premium content require agreements?
• Should some AI platforms be allowed while others are blocked?

Why Associations and Nonprofits Should Pay Attention

Many associations and nonprofits may assume this conversation only matters to large media companies. That would be a mistake.

Associations often possess highly specialized expertise that AI systems value enormously. Technical standards, research libraries, educational content, certification materials, policy documentation, conference archives, and professional guidance all represent highly authoritative information sources. In many industries, associations maintain some of the most structured and trustworthy content available online.

That makes them especially attractive sources for AI systems.

The same is true for nonprofits that publish educational resources, healthcare information, policy research, or public advocacy materials. AI platforms naturally seek authoritative, well-organized content, and many nonprofit organizations already provide exactly that.

Over time, organizations may need to decide which content should remain fully public, which content should support AI visibility, which resources require licensing or member access, and whether AI referrals are generating meaningful value back to the organization.

Those decisions will increasingly intersect with content governance, analytics, membership value, AEO/SEO strategy, and long-term digital business models.

AI Visibility Versus AI Extraction

One of the hardest parts of this discussion is that there is no simple answer.

Some organizations benefit from AI visibility.

If an AI assistant cites an organization frequently, that visibility may strengthen authority, awareness, and trust.

For service organizations, consultants, healthcare providers, universities, and many brands, being included in AI-generated answers may become critically important.

But there is also a difference between visibility and extraction.

If AI systems summarize extensive portions of a publisher’s expertise without driving meaningful engagement back to the source, organizations may begin questioning whether the exchange is sustainable.

That tension is likely to define much of the next phase of digital publishing.

The organizations that succeed will probably be the ones that think strategically rather than emotionally.

Blocking every AI crawler may reduce visibility.

Allowing unrestricted access may reduce long-term content value.

Most organizations will eventually land somewhere in the middle.

Structured Content Will Matter More Than Ever

Regardless of where these standards ultimately land, one thing already appears increasingly clear: structured, organized, authoritative content will become even more important in the AI era.

AI systems rely heavily on clarity, hierarchy, metadata, semantic structure, topical authority, and content relationships. Websites with disorganized architecture, inconsistent governance, duplicate content, or unclear authority signals are likely to struggle as AI search and conversational discovery continue evolving.

Organizations that invest in strong information architecture, clear editorial governance, structured data, semantic relationships, accessibility, clean technical SEO, and accurate metadata will likely position themselves far more effectively for both traditional search engines and AI systems.

That is one reason many organizations are beginning to think about “AI readiness” as part of a broader digital strategy rather than as a standalone chatbot project. The quality of the underlying website increasingly shapes the quality of AI visibility and AI-generated discovery.

The Legal Questions Are Just Beginning

Another reason this discussion continues evolving so quickly is that the legal landscape remains unsettled.

Publishers, authors, artists, and media organizations have already begun challenging aspects of AI training and content usage in court.

Questions surrounding copyright, licensing, fair use, attribution, derivative works, and commercial AI output are still being debated.

Different countries may ultimately adopt different regulatory approaches.

Some regions may favor publisher protections.

Others may emphasize open innovation.

Most likely, the final environment will involve a combination of:

• Technical standards
• Licensing frameworks
• Platform agreements
• Legal precedents
• Industry norms
• Infrastructure controls

The internet itself evolved through multiple phases of governance.

AI content governance will likely do the same.

The Future Probably Looks Hybrid

Despite dramatic headlines, the future will probably not become entirely open or entirely locked down.

Instead, organizations will likely adopt layered approaches.

Some content will remain fully public.

Some content will support AI discovery and summarization.

Some premium resources may require licensing or member authentication.

Some organizations may build direct AI APIs. Others may selectively partner with AI platforms.

And many will continue experimenting while the ecosystem stabilizes.

What seems increasingly unlikely is a future where organizations ignore AI crawl governance entirely.

Organizations that create valuable expertise will increasingly want visibility into how that expertise is being consumed, summarized, monetized, and distributed.

Why This Matters Right Now

Many website owners still assume these issues are years away. In reality, the transition is already happening.

AI assistants are already influencing:

• Website traffic patterns
• Search behavior
• Brand discovery
• Lead generation
• Content strategy
• SEO priorities
• User expectations
• Attribution models

The organizations that begin preparing now will likely adapt more effectively than those that wait for formal standards to emerge.

That preparation does not necessarily mean blocking AI systems.

It means understanding the changing environment and building digital platforms that are structured, governed, measurable, and strategically managed.

Preparation, not Panic

The discussion around llms.txt, pay-per-crawl systems, and AI content licensing is really part of a much larger question:

Who controls digital knowledge in the AI era?

For the last twenty years, the web largely operated on an exchange model where publishers provided content and search engines delivered traffic.

AI systems are reshaping that relationship.

The next phase of the internet will likely involve new negotiations around access, attribution, visibility, monetization, and trust.

Some organizations will embrace broad AI visibility. Others will restrict access aggressively.

Most will probably navigate a middle path that balances discoverability with protection.

Organizations that understand their content value, structure their websites intelligently, govern information carefully, and monitor how AI systems interact with their digital assets will be in a far stronger position as this ecosystem evolves.

At New Target, we help organizations build modern digital platforms that are ready for both traditional search and the rapidly emerging AI discovery landscape. From structured content strategy and technical SEO to governance, analytics, accessibility, and AI-ready website architecture, we help associations, nonprofits, government organizations, and enterprise brands prepare for the next evolution of digital visibility.

As AI search continues changing how people discover information online, organizations will need more than attractive websites. They will need digital ecosystems built for trust, clarity, governance, performance, and long-term adaptability.

Contact us for a deeper discussion of these issues.