Kandeshop LLMS.txt Generator — A WordPress Plugin for the AI Crawler Era

The Context

The web has had robots.txt for decades — a simple file that tells search engine crawlers what they can and can't access. But the rise of large language models has introduced a new class of crawler that operates differently: GPTBot (OpenAI), Google-Extended, CCBot (Common Crawl), and others are scraping content not just to index it, but to train AI models, generate summaries, and build knowledge bases — often without any attribution or opt-in from the content owner.

The llms.txt standard is an emerging response to this: a file, placed in a site's root just like robots.txt, that communicates specific policies to AI crawlers. The problem is that creating and managing this file manually is fiddly and error-prone, and most WordPress site owners have no way to do it at all.

What the Plugin Does

Kandeshop LLMS.txt Generator adds a clean settings page to the WordPress admin that lets any site owner — technical or not — configure and generate their llms.txt file without ever touching a server or writing a line of code.

From the settings panel you can:

• Choose which AI user-agents to target — GPTBot, Google-Extended, CCBot, or any combination
• Set policy positions for Training, Summarization, Indexing, and Attribution — Allow or Disallow for each
• Add disallow rules for specific directories or paths (standard WordPress paths like /wp-admin/ are pre-set, with a custom field for anything else)
• Hit Generate Now to force a fresh write at any time, even if nothing has changed
• Download a copy of the current or preview file for manual upload via FTP where host restrictions apply

Save settings and the file is written immediately to the WordPress document root, publicly accessible at yourdomain.com/llms.txt.

Why It Matters

This is a genuinely forward-looking piece of work. The question of who controls how AI systems use web content is one of the defining tensions of the current moment — publishers, news organisations, and individual creators are all starting to push back on unconsented scraping. The llms.txt format gives them a voice in that conversation, and this plugin makes that voice accessible to the 40%+ of the web running WordPress.

The plugin doesn't overstate what it can do — the README is honest that crawler compliance varies and there's no enforced global standard yet. But that's exactly the right framing: it's an advisory signal, and publishing one is increasingly becoming best practice for any site that cares about how its content is used.

Technical Approach

The plugin is built to WordPress coding standards throughout. File writes use WP_Filesystem as the primary method — which means it supports direct writes, FTP, FTPS, and SSH depending on the host's configuration — with a clean fallback to file_put_contents() where needed. All user input is sanitised before being written to the file, and the admin UI provides clear success and failure notices so the site owner always knows what happened and where to find the generated file.

It's compatible with WordPress 6.0+, PHP 7.4+, and has been tested up to WordPress 6.8. Single-file architecture keeps the plugin lightweight with zero dependencies.

Open Source & Published on WordPress.org

The plugin is free, GPL-licensed, and listed on the official WordPress.org plugin directory — meaning it's available to install directly from any WordPress site's plugin search, with automatic update delivery as new versions are released. The source is also publicly available on GitHub.