Artificial Intelligence changes the way we create and view content on the internet. AI tools are useful, but many website owners want their content not to be used for AI training without permission.
This is why we created the llms.txt file. This is a small file that tells AI crawlers whether or not their content can be used for AI training.
If you are running a WordPress website and want to protect your content, this guide is for you. In this we will explain:
What is llms.txt
- How is it different from robots.txt
- How to install it in WordPress site
If you want to protect your content, care about copyright, or just want to have control over how AI views your site then it is important to know about llms.txt
Want to stay ahead with AI-driven WordPress insights and stay updated with the latest trends? Subscribe for daily search insights at wpguidepro to improve your WordPress strategy!
Table of Contents
What Is an llms.txt File and Why Do You Need One?
llms.txt is a text file that tells AI crawlers and AI tools how to behave with the content on your website. This file is a kind of polite but clear message that tells the rules of your website.
This file states which part of the website can be used for AI training and which cannot. It is not like the complex rules of privacy policy, rather it is straightforward and simple which the AI system can easily understand.
Placing this file can be important for:
Content protection
Writing your blog and articles takes hard work. llms.txt helps ensure that no one uses your content in AI training without your permission.
Copyright control
Many people want that their written content should not be used in AI without their permission. This file explains this to AI.
Professional way
If you have a business, then this file shows that you take care of digital rights.
Thinking about the future
As AI is growing, this file will help protect your content from new AI tools
llms.txt vs. robots.txt: What’s the Difference?
Both robots.txt and llms.txt files are for automatic systems, but their work is different. If you understand their difference, then you can take a better decision about which file is needed.
robots.txt is being used since 1994 and its work is to tell the search engine bots which pages of the website they are allowed to view and show. This file affects whether your website is shown in the search result.
llms.txt is a new file which is only for AI training bots. Its job is to tell whether your content can be used for machine learning or not. This file does not affect search engine ranking.
The biggest difference is this: robots.txt talks to search engines, while llms.txt sends messages to AI training systems.
Some website owners use both files so that there are clear rules for all types of bots.
Some AI companies accept robots.txt, but there are some that do not follow this rule. So if you want to protect your content from AI, then having llms.txt file is a must.
Method 1: Create an llms.txt File Using AIOSEO
With the help of AIOSEO plugin, you can easily create and manage llms.txt file. This method is better for those people who do not want to edit the file manually.
Step 1: Install and activate AIOSEO Plugin
First of all go to your WordPress dashboard.
Then click on Plugins > Add New and search for All in One SEO.
Install and activate any of the Free or Pro versions.
After activating, the AIOSEO option will appear in the side menu of WordPress.
From there click on Tools, then go to Robots.txt Editor.
This section is originally for robots.txt, but now newer versions of AIOSEO also have the option for llms.txt.
There you can easily create your llms.txt file.
Just write your rules you want in the text box.
Some common directives you can use are:
- User-agent: *
- Disallow: /
- Allow: /public-content/
Step 2: Check the llms.txt file
Once the file is created, go to your browser and check it:
yoursite.com/llms.txt If the file is displaying correctly, the rules you wrote will appear there in plain text.
If the file is not visible, check:
- Have you saved the changes in AIOSEO or not
- Is there any caching issue in your WordPress site
Method 2: Create an llms.txt File Manually
If you know how to handle computer files, then creating the llms.txt file manually gives you complete control.
Step 1: Create the file First, use any simple text editor on your computer, such as:
- Notepad (Windows)
- TextEdit (Mac)
- Any code editor
Create a new file and name it llms.txt. Make sure that the file name is just llms.txt it should not have any extra extension like .txt.txt
Step 2: Write your rules
Now write the rules inside this file that you want to tell the AI crawlers. A simple example could be this:
Preferences for AI Training
- User-agent: *
- Disallow: /
- Allow: /public-resources/
These rules mean: All AI bots cannot access your entire site, they can only see the folder with “public-resources”.
Step 3: Upload the file to the website
When the file is ready, upload it to the root folder of your website.
There are 3 ways to upload:
- Using FTP software
- Using hosting panel file manager
- WordPress file editor
Root folder is the place where your wp-config.php file is located.
Step 4: Check the file After uploading, go to your browser and check: Yourwebsite.com/llms.txt
If the file is showing correctly, it means everything is working fine.
Bonus: How to Block AI Bots Using Your robots.txt File
llms file is specifically for AI tools, but your old robots.txt file can also help stop AI crawlers.
This method is good for those who are already using robots.txt. Many AI systems still follow the rules of robots.txt.
Many AI trained crawlers have their own unique user-agent name so you can identify and stop them.
Some of the popular AI crawlers are named as:
- GPTBot
- ChatGPT-User
- CCBot
- anthropic-ai
If you want these bots not to use the content of your site, you can write this in the robots.txt file:
User-agent: GPTBot Disallow: /
User-agent: ChatGPT-User Disallow: /
User-agent: CCBot Disallow: /
User-agent: anthropic-ai Disallow: /This means: these bots are not allowed to see any part of your site.
Method 1: Edit robots.txt Using the AIOSEO Plugin (Recommended)
With the AIOSEO plugin, you can easily edit the robots.txt file from within the WordPress dashboard.
To do this:
- Go to the WordPress admin panel
- Click on AIOSEO > Tools > Robots.txt Editor
Here you will find a simple interface where you can add rules directly without opening any server file.
If you want to keep AI bots away from your site, then paste the rules given below:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /The meaning of these rules is that these AI bots cannot reach any part of your website.
FAQs About llms.txt and robots.txt in WordPress
Do all AI companies respect the llms.txt file?
No, not every AI company follows llms.txt yet. This is a new method. But many big AI companies have said that they will listen to this file. Slowly more companies are following this.
Can I use both llms.txt and robots.txt?
Yes, of course I can. Some website owners use both files to control all kinds of bots. Both files do different work, but together they provide better protection.
Will blocking AI crawlers make my website appear less in Google?
No. llms.txt and robots.txt only block AI trained bots. They don’t have any effect on search engines like Google so your SEO will remain safe.
When should the llms.txt file be updated?
Check the file after every few minutes. When you add new pages or sections to your website, or if new AI bots arrive, it’s a good idea to update the file.
What if I don’t have a llms.txt file?
If this file is not there, then many AI bots assume that your content can be used. Not every bot does this, but if you want them to keep your wishes in mind, then this file is important.
