Agentic Browsing: LLMS.TXT PageSpeed Error – Causes, Fixes, Sample File & Complete Setup Guide

Introduction

As AI-powered search engines and autonomous browsing agents become more common, website owners are increasingly adopting llms.txt files to help Large Language Models (LLMs) understand and discover their content.

However, many webmasters are now encountering warnings such as:

“Agentic Browsing – llms.txt does not follow recommendations”

This issue often appears in AI audits or performance tools and usually indicates that your llms.txt file is incomplete, improperly structured, or missing important sections recommended by the emerging llms.txt specification.

In this guide, you’ll learn:

• What llms.txt is
• Why the “does not follow recommendations” warning appears
• How to fix the issue step by step
• A complete sample llms.txt file
• Best practices for AI SEO and agentic browsing
• Common mistakes to avoid

What Is Agentic Browsing?

Agentic browsing refers to AI systems that independently navigate websites, gather information, and complete tasks on behalf of users.

Unlike traditional search crawlers, these AI agents:

• Read documentation
• Compare products
• Summarize articles
• Perform research
• Recommend businesses
• Generate answers from multiple sources

To make this process easier, websites can publish a structured llms.txt file that highlights important pages and content.

What Is LLMS.TXT?

llms.txt is a plain text or Markdown-formatted file placed at the root of a website:

https://example.com/llms.txt

Its purpose is to provide AI assistants and autonomous agents with a curated overview of your website, including:

• Business information
• Key services
• Documentation
• Important URLs
• Contact details
• Structured navigation

Unlike robots.txt, which controls crawler access, llms.txt is designed to improve understanding and discovery.

Why Do You See the “LLMS.TXT Does Not Follow Recommendations” Error?

The warning generally means that your file doesn’t align with commonly accepted formatting and content recommendations.

Common causes include:

1. Missing H1 Title

Every llms.txt should begin with a top-level heading.

Incorrect

Company Information

...

Correct

# Company Name

2. Missing Description Block

Immediately after the H1, include a concise summary of your business.

Example:

> Crawl Brains is a digital marketing and Generative Engine Optimization agency helping businesses improve visibility across search engines and AI platforms.

3. Poor Heading Structure

Use logical Markdown headings.

Recommended hierarchy:

# Website Name

> Description

## Services

- SEO
- GEO

## Resources

- Blog
- Documentation

## Contact

4. Missing Important URLs

AI agents benefit from direct links to:

• Homepage
• Services
• Blog
• Contact page
• Sitemap
• Documentation
• Support pages

Avoid listing only a few pages if your site has many key resources.

5. Lack of Context

Instead of listing URLs alone:

- https://example.com/service

provide context:

- SEO Services: Comprehensive search optimization for businesses.

6. Inconsistent Formatting

Keep formatting consistent throughout the file:

• Use Markdown headings
• Use bullet lists
• Add descriptions
• Avoid unnecessary HTML
• Avoid excessive promotional language

Sample LLMS.TXT File

Below is a simplified example following current best practices.

https://crawlbrains.ae/llms.txt

# Crawl Brains

> Digital marketing and Generative Engine Optimization (GEO) agency based in Dubai, UAE. 16+ years of experience, serving 130+ clients across 12+ industries in the UAE, Kuwait, Qatar, Bahrain, and Saudi Arabia. Bilingual (English/Arabic) team specializing in SEO, GEO, paid advertising, and digital analytics.

## Services

- [SEO Services](https://crawlbrains.ae/seo-company-agency-dubai/): Core search engine optimization services to grow organic visibility and traffic.
- [Local SEO](https://crawlbrains.ae/local-seo-services-dubai/): SEO for businesses targeting local and regional search results.
- [Multilingual SEO](https://crawlbrains.ae/multilingual-seo-services-dubai/): SEO strategy and execution across multiple languages.
- [E-Commerce SEO](https://crawlbrains.ae/ecommerce-seo-dubai/): SEO for online stores and product pages to improve sales and rankings.
- [Arabic SEO](https://crawlbrains.ae/arabic-seo-services-dubai/): SEO tailored specifically to Arabic-language search behavior.
- [Global / International SEO](https://crawlbrains.ae/international-seo-services-dubai/): SEO for businesses expanding into multiple countries and markets.
- [SEO Consulting](https://crawlbrains.ae/seo-consulting-dubai/): Strategic SEO advisory and audits.
- [Wikipedia Page Creation](https://crawlbrains.ae/wikipedia-page-creation/): Creation and management of Wikipedia pages for brands.
- [Content Writing](https://crawlbrains.ae/content-writing-services-dubai/): SEO-driven content and copywriting services.
- [Generative Engine Optimization (GEO)](https://crawlbrains.ae/geo-services-company-dubai/): Optimizing brand visibility across AI search and chat platforms.
- [AI Optimization](https://crawlbrains.ae/ai-optimization-services-dubai/): General optimization services for discovery by AI systems.
- [ChatGPT Optimization](https://crawlbrains.ae/chatgpt-optimization-services-dubai/): Visibility optimization specifically for ChatGPT.
- [Perplexity Optimization](https://crawlbrains.ae/perplexity-optimization-services-dubai/): Visibility optimization specifically for Perplexity.
- [Gemini Optimization](https://crawlbrains.ae/gemini-optimization-services-dubai/): Visibility optimization specifically for Google Gemini.
- [DeepSeek Optimization](https://crawlbrains.ae/deepseek-optimization-services-dubai/): Visibility optimization specifically for DeepSeek.
- [Claude Optimization](https://crawlbrains.ae/claude-optimization-services-dubai/): Visibility optimization specifically for Claude.
- [Influencer Marketing](https://crawlbrains.ae/influencer-marketing-agency-services/): Influencer marketing campaign planning and management.
## Company
- [Home](https://crawlbrains.ae/): Overview of Crawl Brains' digital marketing and GEO services, case studies, and certifications.
- [Contact](https://crawlbrains.ae/contact/): Request a free audit, quote, or strategy call.
- [Blog](https://crawlbrains.ae/blogs/): Articles on SEO, digital marketing, and GEO topics.
- [Sitemap](https://crawlbrains.ae/sitemap/): Full structural map of the website.

## Optional
- Offices in Dubai and Sharjah, UAE, and Hyderabad, India. Contact: hello@crawlbrains.ae, +971 566 06 1239.
- Partner certifications: Google Analytics 4, Google Ads, Meta Business, Snapchat, TikTok, and Semrush.

Real Example Structure

A production-ready llms.txt typically contains:

• Website title
• Business summary
• Services
• Products
• Documentation
• Resources
• Blog
• Contact information
• Sitemap
• Optional office locations or certifications

This structure makes it easier for AI systems to identify the most valuable parts of your website.

How to Fix the PageSpeed or AI Audit Error

Step 1: Create llms.txt

Create a file named exactly:

llms.txt

Step 2: Place It in the Root Directory

Upload it so it is accessible at:

https://yourdomain.com/llms.txt

Step 3: Add a Proper H1

Start with:

# Your Website Name

Step 4: Include a Short Description

Add a blockquote explaining your organization in one or two sentences.

Step 5: Organize Information into Sections

For example:

• Services
• Products
• Resources
• Contact
• Documentation
• Optional Information

Step 6: Use Descriptive Bullets

Instead of bare links, include brief explanations for each page.

Step 7: Keep It Updated

Whenever you launch new services or major resources, update llms.txt accordingly.

Recommended Commands and Formatting

Use Markdown elements consistently:

Main Title

# Website Name

Description

> Short summary of your organization.

Section Heading

## Services

Bullet List

- SEO Services: Organic search optimization.
- Contact: Customer support and inquiries.

Links

- Blog: https://example.com/blog

Common Mistakes to Avoid

❌ Missing H1 heading

❌ Empty file

❌ Plain list of URLs without descriptions

❌ Broken or outdated links

❌ Overly promotional language

❌ Inconsistent Markdown formatting

❌ Omitting key pages such as Contact or Sitemap

Best Practices for AI SEO

To maximize discoverability by AI systems:

• Keep llms.txt concise and well structured.
• Use meaningful section headings.
• Describe pages instead of only linking to them.
• Include your most important services and resources.
• Review and update the file regularly.
• Ensure your website content remains crawlable and technically sound.

Frequently Asked Questions

Does llms.txt replace robots.txt?

No. robots.txt controls crawler access, while llms.txt provides structured information for AI agents.

Does Google require llms.txt?

At present, llms.txt is not a mandatory Google ranking factor. It is an emerging convention intended to help AI systems understand website content more effectively.

Can llms.txt improve AI visibility?

A well-structured file may make it easier for AI-powered tools to identify and navigate key resources on your site, especially when combined with high-quality content and sound technical SEO.

Where should I place the file?

At the root of your domain:

https://yourdomain.com/llms.txt

Final Thoughts

As AI-driven search and agentic browsing continue to evolve, maintaining a well-organized llms.txt file can complement your broader technical SEO and content strategy. By including a clear title, concise business summary, structured sections, descriptive links, and up-to-date information, you can reduce validation warnings and make your website easier for AI systems to interpret.

For organizations investing in AI visibility, treating llms.txt as a living document—reviewed alongside your sitemap and technical SEO audits—is a practical step toward better discoverability.