🤖 AI Bot Tracking: Prove Your llms.txt Is Working

Why AI bot tracking is essential

Most analytics tools – Google Analytics, Plausible, Matomo – measure user traffic by running JavaScript in the browser and setting cookies. AI crawlers do neither. They load pages server-side, don't execute JavaScript and don't accept cookies. The result: your analytics show zero visits from GPTBot or ClaudeBot – even though these bots may be calling your site every day.

This creates a dangerous information vacuum: you're optimizing for AI visibility, but you don't know whether your efforts are landing. AI bot tracking closes this gap. It provides the only reliable answer to the question: "Is ChatGPT actually reading my llms.txt?"

What AI bot tracking concretely shows

• Which AI systems are active: GPTBot, ClaudeBot, PerplexityBot, Google-Extended – or none at all
• When the last visit was: Date and time of the most recent llms.txt fetch
• How often crawls happen: Frequency over 30 days as a bar chart
• Which domains are covered: With multiple domains under one Pro license, each is shown separately

How tracking works technically

Tracking is based on a server-side .htaccess redirect for known AI user agents. When GPTBot fetches your llms.txt, a rewrite rule fires: the bot is briefly redirected to our tracking endpoint, which logs the access with a timestamp and user agent – then delivers the original llms.txt completely unchanged.

The technical flow in three steps:

1. Bot requests your llms.txt GPTBot, ClaudeBot or another AI crawler fetches https://yourdomain.com/llms.txt – as usual.
2. Apache rule fires (302 redirect) mod_rewrite detects the AI user agent and briefly redirects to api/bot-tracker.php. Runtime: under 50 ms.
3. llms.txt delivered unchanged The tracker logs domain, user agent and timestamp, then responds with the original llms.txt content.

Requirements

• Apache web server with mod_rewrite (standard at most shared hosting providers)
• An active Pro license for llmstxtgenerator.de
• A reachable llms.txt in your webroot (https://yourdomain.com/llms.txt)

Step by step: Setting up tracking

Setup takes under 3 minutes and only requires FTP access to your .htaccess file.

1. Activate your Pro license Log in at /en/tools/bot-stats/ with your license key. Format: LLMS-XXXX-XXXX-XXXX-XXXX.
2. Copy the snippet The bot tracking tool shows you a pre-configured .htaccess snippet for your domain. Just click "Copy".
3. Insert into .htaccess Open the .htaccess in your shop's root directory via FTP (enable hidden files). Add the snippet at the very end – after all other rules.
4. Save and wait As soon as the next AI crawler visits, the entry appears in your statistics. New accesses show up with up to 1 hour delay.

The snippet looks like this:

The 13 detected AI crawlers

Most relevant for e-commerce: GPTBot, ChatGPT-User and PerplexityBot are the platforms actively used for product recommendations. If these three aren't reading your llms.txt, you're missing the direct channel to the most widely used AI assistants.

Understanding and interpreting results

After setup, the bot statistics tool shows three key metrics at a glance: total fetches (last 3 months), number of detected AI systems and number of domains. Below that is a 30-day bar chart and a per-domain table.

What do the different scenarios mean?

Crawl frequency: What's normal?

AI crawlers behave very differently from Googlebot. While Google visits your site daily or hourly, AI bots come much less frequently:

• GPTBot: Every 1–3 weeks on active domains
• ClaudeBot: Irregular, sometimes weekly
• PerplexityBot: More frequent, sometimes daily for highly relevant content
• Google-Extended: Tied to the regular Googlebot crawl cycle

No visits in the first 7 days after setup are normal. Most sites see their first crawler visits after 1–4 weeks.

When no bot shows up – causes and fixes

If tracking shows no visits after more than two weeks, work through this checklist in order:

1. Is your llms.txt directly reachable?
   Open https://yourdomain.com/llms.txt in your browser. It should load without any redirect. If it's missing or returns 404, use the Generator.
2. Is robots.txt blocking AI crawlers?
   Check with the robots.txt AI Check whether GPTBot or ClaudeBot are being blocked. A Disallow: / for these user agents prevents all crawler visits.
3. Is the snippet in the right place?
   The .htaccess snippet must go at the end of the file – after all other rewrite rules. If it's placed before them, shop-specific rules can override it.
4. Is mod_rewrite active?
   On some hosts, mod_rewrite needs to be enabled in the hosting control panel. When in doubt, contact your hosting support.
5. Was the llms.txt submitted anywhere?
   Active submission via AI Push significantly accelerates discovery – instead of passively waiting for the next crawl cycle.

Frequently asked questions

Does tracking slow down my website?

No. The redirect only applies to known AI user agents, not to regular visitors. The tracking request takes under 50 ms and has no impact on page load time for your users.

Is my content transferred to third parties?

The tracking system only logs: domain name, crawler user agent and timestamp. No content, no visitor IP addresses, no cookies. The data is used exclusively to display your statistics.

Does it work with llms-full.txt too?

Yes. The snippet monitors both llms.txt and llms-full.txt – as shown by llms(-full)?\.txt in the RewriteRule.

What happens if I remove the snippet?

AI crawlers will then fetch your llms.txt directly again – without tracking. All previously stored statistics remain and are still accessible via the bot statistics tool.

Can I track multiple domains with one license?

Yes. Pro licenses support multiple domains. The statistics tool shows each domain with its own history and crawler breakdown.

From tracking to a complete GEO strategy

AI bot tracking is the proof that your optimization efforts are landing. But it's only one part of a complete GEO strategy:

1. Audit – Check robots.txt and AI visibility (→ Full Audit)
2. Create – Generate llms.txt with real product data (→ Generator)
3. Submit – Actively submit to AI platforms (→ AI Push)
4. Measure – Set up bot tracking, establish a baseline (→ Bot Statistics)
5. Repeat – Update llms.txt whenever your product range changes