The Test Shop Setup
Solar-autark.com is a Gambio GX5-based online shop with around 120 categories and several hundred products in the solar and energy technology sector. The shop has been active since 2003 – it has domain authority, but like most shops, had no AI-specific optimisation before this experiment.
The experiment setup: in early March 2026, llms.txt and llms-full.txt were generated using the LLMs.txt Generator and uploaded to the webroot. At the same time, the AI bot tracking snippet was added to the .htaccess file. Since then, tracking has been running passively in the background – with no further changes to the shop.
One-time effort of about 15 minutes: generate llms.txt, upload it, insert the .htaccess snippet. Fully automated from that point on.
First Data: March 2026
Just a few days after setup, the first visitor arrived: PerplexityBot queried the shop ten times in a single crawl on 28 March 2026. The result was clear: the llms.txt works, and AI crawlers find it.
What stands out: in this first crawl, PerplexityBot overwhelmingly preferred the llms-full.txt (9 out of 10 requests), not the more compact llms.txt. The message is clear: Perplexity wants the complete product structure – categories, manufacturers, prices, availability. Not just the summary.
PerplexityBot prefers llms-full.txt over llms.txt at a 9:1 ratio. The effort of creating a complete product listing pays off.
April 2026: More Bots, Wider Spectrum
April shows a significantly more diverse picture. Within just over two weeks, five different AI systems visited the shop – including well-known names from the AI assistant space and some surprises.
Chronological Timeline
First Applebot visit. Apple is increasingly integrating web content into Siri and Apple Intelligence – a sign that Apple is indexing AI-ready content.
ChatGPT-User is different from GPTBot: here, an active ChatGPT user triggered a live web lookup in their session. The shop was actually consulted in the context of a real user query.
Perplexity returns – this time on two consecutive days with one request each to the llms.txt. Regular crawling is a positive sign of active indexing.
ByteDance, TikTok's parent company, crawls web content for its AI products. Bytespider is one of the most active AI crawlers worldwide – even if TikTok itself has limited reach in some markets.
What the Bots Actually Read
In April, reading behaviour changed significantly compared to March:
| Period | Bot | Requests | File | Status |
|---|---|---|---|---|
| March | PerplexityBot | 10 | llms-full.txt (9×), llms.txt (1×) | Active |
| April | Applebot | 1 | llms.txt | Active |
| April | ChatGPT-User | 2 | llms.txt | Active |
| April | Meta-ExternalAgent | 1 | llms.txt | Active |
| April | PerplexityBot | 2 | llms.txt | Active |
| April | Bytespider | 1 | llms.txt | Active |
| Total | 17 requests, 5 different AI systems | |||
Notably: in April, all bots exclusively read the llms.txt (short version), no longer the llms-full.txt. Possibly, PerplexityBot indexed the full file on the first crawl and now uses the compact version for regular updates. The other bots use the short version as their standard entry point.
Honest Assessment of the Data
It would be wrong to overstate these numbers. Here is a frank evaluation:
What the Data Shows
- The system works technically: bots find and read the llms.txt.
- The diversity of bots is impressive – 5 different AI systems in 2 weeks.
- PerplexityBot crawls regularly, suggesting active indexing.
- ChatGPT-User is particularly valuable: this is not a crawler but a real user query in action.
What the Data Does Not Show
- No direct revenue attribution: whether an AI user makes a purchase after the bot read the llms.txt is not trackable – not yet.
- No recommendation guarantee: a bot reading the file does not automatically mean the shop will be recommended in AI responses.
- Small sample size: 17 requests over ~6 weeks is a good start, but not a statistically robust dataset.
- GPTBot is still absent: OpenAI's training crawler has not yet appeared – only ChatGPT-User (real-time search).
- ClaudeBot is absent: Anthropic's crawler has not shown up yet. This may be due to crawl frequency or the domain not yet being in the crawl queue.
AI bot tracking proves technical accessibility – not AI recommendation. The journey from "bot has read" to "user gets recommended" can take weeks or months, depending on content quality, authority and competition.
What's Still Missing – and What We Expect
The experiment continues. Some observations and open questions:
GPTBot vs. ChatGPT-User
The difference between these two matters: GPTBot crawls for training future models – its visit would have long-term impact. ChatGPT-User retrieves content in real time when a user actively asks ChatGPT something. The ChatGPT-User visit is more immediate but also more direct: someone just asked ChatGPT about solar products or topics related to solar-autark.com.
Crawl Frequency and Patterns
PerplexityBot visited on two consecutive days (6th and 7th April) with one request each – suggesting a regular update crawl cycle. Applebot, Meta and Bytespider each appeared once. It remains to be seen whether they return and at what frequency.
The experiment confirms: an optimised llms.txt leads to measurable crawl activity from relevant AI systems within a few weeks. This is the prerequisite for AI visibility – not a guarantee, but the necessary first step.
Pro Analytics: Time Series, Bot Comparison and CSV Export
Since April 2026, the bot tracking dashboard offers four analysis tabs. Here are all four – with the real data from solar-autark.com:
Tab 1: 30-Day Overview
Stacked bars show daily which AI systems accessed the llms.txt. The monitored domain appears as a badge in the chart title:
Tab 2: Time Series Analysis
Line charts show crawl patterns over time. PerplexityBot is the only crawler that was active 10 times in a single run (March 28) – then only sporadically. ChatGPT-User first appears in April:
Tab 3: Bot Comparison
Horizontal bars for all detected AI crawlers, sorted by activity. Grey: known systems not yet seen:
Tab 4: CSV Export
One click exports daily raw data as a CSV file – here is an excerpt of the documented requests on solar-autark.com:
Conclusion and Next Steps
After about six weeks of tracking, we can say: AI bot tracking works, and the data is more informative than expected. Five different AI systems – including ChatGPT, Perplexity, Meta AI, Applebot and Bytespider – visited solar-autark.com. That is not something that happens by default.
For online shops and content sites: those without an llms.txt today, and who are not tracking AI crawlers, have no baseline data for the months ahead, as AI-powered search continues to grow. Setup takes 15 minutes. Monitoring and optimising is then the ongoing task.
Over the coming weeks, we are watching whether GPTBot and ClaudeBot appear, how PerplexityBot's crawl frequency develops, and whether ChatGPT-User traffic evolves into a measurable pattern. We will report back.
Set Up Your Own AI Bot Tracking
In 15 minutes, you'll know whether and which AI systems are visiting your website – free with the LLMs.txt Generator.
Start Bot Tracking →