REST API Documentation

Programmatic llms.txt generation for Agency customers. Crawl websites and generate structured AI files – fully automated via API call.

1. Authentication

Every API request must be authenticated. Your API key is identical to your Agency License key (format: LLMS-XXXX-XXXX-XXXX-XXXX).

There are three ways to pass the key (in priority order):

Pro and Free keys will receive a 403 error with a note about the Agency upgrade path.

2. Endpointe

All endpoints are located under the base URL:

https://www.llmstxtgenerator.de/api/generate.php

3. Website generate (POST)

Request-Body (JSON)

Ablauf im Detail

After receiving your request, the API goes through the following steps:

1. Sitemap-Suche – The API checks /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and also parses /robots.txt for sitemap entries. Sitemap indexes are resolved recursively.

2. Fallback to Homepage Links – If no sitemap is found, all internal links from the homepage are extracted.

3. Metadata Crawling – For each URL, <title>, <meta description>, <h1> and Schema.org @type annotations are extracted.

4. Categorization – Each URL is automatically categorized (main pages, products, categories, blog, guides, FAQ, about us, legal) based on URL paths and Schema.org types.

5. Generation – A structured llms.txt and an extended llms-full.txt are generated from the metadata.

4. Status & Kontingent (GET)

GET /api/generate.php?action=status
Header: X-API-Key: LLMS-XXXX-XXXX-XXXX-XXXX

Response

{
  "status": "active",
  "plan": "agency-monthly",
  "expires": "2026-04-22",
  "usage_today": {
    "requests": 12,
    "limit": 100,
    "remaining": 88,
    "urls_crawled": 340
  }
}

5. Response-Struktur

Eine erfolgreiche Generierung liefert folgende JSON-Struktur:

{
  "success": true,
  "url": "https://example.com",
  "pages_found": 87,
  "pages_crawled": 50,
  "duration_seconds": 14.3,
  "llms_txt": "# Example.com\n> Description...\n\n## Hauptseiten\n...",
  "llms_full_txt": "# example.com – Complete page overview\n...",
  "generated_at": "2026-02-22T15:30:00+01:00",
  "warnings": ["https://example.com/old-page: HTTP 404"]
}

6. Code-Examplee

curl -X POST https://www.llmstxtgenerator.de/api/generate.php \
  -H "Content-Type: application/json" \
  -H "X-API-Key: LLMS-XXXX-XXXX-XXXX-XXXX" \
  -d '{"url": "https://example.com", "max_urls": 100}'

import requests

response = requests.post(
    "https://www.llmstxtgenerator.de/api/generate.php",
    headers={"X-API-Key": "LLMS-XXXX-XXXX-XXXX-XXXX"},
    json={"url": "https://example.com", "max_urls": 100}
)

data = response.json()

# Save llms.txt directly as a file
with open("llms.txt", "w") as f:
    f.write(data["llms_txt"])

print(f"{data['pages_crawled']} pages crawled in {data['duration_seconds']}s")

const response = await fetch(
  "https://www.llmstxtgenerator.de/api/generate.php",
  {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "X-API-Key": "LLMS-XXXX-XXXX-XXXX-XXXX"
    },
    body: JSON.stringify({
      url: "https://example.com",
      max_urls: 100
    })
  }
);

const data = await response.json();
console.log(data.llms_txt);
console.log(`${data.pages_crawled} pages in ${data.duration_seconds}s`);

$ch = curl_init("https://www.llmstxtgenerator.de/api/generate.php");
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => [
        "Content-Type: application/json",
        "X-API-Key: LLMS-XXXX-XXXX-XXXX-XXXX"
    ],
    CURLOPT_POSTFIELDS => json_encode([
        "url" => "https://example.com",
        "max_urls" => 100
    ])
]);

$response = curl_exec($ch);
curl_close($ch);

$data = json_decode($response, true);

// Save llms.txt directly as a file
file_put_contents("llms.txt", $data["llms_txt"]);

echo $data["pages_crawled"] . " pages crawled\n";

Practical example: Batch generation for multiple clients

#!/bin/bash
# Monthly llms.txt generation for all clients
API_KEY="LLMS-XXXX-XXXX-XXXX-XXXX"

KUNDEN=(
  "https://kunde1-shop.de"
  "https://kunde2-agentur.com"
  "https://kunde3-handwerk.de"
)

for url in "${KUNDEN[@]}"; do
  DOMAIN=$(echo "$url" | sed 's|https://||;s|/.*||')
  echo "Generiere: $DOMAIN ..."

  curl -s -X POST https://www.llmstxtgenerator.de/api/generate.php \
    -H "Content-Type: application/json" \
    -H "X-API-Key: $API_KEY" \
    -d "{\"url\": \"$url\", \"max_urls\": 200}" \
    | jq -r '.llms_txt' > "${DOMAIN}-llms.txt"

  echo "  → ${DOMAIN}-llms.txt saved"
  sleep 2  # Fair-Use-Pause
done

import requests, time

API_KEY = "LLMS-XXXX-XXXX-XXXX-XXXX"

kunden = [
    "https://kunde1-shop.de",
    "https://kunde2-agentur.com",
    "https://kunde3-handwerk.de",
]

for url in kunden:
    domain = url.replace("https://", "").replace("/", "")
    print(f"Generiere: {domain} ...")

    r = requests.post(
        "https://www.llmstxtgenerator.de/api/generate.php",
        headers={"X-API-Key": API_KEY},
        json={"url": url, "max_urls": 200},
        timeout=120
    )

    data = r.json()
    if data.get("success"):
        with open(f"{domain}-llms.txt", "w") as f:
            f.write(data["llms_txt"])
        print(f"  → {data['pages_crawled']} pages, {data['duration_seconds']}s")
    else:
        print(f"  → Error: {data.get('error')}")

    time.sleep(2)  # Fair-Use-Pause

7. Limits & Fair Use

8. Error Codes

Error responses always follow this format:

{
  "error": "Error description",
  "code": 422,
  "detail": "Optional additional information"
}

9. FAQ