AI crawler tracking

AI tools such as ChatGPT, Claude, and Perplexity send crawlers to read your pages. These crawlers do not run JavaScript, so the tracking script never sees them. To count them, your own server tells Dodolytics about each visit of an AI crawler.

What you get

The Bots report gets a card, AI crawlers on your server:

• Why they came: the hits of three groups of crawlers.
  • AI assistants: a person asked an AI tool, for example ChatGPT, and it read your page for them (ChatGPT-User, Claude-User, Perplexity-User).
  • AI search: indexing for an AI search engine (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
  • AI training: collecting text to train AI models (GPTBot, ClaudeBot, CCBot).
• Each crawler, with its Purpose, its Hits (requests), the number of different Pages it read, and when it was Last seen.
• Pages they read: the 20 pages with the most hits, with the hits of each group. Use the search box above the table to find other pages. * matches any characters: /blog/* finds every page under /blog/. A search with no * finds pages that contain the text.

These hits are not counted in any other report. They are not visitors, so the Dashboard and Pages do not change. The report filters do not apply to this card, and Real-time does not show it.

Before you start

You need your tracking ID. Open the website, click Settings, and copy the Tracking ID. Until your server sends the first hit, the card shows Not set up yet with your tracking ID.

Use one of the three ways below. Each one sends only the requests of AI crawlers. Requests of people are not sent.

Way 1: PHP

Put this code in a file that every page loads, for example wp-config.php or your index.php, on the line after the first <?php. Change YOUR_TRACKING_ID.

// Dodolytics: count AI crawlers.
(function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (!preg_match('/GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Google-CloudVertexBot|GoogleOther|meta-externalagent|meta-externalfetcher|Amazonbot|MistralAI-User|DuckAssistBot|cohere-ai|YouBot|Diffbot|CCBot|Bytespider/i', $ua)) {
        return;
    }
    $ch = curl_init('https://dodolytics.com/collect-bot');
    curl_setopt_array($ch, [
        CURLOPT_POST => true,
        CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
        CURLOPT_POSTFIELDS => json_encode([
            'website_id' => 'YOUR_TRACKING_ID',
            'user_agent' => $ua,
            'path' => $_SERVER['REQUEST_URI'] ?? '/',
            'hostname' => $_SERVER['HTTP_HOST'] ?? '',
        ]),
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT_MS => 1000,
    ]);
    curl_exec($ch);
})();

Only a request of an AI crawler waits for this call, at most one second. Pages for people do not wait.

Way 2: nginx

nginx can copy each crawler request to Dodolytics with mirror, with no change to your website's code. Add this block inside your server { } block. Change YOUR_TRACKING_ID.

location = /_dodolytics_ai {
    internal;
    if ($http_user_agent !~* "GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Google-CloudVertexBot|GoogleOther|meta-externalagent|meta-externalfetcher|Amazonbot|MistralAI-User|DuckAssistBot|cohere-ai|YouBot|Diffbot|CCBot|Bytespider") {
        return 204;
    }
    proxy_pass https://dodolytics.com/collect-bot?website_id=YOUR_TRACKING_ID;
    proxy_method POST;
    proxy_pass_request_headers off;
    proxy_pass_request_body off;
    proxy_set_header Host dodolytics.com;
    proxy_set_header X-Dodolytics-User-Agent $http_user_agent;
    proxy_set_header X-Dodolytics-Path $request_uri;
    proxy_set_header X-Dodolytics-Host $host;
    proxy_ssl_server_name on;
    proxy_connect_timeout 2s;
    proxy_read_timeout 2s;
}

Then add these two lines to one location block: the block that sends your pages.

mirror /_dodolytics_ai;
mirror_request_body off;

• PHP website (for example WordPress): the location ~ \.php$ block. Not location /: there, try_files sends the request on to index.php before the copy is made, and most pages are not counted.
• An app behind proxy_pass (for example Node.js or Python): the block with the proxy_pass to your app.
• Static files: location /, with try_files $uri $uri/index.html =404;. With $uri/ instead, nginx counts a folder address such as /blog/ two times.

Run sudo nginx -t to check the configuration, then sudo systemctl reload nginx. The copy does not send cookies or other headers of the request.

Way 3: any other language

When the User-Agent of a request names an AI crawler, send one POST request with a JSON body:

curl -X POST https://dodolytics.com/collect-bot \
  -H 'Content-Type: application/json' \
  -d '{"website_id":"YOUR_TRACKING_ID","user_agent":"GPTBot/1.2","path":"/blog/post","hostname":"example.com"}'

• website_id: your tracking ID.
• user_agent: the User-Agent of the crawler's request.
• path: the address that the crawler asked for. A full URL is also accepted.
• hostname (optional): the host of your website. Use it if allowed hostnames are on.

The answer is always 204, also when Dodolytics does not keep the hit. Your server does not need to read it.

What Dodolytics keeps

• The name of the crawler, the path without the query string, and the time. Not the IP address.
• Only hits of the AI crawlers on the list below. Dodolytics ignores every other request, also of other bots.
• Hits count only from the time that you add the code. Older visits of crawlers are not known.
• Up to 100,000 hits for each website in a calendar month. More hits that month are not stored. The card shows how many hits are stored this month.

The AI crawlers on the list

GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-CloudVertexBot, GoogleOther, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, MistralAI-User, DuckAssistBot, cohere-ai, YouBot, Diffbot, CCBot, Bytespider.