Capturing AI crawlers

AI bots (GPTBot, ClaudeBot, PerplexityBot…) fetch your HTML without running JavaScript. The w.js tag therefore cannot see them: counting their hits requires a small server-side module. This page explains how to install it.

#Why a server-side module

A classic measurement tag runs in the visitor's browser. A bot requests the page and leaves: no JavaScript is executed, no event is sent. The server module fills that gap. It looks at the user-agent of each request and, only when it recognises an AI bot, reports the page to Snorklee.

It never touches human traffic: your visitors are still measured by w.js alone, with no double counting.

#Install the module for your platform

Each file is ready to use, with no dependencies:

• WordPress — nothing to do, the module ships with the official plugin and is on by default.
• Cloudflare — cloudflare-worker.js, deployed as a Worker. Works in front of any site proxied by Cloudflare, including Shopify, Webflow or Wix.
• Next.js / Vercel — vercel-middleware.js, dropped in as middleware.js at the root of the project.
• Netlify — netlify-edge.js, dropped into netlify/edge-functions/ with path /*.
• Node / Express — node-express.js, middleware mounted before your routes.
• PHP — php-generic.php, included at the top of the page outside WordPress.

The module is non-blocking: it never waits for Snorklee's response, and an outage on our side neither slows down nor breaks your page.

#Set your domain

At the top of each file, a single value needs filling in: your domain. It must be identical to the data-site attribute of your w.js snippet.

A different value points to a site Snorklee does not know: reports are then refused and nothing shows up in your dashboard, with no visible error on your site. That is the first thing to check if the counter stays at zero.

#Check that bots are coming through

Open the AI traffic screen. Robot hits show up there per robot, per page and per day.

Don't worry if nothing shows up within the hour: bots come at their own pace, sometimes once a week on a small site. Give it several days before concluding something is wrong.

#Where it isn't possible

On Wix, Squarespace, Webflow or Framer with native hosting, no code can run server-side on the page request: crawler capture is impossible there. The dashboard says so plainly rather than showing a misleading zero.

The only workaround is to put the domain behind Cloudflare and use the Worker. Human visits coming from AI chats are still measured normally on all of these platforms.