Our crawler

CouponFlint operates one automated crawler. It checks that the outbound coupon and store links published on this site still work, so offers that lead nowhere can be found and fixed. This page describes exactly what it does, so anyone seeing it in their logs can identify, verify, or block it.

User-Agent

Every request it makes carries this header, unchanged:

What it requests

Only URLs already published on this site: the destination of a coupon or a store's website link. It sends a HEAD request first and retries with GET only when a server rejects HEAD. It submits no forms, sends no cookies or credentials, does not log in, does not crawl beyond the single URL it was given, and reads only the HTTP status code — page content is discarded.

How often

Once a day, at 03:00 UTC. Each run checks a batch of up to 40 URLs, at most 5 at a time, with an 8-second timeout per request. Links are checked least-recently-checked first, so any single destination sees roughly one request per rotation of the catalogue rather than repeated hits.

It respects robots.txt and blocks

A 401, 403, 405, 429or a bot challenge is recorded as “blocked” and treated as a normal outcome, never as a broken link. Blocking this crawler will not cause your links to be removed from the site or flagged as dead.

Verifying a request came from us

Requests are signed using Web Bot Auth (HTTP Message Signatures, RFC 9421) with an Ed25519 key. Each request carries Signature, Signature-Input and Signature-Agent headers. The public key needed to verify them is published here:

A User-Agent string can be copied by anyone; a valid signature cannot. If you are filtering traffic, verify the signature rather than trusting the header.

Questions, or asking it to stop

Disallow the User-Agent above in your robots.txt, or email [email protected] and we will exclude your domain. See also our contact page.