Crawler
KeelSignals, and how to stop it
If you reached this page from a line in your access log, one request identified as KeelSignals fetched your homepage. This page says what that was, what it asked for, what we kept, and how to make it stop. The exact string it sends is:
KeelSignals/0.1 (+https://keelgrc.com/crawler)
What KeelSignals is
It is the lookup behind Keel’s free vendor discovery tool. Somebody enters a domain, and it reads what that domain already declares in public — verification tokens in DNS, the email senders the domain authorizes, where its mail is routed, what its names point at, and the content-security policy the site publishes for itself — and turns that into a first draft of a third-party register. The same lookup runs inside Keel when a customer starts one for their own domain while setting up a workspace.
It is not a search-engine crawler, it is not an index, and it is not a security scan. It sends no port scan, no directory or path guessing, no admin-panel probing, no credential testing and no payload of any kind.
It only runs when somebody asks it to
There is no schedule, no queue of domains, and no crawl frontier. Nothing runs unless a person types a domain and submits it, and the run ends when that request is answered. It never follows a link, so it cannot arrive at your site from someone else’s, and it never walks from one page of yours to another.
Whoever ran it had to affirm that they are authorized to assess that domain; without that affirmation the request is refused outright. We cannot verify the claim, which is exactly why what follows is limited to things any visitor to your homepage could already see.
What one run does, at most
-
DNS-over-HTTPS queries to a public resolver —
cloudflare-dns.com— about public records for the domain: TXT records, the SPF record and the senders it includes, MX, DMARC reporting addresses, CNAME targets, nameservers and CAA. These are questions anyone can ask, and they go to a resolver rather than to you. -
One
GETof the homepage.https://your-domain/, and if that does not answer,https://www.your-domain/— the same page under the other host. That is the only request that reaches your web server, and there is no second one. -
Nothing else. No path walking, no filename or directory guessing, no
/.well-known/sweep, no second page, and no request your own visitors would not make.
The homepage request is a plain GET asking for HTML. Its response headers
are read for a content-security policy; only if there is none in a header do we look at
the page itself, and then only the first 64 KB of it, for a
<meta> policy. What that policy names is the point: a site’s own
script-src and connect-src list is a set of third parties the
company chose to trust, written by the company itself.
How often it can happen
Two limits are enforced at the public entry point, before anything is fetched:
- 10 requests per 60 seconds from one calling IP address.
- 5 requests per 60 seconds against any one target domain, across every caller. This is the limit that bounds how hard Keel can be made to hit your server, no matter how many people ask.
What we keep
From the free tool, nothing. No account, no record of the domain, no stored result and no copy of your page: the response is read in memory, the hostnames are extracted, and it is discarded. A run started by a customer inside a Keel workspace is different: it records that run and the candidate vendors it produced, in that customer’s own workspace.
Neither path stores your HTML or a screenshot of it, and neither collects anything about individual people. Where a record contains an address — a DMARC reporting address, for instance — only its domain is kept and the mailbox is dropped.
How to identify or block it
Match KeelSignals in the User-Agent header. It is the first
token of the full string at the top of this page, and it is the same on every request.
A rule at your CDN, WAF or web server that blocks it takes effect immediately and does
not depend on us doing anything.
A robots.txt rule will not stop it today, and we would
rather tell you that than let you write one that does nothing. The homepage fetch is a
single request and the engine does not read robots.txt before making it.
Block on the User-Agent instead.
There is no published IP range to match against. Keel runs on Cloudflare, so the address the request arrives from is shared with unrelated traffic and is not a reliable way to identify it.
Contact
If you would rather your domain was not looked up at all, or you have a question about a
request you saw, email [email protected] and include the domain. If you
are reporting a security issue, our
security page is the faster route.
Keel GRC LLC publishes this page because the address is in the request. More about the product is on the about page, and the tool this serves is at vendor discovery.