Only a group that names us counts
A blanket User-agent: * disallow does not trigger the scanner refusal. Blanket disallows are often accidental, so we only treat a rule that names us as a refusal.
Disclosure
Our scanner and crawler
The scanner names itself with the User-Agent below. If you found it in an access log, this page explains what it fetches and how to stop it.
User-Agent
rebilder-ars/0.3 (+https://rebilder.com/bots)
We operate one fetcher. It measures how readable a page is for AI agents, under the published Agent Readability Spec 0.3.0. It does not index your content, train on it, or store your response bodies for publication.
Opting out
Name rebilder-ars in robots.txt.
One group, two lines, and we stop. We publish a token so you can refuse us without refusing assistants.
User-agent: rebilder-ars
Disallow: /A refusal produces no score
A scan of a domain that has named us produces no score and no letter grade, just unscored / robots-disallow-scanner. We do not fall back to a cached number, and we never publish a refusal as a negative.
Refusing assistants is a separate thing
A deliberate, well-formed disallow that names assistant crawlers is recorded as an opt-out, not a failure. It carries no letter grade and is never listed or ranked.
Blocking training crawlers costs nothing
Blocking model-training crawlers while allowing assistant fetches scores identically to a fully open site. Two published conformance fixtures prove it: the same page with and without a training-crawler disallow, both scoring 93. The only difference is an informational flag with a stated score effect of “none”.
Removal from anything we published
robots.txt stops future measurement. If a domain has an entry in the public index, removal is separate, free, self-serve, permanent, and never purchasable. An unverified removal request still triggers suppression pending review.
Refusing publication without blocking anything
robots.txt governs whether we measure. A file at /.well-known/rebilder-optout governs whether we publish: whether your domain may appear in any index, ranking or badge. They are separate on purpose, so refusing to be listed does not require blocking a fetcher you may otherwise want running.
# https://trymumm.com/developers/spec/optout
Version: 1
Index: denyThe format is published at the opt-out file specification, and rebilder.com serves its own file at the same path. A malformed file, a 5xx, or a timeout counts as a refusal: when in doubt, we do not publish.
One request per origin per day, before anything else, cached for 24 hours. It belongs to the index crawler and is not part of the scan budget below: a user-initiated scan produces a private result that is never published, so a publication refusal has nothing to govern there.
Request budget
What a scan costs you.
Two requests per URL, three per origin, cached. A cold single-URL scan is five requests; a warm origin is two.
| Request | When | Why |
|---|---|---|
| /robots.txt | First, always | Policy. Nothing else is fetched until this resolves. |
| the target URL | Agent probe | Accept prefers a machine representation. This is what a caller that wants facts receives. |
| the target URL | Browser control | Accept prefers HTML. Identical to the agent probe in every other respect: same User-Agent, no cookies. |
| the Markdown copy the page links to | Only when the page links to one and did not send one itself | One request, same site only, with the agent probe’s headers. Checks that the linked copy exists and matches the page. |
| /llms.txt | Per origin | Scored by D6.3. |
| /.well-known/ucp | Per origin | Scored by D6.5. |
| the target URL | Only on detected divergence, ≥30s later | Parity confirmation. Skipped on most scans. |
Rate limits we impose on ourselves
- At most one concurrent request per host, and a small global concurrency limit.
- At least one second between requests to the same host.
- A scan of one URL costs a small fixed number of requests: the page itself, twice (one HTML, one markdown), plus robots.txt, llms.txt and /.well-known/ucp, and the Markdown copy the page links to when it links to one. The public scanner caches the whole result for 24 hours per URL, so a repeated scan of the same URL costs nothing.
- A soft per-run cap with an explicit override, so a runaway loop stops on its own rather than on your rate limiter.
- The probe does not read Crawl-delay. Its fixed one-second interval is the guarantee, and Crawl-delay is never scored, so nothing in your robots.txt can hurt your grade.
- A 429 ends the scan for that URL rather than retrying it, and the page is left unscored. A persistent 5xx on robots.txt stops the scan; a single hiccup does not.
What we never fetch
- We do not guess at /sitemap.xml, /mcp, /acp, /openapi.json, or any other path. The three probed files above, and a Markdown copy the page itself links to, are the whole list.
- We never execute JavaScript, so we make no subresource requests at all: no images, no scripts, no stylesheets, no XHR.
- We send no cookies and store none.
- We never send a User-Agent belonging to Googlebot, ChatGPT, Claude, or any other operator. Every request identifies as ours.
The two policies
Two kinds of fetch, two rules.
Index crawling and one-off checks follow different rules. Both are published here.
Published or permanent artifacts
Full robots.txt compliance
Anything that results in a stored, published, or permanent artifact (an index entry, a badge, a row in an aggregate) obeys robots.txt, Crawl-delay, 429/Retry-After, and the published opt-out endpoint. This is the index crawler, and it is the strict policy with no carve-outs.
Session-private, user-initiated checks
The PageSpeed Insights precedent
A person pasting one URL into a tool and seeing the result themselves is a one-off, user-initiated fetch on that person’s behalf, in the same category as a browser developer tool or a page-speed checker. Those results are shown to the requesting session and are not published, indexed, or aggregated.
Named ARS 0.3.0 entries require verified opt-in. Aggregates combine at least 25 independent domains without identifying individual sites.
Naming rebilder-ars in robots.txt stops both, including user-initiated scans from our public vantage. The one exception: a site’s own operator can always scan their own domain.
Identity
How to verify a request is really us.
The User-Agent is stable
Every request carries rebilder-ars/0.3 (+https://rebilder.com/bots), and the token rebilder-ars is what robots.txt matches on. Matching is case-insensitive.
Confirm a request
To check a request, email security@rebilder.com with its URL, timestamp and User-Agent.
A User-Agent header identifies the claimed sender; it does not authenticate them.