RetilyCrawler

RetilyCrawler reads a website so that 15 or so of its pages can answer questions in a support agent built for that same website. If you found this page in your access log, the sections below tell you exactly what it did and how to stop it.

Every visit is requested by the site's own owner

RetilyCrawler does not discover sites, keep a frontier, or run on a schedule. It only ever moves when a person types a URL into retily.com/try and asks us to build a support agent for that address. One request, one site, one visit — then it stops.

So it is not a search crawler, not a training crawler, and not an autonomous agent browsing on its own initiative. It is a user-driven fetch, and in the overwhelming majority of cases the person driving it is the owner of the site being read, looking at their own content.

The pages it reads are used for one thing: answering questions about that business, in that business's own agent. They are not added to a search index, not sold, not shared between customers, and never used to train a machine-learning model.

How to identify it

User-Agent: RetilyCrawler/1.0 (+https://retily.com/bot)
IPv4: 141.94.95.103
IPv6: 2001:41d0:20a:900::1bbc
Operator: Retily — support@retily.com

Both addresses are static and we publish any change here before it takes effect. The User-Agent string is stable; if we ever version it, the product token stays RetilyCrawler so a rule matching that keeps working.

What it does, precisely

robots.txt: Fetched and parsed before any page. A site-wide disallow stops the crawl before a single page is requested; a path-level disallow skips that path.
Pages read: At most 15 indexed, from at most 25 requests, per visit. The homepage plus links found on it — no deep recursion.
Concurrency: 5 requests at a time, with a 150ms gap per slot, and a 10s timeout per page.
Repeat visits: One visit per domain per 24 hours, capped platform-wide at 50 crawls a day and 6 per requesting address.
Method: GET only. It never submits a form, never logs in, never follows a paywall, and never executes JavaScript.
Content: Readable page text and the site's own name, logo and colours, for the agent's appearance. No images beyond the logo, no files, no downloads.

Blocking it

We obey robots.txt and we check it first, so this is enough and takes effect on the very next request:

User-agent: RetilyCrawler
Disallow: /

Blocking by User-Agent or by the addresses above works too, and we will not route around either. If we ever behave otherwise, that is a bug — please tell us at support@retily.com and we will fix it.

Allowing it

If you are trying to build an agent for your own site and it is not working, a bot protection product in front of your site is the usual reason — it cannot tell that you asked for this visit. Allowing the User-Agent or the IP addresses above resolves it. On Cloudflare that is a WAF custom rule with a Skip action matching http.user_agent contains "RetilyCrawler".

You never have to. Every agent can also be built by pasting your content in directly, and a site we cannot read says so plainly rather than pretending otherwise.

Contact

Questions, complaints, or a request to stop: support@retily.com. We answer crawler mail from real site operators first.

Last updated: July 31, 2026