RetilyCrawler

RetilyCrawler reads a website so that its pages can answer questions in a support agent built for that same website. If you found this page in your access log, the sections below tell you exactly what it did and how to stop it.

Every visit is requested by a person

RetilyCrawler does not discover sites, keep a list of sites to revisit, or run on a schedule. It only moves when a person asks it to read one website, in one of two places: someone types an address into retily.com/try to build a demo support agent for it, or a Retily customer scans their own website from the Retily app to add its pages to their knowledge base. Both follow the rules on this page. One request, one site, one visit, and then it stops.

So it is not a search crawler, not a training crawler, and not an autonomous agent browsing on its own initiative. It is a user-driven fetch, and usually the person driving it is the owner of the site being read, looking at their own content.

The pages it reads are used for one thing: answering questions about that business, in that business's own agent. They are not added to a search index, not sold, not shared between customers, and never used to train a machine-learning model.

How to identify it

User-Agent
RetilyCrawler/1.0 (+https://retily.com/bot)
IPv4
141.94.95.103
IPv6
2001:41d0:20a:900::1bbc
Operator
Retily (support@retily.com)

Both addresses are static and we publish any change here before it takes effect. The User-Agent string is stable; if we ever version it, the product token stays RetilyCrawler so a rule matching that keeps working. The one exception is a page rendered in a browser (see JavaScript below): that single request comes from our rendering provider's browser and addresses, not from the User-Agent and addresses above.

What it does, precisely

robots.txt
Fetched and parsed before any page, by both scanners. A site-wide disallow stops the crawl before a single page is requested; a path-level disallow skips that path. A page robots.txt disallows is never fetched, by us or through our rendering provider.
When refused
A 401, 403, 429 or 451 response, or a bot-challenge page, stops the visit: nothing more is fetched from that site in that scan. We do not retry through another client or address, and we do not use any anti-bot service.
Pages read
From retily.com/try: at most 15 indexed, from at most 25 requests, per visit. From a customer's website scan in the app: at most 30 pages per scan. Both start at the homepage and follow links found on the site's own pages; the app's scan also tries a few common addresses that may not be linked, such as /faq, /pricing, /contact and /shipping.
Concurrency
From retily.com/try: 5 requests at a time, with a 150ms gap per slot, and a 10s timeout per page. From the app: one request at a time, 1 second apart.
Repeat visits
From retily.com/try: one visit per domain per 24 hours, capped platform-wide at 50 crawls a day and 6 per requesting address. From the app: at most 3 scans an hour and 10 a day per customer workspace.
Method
GET only. It never submits a form, never logs in, and never follows a paywall.
JavaScript
RetilyCrawler does not run JavaScript. If a page loaded normally but builds its text in the browser (an empty app shell), we may have that one page rendered once in a headless browser by a third-party scraping provider, Scrapfly, from that provider's IP addresses. Never for a site that refused us, and never for a page robots.txt disallows.
Content
Readable page text and the site's own name, logo and colours, for the agent's appearance. No images beyond the logo, no files, no downloads.

Blocking it

We obey robots.txt and we check it first, so this is enough and takes effect on the very next request, for both scanners and for any page rendering:

User-agent: RetilyCrawler
Disallow: /

Blocking by User-Agent or by the addresses above works too. A refusal stops the visit and we do not route around it: no retry through another client or address, and no anti-bot service. If we ever behave otherwise, that is a bug. Please tell us at support@retily.com and we will fix it.

Allowing it

If you are trying to build an agent for your own site and it is not working, a bot protection product in front of your site is the usual reason, because it cannot tell that you asked for this visit. Allowing the User-Agent or the IP addresses above resolves it. On Cloudflare that is a WAF custom rule with a Skip action matching http.user_agent contains "RetilyCrawler".

You never have to. Every agent can also be built by pasting your content in directly, and a site we cannot read says so plainly rather than pretending otherwise.

Contact

Questions, complaints, or a request to stop: support@retily.com. We answer crawler mail from real site operators first.

Last updated: October 16, 2026