skip to content
search.
Follow the system Light Dark

About the crawler

This site runs a small web crawler that indexes technical documentation, package registries and reference material for a search engine used mostly by software agents. If it visited you, this is what it did and how to make it stop.

How it identifies itself

Mozilla/5.0 (compatible; PkgFireBot/0.2; +https://search.pkgfire.org/bot)

Every request carries that user agent, with a link back to this page.

Verifying it

The crawler sends every request from these addresses and no others:

91.99.61.80

The same list as JSON, in the format the large search engines use for theirs: /bot/ips.json. A request with this user agent from any other address is not from this crawler. The list changes only when the crawler moves, and this page and the file change with it.

Or check it in DNS, the way the large search engines document for theirs: each address resolves back to crawler.search.pkgfire.org, and that name resolves forward to the same address. Both must match.

host 91.99.61.80
host crawler.search.pkgfire.org

Until 30 October 2026 the crawler was called RicciutiBot and answered to crawler.search.caioricciuti.com. Rules written for RicciutiBot in a robots.txt still apply to it until the end of January 2027.

How it behaves

  • Reads robots.txt first and follows it, including Crawl-delay.
  • One connection per host, at least 2 seconds between requests to the same host.
  • Conditional requests (ETag, If-Modified-Since): an unchanged page costs you nothing.
  • Recrawls slower the longer a page stays the same, up to once in 90 days.
  • Never renders JavaScript, never logs in, never fills forms, never fetches images, video or fonts.
  • Pages marked noindex or noarchive (meta tag or X-Robots-Tag) are never stored.
  • Package registries are read through their public APIs under their usage policies, not by crawling their pages.
  • Stores the extracted text and a short description of a page, with its address and when it was fetched. No personal data is extracted or kept.

Opting out

User-agent: PkgFireBot
Disallow: /

Takes effect within a day: the host is not fetched again. A noindex meta tag on a single page keeps that page out of the index at its next fetch.

To have pages removed sooner, or for anything else, write to me@caioricciuti.com (mail) with the addresses concerned.