Publisher information

FeedFinder

FFBot discovers public feeds and checks that they remain available and current. It is operated by Sulzman Technology Group for rssfeeds.org.

How to recognize us

Our HTTP user agent is:

FFBot/1.0 (+https://rssfeeds.org/feedfinder/)

The product token for robots.txt rules is FFBot. This information page is public but unlisted.

What we request

Submitted and discovered URLs enter one page queue for inspection and automatic feed validation. A website can advertise a feed using a standard HTML link rel="alternate" element; we also inspect a bounded set of likely feed links and common feed paths. Uncertain duplicate matches can be held for review.

We retain compact URL identities and check outcomes to avoid repeatedly fetching the same non-feed pages or adding duplicate feeds. A configurable waiting period allows later rechecks without deleting the previous result. Explicit crawl exclusions still apply. Search-engine query definitions are supported, but no search provider is currently connected.

An administrator can start a bounded Spider Mode run on a selected website, with optional discovery of linked sites. Each run limits link depth, total URLs, requests, response bytes and elapsed time. Discovered pages and feed checks share those limits; redirects and robots requests also count. Previously checked URLs retain their identity and cooldown, and known feeds follow their own refresh schedule. We do not execute page JavaScript, enumerate query-parameter combinations, or start an unrestricted whole-web crawl.

We support RSS, RDF/RSS 1.0, Atom and JSON Feed. We retain feed metadata and collected entries with stable identities. Entry age, count, and disappearance from a publisher's latest feed do not expire previously collected data. Query parameters are part of the feed URL: we do not enumerate possible parameter combinations.

Page inspection records bounded descriptive metadata, declared license and terms links, and applicable crawl and indexing instructions. Pages linked from feed entries do not automatically enter a page crawl. Retaining page text for indexing requires separate storage and indexing permission decisions. Public display and AI summarization are separate choices; a successful fetch does not grant all of them. AI summaries are not currently generated.

Feed thumbnails

When a feed supplies image information, we record at most one selected thumbnail URL for the feed and one for each entry, together with declared dimensions, media type, credits, copyright and license details when available. We prefer explicitly declared thumbnails and images, with a bounded fallback to an image in the entry's supplied HTML. Inspected pages can also supply one representative image URL through their metadata. We do not fetch image files or visit license links to collect this metadata.

We automatically enable thumbnails when a feed supplies a recognized license permitting redistribution, including supported Creative Commons and public-domain declarations. Feed-level licenses are inherited by images unless a more specific item or image declaration overrides them. Images without recognized permission stay hidden; they do not enter a review queue. Each full feed refresh updates this decision automatically.

Permitted thumbnails are fetched into a bounded local cache and appear alongside feeds and their entries, with source, credit and license links where provided. Image requests follow our normal access, robots and rate limits; cache restrictions and permission changes can prevent or remove display. We keep one shared copy per image URL, limit image size and total cache storage, and expire unused copies. We do not generate resized files. Merely including an image or a copyright statement does not enable it. License declarations are publisher-supplied metadata, not independently verified ownership.

Where supplied, we also retain and display categories, keywords, authors, contributors, source attribution and content rights. We preserve these as publisher metadata; they are not inferred classifications.

Controlling requests

We check /robots.txt for each origin before fetching its resources, including redirect destinations. To prevent FFBot from crawling your site:

User-agent: FFBot
Disallow: /

To block only a section, replace / with that path. We support allow/disallow matching and the Crawl-delay extension. A long crawl delay that exceeds our current request window pauses the work instead of shortening your delay. A missing robots file (HTTP 404 or 410) allows requests; access failures, server errors and timeouts defer crawling conservatively.

We space requests to the same origin, honor applicable Retry-After responses, back off on failures and use ETag and Last-Modified validators when available. Administrators can set global rate limits and individual feed refresh intervals. Request time, redirects, response bytes and discovered candidates are bounded. We process all entries in an accepted feed response.

Our administrator also maintains explicit do-not-crawl decisions for URLs, feeds, paths, origins, hosts or domains. Discovery, queued work, refreshes and redirect requests check the current applicable decision. Known duplicates resolve to their established feed identity; an uncertain match can be held for review. We preserve meaningful differences in parameterized feed URLs.

Instructions on a resource

We inspect HTTP X-Robots-Tag headers and, for HTML pages, generic robots and FFBot meta tags. Supported directives include noindex, nofollow, nosnippet, noarchive and none. These apply to the resource carrying them; they do not grant rights over other resources.

For inspected pages, noindex prevents text indexing, nofollow stops link discovery, nosnippet restricts excerpts, and noarchive prevents retained page text. Storage and indexing also require explicit permission; a license declaration alone does not supply that decision.

X-Robots-Tag: FFBot: noindex, nosnippet

<meta name="FFBot" content="nofollow">

The bot must be able to fetch a resource to read its headers or meta tags. A robots exclusion prevents that fetch. For catalogued feeds, we record directive history. A later full response can update the current directives; a not-modified response can add restrictions without clearing previously observed ones.

We do not currently interpret llms.txt, ai.txt, AGENTS.md or arbitrary text as permission grants or executable instructions. None is a substitute for our supported crawl directives or an explicit publisher permission record.

Retention and publisher requests

Explicit publisher content expiration can exclude entries from current views and search inputs while their data remains retained for administration. We recognize direct feed or entry dcterms:valid end timestamps; feed expiration also applies to its entries. RSS cache TTL and JSON Feed's finished-feed flag do not expire articles. Operational logs and completed jobs use separate archive periods. We do not publish an article archive.

Rejected discovery candidates have a different lifecycle: copied descriptions and page content are discarded, while minimal identity, disposition and decision records remain to prevent repeated collection. A duplicate decision retains the reference to the accepted feed. Do-not-crawl decisions remain enforceable independently of job and log retention.

For corrections, removal, permission questions or a crawl problem, email privacy@sulzman.com with the affected URL. Please do not send access tokens or private feed credentials.

Last updated October 14, 2026.