What does FeedFetcher do?

FeedFetcher is the mergeflow component that downloads web feeds (like RSS, Atom or RDF) when users add them to their mergeflow.net repositories. FeedFetcher refreshes these feeds at regular intervals.

What is mergeflow?

mergeflow is a "smart RSS feed reader". Beyond what traditional RSS feed readers offer, mergeflow clusters news by topic, discovers emerging topics, and shows the development of news topics over time. All this is done completely automatically, without any manual intervention.

How often will my feeds be downloaded?

FeedFetcher will not download feeds more often than once an hour on average. We support HTTP compression and conditional GET to minimize the traffic.

Why is FeedFetcher trying to download content that isn't a web feed, or from incorrect links?

Sometimes our users try to add URLs that are not actually a web feed. FeedFetcher will retrieve the URL once to analyze the content, and will not schedule it for regular downloads if it is not a valid web feed.

Is FeedFetcher obeying my robots.txt file?

FeedFetcher is not a crawler. It downloads only feeds that our users have explicitly added to their repositories. It behaves like any other feed reader application. However, no matter how many users have added your feed, only one copy will be downloaded.

Please note that there is also another mergeflow component, PageReader, that retrieves content from your site. PageReader obeys robots.txt.

To learn more about mergeflow, please visit our website.