fiperbot: Robots.txt & Crawl Policy Reference

AI Summary: fiperbot is the web crawler and RSS feed fetcher for Fiper, a personalized news reader application. It fetches RSS feeds and article content so that users can read their favorite media within the Fiper app. Blocking this bot will prevent your site's content from being delivered to Fiper users who have subscribed to your feeds.

Role and policy boundary

The registry describes fiperbot as a web content indexing crawler. Headless-browser review of the canonical domain (fiper.net) confirms that Fiper is an RSS news reader application (similar to Feedly or Flipboard). The crawler identifies itself using the User-Agent string Mozilla/5.0 (compatible; fiperbot/0.1 +https://www.fiper.net/bot.html).

As a feed fetcher, its primary purpose is to retrieve RSS feeds and the associated article content requested by its users. Do not infer that its primary purpose is AI training or building a public search index. It acts as an intermediary delivering content to its app users.

If your logs confirm an exact fiperbot token and you want to prevent your feeds from being accessed by the Fiper app, publish:

User-agent: fiperbot
Disallow: /

For selective access (e.g., allowing it to fetch feeds but not crawl the rest of the site):

User-agent: fiperbot
Allow: /feed/
Allow: /rss/
Disallow: /

Robots.txt is advisory and cannot protect private or licensed content. Use authentication, authorization, signed URLs, and origin controls for those boundaries.

Layered verification

Start with raw access logs and preserve the complete User-Agent, source IP, ASN, reverse DNS, method, path, status, response size, redirects, timestamp, and request rate. Fiper does not currently publish a canonical list of IP addresses or a reverse DNS verification method on its main site.

Analyze behavior without assigning purpose prematurely. Requests primarily targeting RSS/Atom feeds (.xml, /feed/) and the specific articles linked within those feeds resemble authorized feed fetching; deep traversal of private areas, high concurrency across the entire site, or aggressive polling may indicate spoofing or misconfiguration. These patterns demonstrate operational impact but cannot prove the operator or downstream use.

Evaluate /robots.txt independently. Confirm the canonical host, response status, content type, exact user-agent group, and path match. A page-level directive may express a discoverability preference:

<meta name="robots" content="noindex, nofollow">

X-Robots-Tag: noindex, nofollow

These signals do not establish an opt-out for an undocumented client and do not secure private routes. Use authenticated delivery, signed URLs, and application authorization.

WAF and Nginx remediation examples

Once logs confirm an exact unwanted token (e.g., if you intentionally want to block Fiper from aggregating your content), a narrow WAF rule can block the declared identity. Replace the example expression if your observed header differs:

{
  "description": "Block observed fiperbot token",
  "expression": "lower(http.user_agent) contains \"fiperbot\"",
  "action": "block"
}

For Nginx, scope enforcement to private and high-cost routes while investigating public access:

map $http_user_agent $block_fiperbot {
    default 0;
    ~*fiperbot 1;
}

server {
    location ~ ^/(private|internal|account|uploads|paywall|api)/ {
        if ($block_fiperbot) { return 403; }
        try_files $uri $uri/ =404;
    }
}

A User-Agent rule is easy to spoof or evade and may block a legitimate client using the substring. Do not create an IP allowlist or broad network block without current operator evidence. Test browsers, social previews, feed readers, search crawlers, approved monitors, and customer integrations. Pair edge matching with authentication, rate limits, signed assets, and anomaly detection.

Review checklist

Search logs for every exact header that may be associated with fiperbot and preserve representative requests. Record paths, response sizes, statuses, source networks, timing, and rate. Monitor for aggressive crawling behavior, as the lack of official IP verification makes this token susceptible to spoofing.

Decide whether your objective is to preserve syndication visibility on RSS readers, prevent extraction, protect private content, or reduce crawl load. Publish a targeted robots group only for the exact observed token, enforce sensitive routes with WAF and application controls, and test docs, media, feeds, sitemaps, uploads, and APIs separately.

References

1. Fiper Official Site — verified as an active RSS reader app during the 2026-08-24 headless-browser review.
2. Google Robots.txt Introduction — general explanation of crawler directives and their limitations.
3. RFC 9309 — Robots Exclusion Protocol standard; it does not authenticate a User-Agent.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.