FAST Enterprise Crawler: Robots.txt & Crawl Policy Reference

AI Summary: FAST Enterprise Crawler is a legacy inventory label associated with a historical web crawler, most notably used by the defunct Scirus science search engine. It historically identified as FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no. Because Scirus was retired in 2014 and FAST Search & Transfer was acquired, no active first-party policy exists. Treat any local observation as evidence to investigate and use layered controls.

Role and policy boundary

The registry describes FAST Enterprise Crawler as an enterprise-grade web crawler for search. Historically, this User-Agent was associated with Scirus, a scientific search engine powered by FAST Search & Transfer technology. Scirus was officially retired by Elsevier in early 2014, and FAST was acquired by Microsoft in 2008.

Because the original operators are defunct or have pivoted away from public web crawling under this name, this attribution cannot currently be verified with a primary source. A search for a current policy did not produce a usable first-party document.

No canonical source IP list, crawl schedule, rate limit, robots statement, or verification method was confirmed. Do not infer that an observed request is operated by a specific entity, that it is currently active, that it builds a public index, or that it collects AI-training data. The name in a registry is not an identity credential.

If your logs confirm an exact FAST Enterprise Crawler token and you want to communicate a restriction, publish:

User-agent: FAST Enterprise Crawler
Disallow: /

For selective access:

User-agent: FAST Enterprise Crawler
Allow: /public-reference/
Allow: /docs/
Disallow: /private/
Disallow: /internal/
Disallow: /api/

Robots.txt is advisory and cannot protect private or licensed content. Use authentication, authorization, signed URLs, and origin controls for those boundaries.

Layered verification

Start with raw access logs and preserve the complete User-Agent, source IP, ASN, reverse DNS, method, path, status, response size, redirects, timestamp, and request rate. Because the original operators associated with this name are defunct, any traffic using this User-Agent is highly likely to be spoofed or a legacy script.

Analyze behavior without assigning purpose prematurely. Requests for public pages, feeds, sitemaps, and metadata may resemble discovery; deep traversal, high concurrency, repeated retries, original-asset downloads, or private API access may indicate scraping or abuse. These patterns demonstrate operational impact but cannot prove the operator or downstream use.

Evaluate /robots.txt independently. Confirm the canonical host, response status, content type, exact user-agent group, and path match. A page-level directive may express a discoverability preference:

<meta name="robots" content="noindex, nofollow">

X-Robots-Tag: noindex, nofollow

These signals do not establish an opt-out for an undocumented client and do not secure private routes. Use authenticated delivery, signed URLs, and application authorization.

WAF and Nginx remediation examples

Once logs confirm an exact unwanted token (especially given the defunct status of the original operators), a narrow WAF rule can block the declared identity. Replace the example expression if your observed header differs:

{
  "description": "Block observed FAST Enterprise Crawler token",
  "expression": "lower(http.user_agent) contains \"fast enterprise crawler\"",
  "action": "block"
}

For Nginx, scope enforcement to private and high-cost routes while investigating public access:

map $http_user_agent $block_fast_enterprise_crawler {
    default 0;
    ~*FAST\ Enterprise\ Crawler 1;
}

server {
    location ~ ^/(private|internal|account|uploads|paywall|api)/ {
        if ($block_fast_enterprise_crawler) { return 403; }
        try_files $uri $uri/ =404;
    }
}

A User-Agent rule is easy to spoof or evade and may block a legitimate client using the substring. Do not create an IP allowlist or broad network block without current operator evidence. Test browsers, social previews, feed readers, search crawlers, approved monitors, and customer integrations. Pair edge matching with authentication, rate limits, signed assets, and anomaly detection.

Review checklist

Search logs for every exact header that may be associated with FAST Enterprise Crawler and preserve representative requests. Record paths, response sizes, statuses, source networks, timing, and rate. Re-check for canonical documentation before upgrading this profile; during this review no first-party policy was found due to the services being defunct.

Decide whether your objective is to preserve potential discovery, prevent extraction, protect private content, or reduce crawl load. Publish a targeted robots group only for the exact observed token, enforce sensitive routes with WAF and application controls, and test docs, media, feeds, sitemaps, uploads, and APIs separately.

References

1. Google Robots.txt Introduction — general explanation of crawler directives and their limitations.
2. RFC 9309 — Robots Exclusion Protocol standard; it does not authenticate a User-Agent.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.