GeedoShopProductFinder: Robots.txt & Crawl Policy Reference

AI Summary: The active Geedo product-search crawler is documented as GeedoShopProductFinder. It indexes publicly accessible product pages for Geedo’s regional shopping search sites, states that it follows robots.txt, supports Crawl-delay, and publishes the IPv4 range 83.99.206.0/24. The registry label GeedoBot contains an older or mismatched User-Agent, so verify the complete observed header before applying controls.

Role and policy boundary

Geedo’s official crawler page describes GeedoShopProductFinder as an automated crawler for a global product-search engine. It indexes products from online stores and distributes results across regional Geedo sites. The documented current User-Agent is:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; GeedoShopProductFinder) Chrome/142.0.0.0 Safari/537.36

The registry entry for the geedobot slug instead records Mozilla/5.0 (compatible; GeedoBot; +http://www.geedo.com/bot.html). That is a material identity mismatch. Treat the current first-party page as the source for the published crawler name, but retain the registry label for historical lookup. Do not block a broad Geedo substring without confirming the actual request header.

Geedo states that the crawler focuses on publicly accessible product pages and follows robots.txt rules, rate limits, and restrictions defined by site owners. The documented purpose is product and shopping search indexing. Nothing in the reviewed source proves AI-training use, private-data access, or a right to bypass authentication.

To exclude the current published crawler from product indexing, use:

User-agent: GeedoShopProductFinder
Disallow: /

For selective access to a public product catalogue while protecting administrative and private paths:

User-agent: GeedoShopProductFinder
Allow: /products/
Allow: /category/
Disallow: /admin/
Disallow: /account/
Disallow: /private/
Disallow: /api/
Crawl-delay: 10

If logs show the older registry token instead, add a separate exact group only after confirming that it is still used. Robots.txt is advisory and cannot protect private or licensed content; use authentication, authorization, signed URLs, and origin controls for those boundaries.

Layered verification

Begin with raw access logs and preserve the complete User-Agent, source IP, ASN, reverse-DNS result, HTTP method, requested path, response status, response size, redirect chain, timestamp, and request rate. The current Geedo documentation publishes one IPv4 range, 83.99.206.0/24, and gives a forward-confirmed reverse DNS example for 83.99.206.111.

For an observed request, first perform a PTR lookup. The hostname should end with .geedo.com; the documented example resolves to product-search-83-99-206-111.geedo.com. Then perform a forward lookup and confirm that the hostname returns the original IP. A matching DNS result supports, but does not by itself prove, that the request came from Geedo. Use the full header and request behavior as additional evidence.

Compare behavior with the product-indexing purpose without overclaiming. Public product pages, category pages, prices, availability data, and ordinary product assets may be consistent with the stated service. High-concurrency traversal, private endpoint access, unexpected file downloads, repeated retries, or traffic outside 83.99.206.0/24 may indicate spoofing, a changed infrastructure, or a different client. These observations establish impact, not downstream use.

Evaluate /robots.txt independently. Confirm that the response is served from the correct host, has a successful status and text content type, contains an exact GeedoShopProductFinder group, and applies Crawl-delay and path rules as intended. Page-level directives can express discovery preferences:

<meta name="robots" content="noindex, nofollow">

X-Robots-Tag: noindex, nofollow

These signals do not secure private routes or authenticate the crawler. Enforce sensitive boundaries in the application and at the origin.

WAF and Nginx remediation examples

After confirming the current User-Agent and checking the source address, a narrow WAF rule can restrict the documented crawler. Replace the expression with your provider’s syntax, and prefer a report-only phase before enforcement:

{
  "description": "Restrict verified GeedoShopProductFinder traffic",
  "expression": "lower(http.user_agent) contains \"geedoshopproductfinder\"",
  "action": "block"
}

For Nginx, apply a route-scoped control and leave a record of the registry mismatch for operators:

map $http_user_agent $block_geedo_product_finder {
    default 0;
    ~*GeedoShopProductFinder 1;
}

server {
    location ~ ^/(admin|account|private|internal|api)/ {
        if ($block_geedo_product_finder) { return 403; }
        try_files $uri $uri/ =404;
    }
}

Do not treat the published 83.99.206.0/24 range as a permanent allowlist or blocklist without rechecking the official page. A User-Agent rule is easy to spoof or evade, and a network rule can become stale. Test product pages, feeds, sitemaps, checkout, accounts, APIs, and approved monitoring integrations. Pair edge matching with authentication, rate limits, signed assets, caching, and anomaly detection.

Review checklist

Search logs for both the current published token (GeedoShopProductFinder) and the registry token (GeedoBot). Preserve representative requests and record source IP, ASN, PTR result, forward lookup, path, method, response size, status, timing, and rate. Confirm whether the request source belongs to 83.99.206.0/24, and do not classify a request as authentic from its User-Agent alone.

Decide whether your objective is to preserve product-search visibility, limit catalogue extraction, protect private content, or reduce crawl load. Publish exact robots groups for the identity you have actually observed, use Crawl-delay where appropriate, and enforce private routes with WAF and application controls. Re-check the official policy whenever Geedo changes its User-Agent or IP documentation. Do not claim successful blocking from configuration alone; verify subsequent access logs.

References

1. GeedoShopProductFinder crawler documentation — official page documenting purpose, User-Agent, IP range, DNS verification, rate limiting, and robots controls, accessed 2026-08-24.
2. Geedo homepage — active product-search service observed during the same review.
3. Google Robots.txt Introduction — general explanation of crawler directives and their limitations.
4. RFC 9309 — Robots Exclusion Protocol standard; it does not authenticate a User-Agent.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.