Robots.txt Wildcards & Pattern Matching (RFC 9309)

AI Summary: Under RFC 9309, robots.txt supports the wildcard (*) representing any character sequence and the anchor ($) denoting the exact end of a URL path. When overlapping Allow and Disallow rules match a URL, the rule with the longest path pattern takes precedence.

The Modern Rules of Robots Pattern Matching

Historically, robots.txt pattern matching varied significantly across search engines. In September 2022, the Internet Engineering Task Force (IETF) published RFC 9309 (Robots Exclusion Protocol), formalizing wildcard syntax and rule precedence for all compliant crawlers.

Understanding RFC 9309 syntax prevents accidental site-wide de-indexing caused by improperly formatted wildcards.

Wildcard Syntax Essentials

1. The * Wildcard (Any Sequence of Characters)

Matches zero or more valid URL path characters:

# Disallow any URL containing "/temp/" anywhere in the path
User-agent: *
Disallow: /*/temp/*

# Disallow URLs with tracking query parameters
User-agent: *
Disallow: /*?*utm_*

2. The $ Anchor (End of URL Path)

Matches the exact terminus of the URL string:

# Block PDF files strictly, while allowing /docs/file.pdf.html
User-agent: *
Disallow: /*.pdf$

# Block URLs ending with a trailing slash
User-agent: *
Disallow: /*/$

The Law of Rule Precedence: Longest Match Wins

A common misconception is that Disallow always overrides Allow, or that the order of rules in the file dictates priority. Under RFC 9309:

When both an Allow and a Disallow rule match the same URL, the rule with the longest character pattern takes precedence.

Precedence Example

User-agent: Googlebot
Disallow: /catalog/
Allow: /catalog/public/

For the URL https://example.com/catalog/public/item-123:

• Pattern /catalog/ has length 9.
• Pattern /catalog/public/ has length 16.
• Result: The crawler is ALLOWED because 16 > 9.

If the pattern lengths are identical, the more permissive directive (Allow) takes precedence.

4 Costly Wildcard Mistakes to Avoid

1. Disallow: /*: Blocks the entire website completely.
2. Missing Leading Slash (Disallow: private/): Matches nothing; paths must begin with a forward slash /.
3. Case Sensitivity: RFC 9309 states paths are strictly case-sensitive. Disallow: /*.PDF$ will not match file.pdf.
4. Relying on noindex in robots.txt: The noindex directive in robots.txt is deprecated and ignored by Googlebot since 2019.

Verify that your robots.txt wildcards and path precedence work as intended without blocking valuable content. Audit with Geolify.ai.