How the public benchmark is produced

Frozen benchmark cohort

The public benchmark uses a frozen cohort with a defined universe and active Tier 1 set. Snapshot aggregates and histograms are derived from benchmark-eligible Tier 1 domains only, using a fixed snapshot data window.

• Cohort universe and active set are imported from a bounded Tranco-based run.
• Cohorts can be frozen before validation, pilot, and publication runs.
• Ordering and staged crawls follow Tranco rank ascending for repeatability.

HTML-only crawl model

Crawler analysis is HTML-only. It does not execute JavaScript and uses conservative network controls (timeouts, retries, rate limits, robots.txt basic disallow checks).

• Homepage probe establishes a seed host and strict host-locking for discovery.
• URL discovery and sampling are bounded and deterministic.
• Image checks store metadata only (HTTP status and dimensions), never image bytes.

Classification and scoring

URL checks are classified into Stable, Degraded, or Unreliable using conservative rules. Domain scores combine homepage and product reliability signals with proprietary weighting. The benchmark documents which signals are used, but not exact weights.

• Classification reasons are stored as a compact bitmask for aggregation.
• Product unreliability is weighted heavily in the overall domain score.
• No scoring changes are made by the public publication layer.

Benchmark eligibility and diagnostics

Domains must pass benchmark eligibility gates within the snapshot window to be included in public benchmark aggregates. Eligibility diagnostics are tracked to support calibration and crawl-quality validation.

• Minimum product URL checks and minimum total URL checks are enforced.
• Discovery failures and fetch failures are tracked separately for diagnostics.
• Validation and pilot stages are run before full benchmark crawls.

Strict separation from on-demand analysis

On-demand (for example Slack-triggered) domain analyses can produce a current domain score for lookup pages, but they are isolated from cohort-level benchmark statistics.

• On-demand analyses do not modify cohort tiers.
• On-demand analyses do not affect SnapshotAggregate or public histograms.
• Public benchmark artifacts use benchmark-eligible rows only.

Conservative publication policy

Public pages present aggregated statistics, distribution summaries, and platform-level aggregates only. They do not publish domain leaderboards, “worst domain” lists, or per-domain revenue estimates.