llms.txt checker
Checks presence or format of llms.txt.
AI discovery audit
Evaluates crawler policy, llms.txt quality, sitemaps and structured discovery signals.
Why AI crawlability depends on more than whether a single text file exists.
Checks presence or format of llms.txt.
Evaluates crawler policy, llms.txt quality, sitemaps and structured discovery signals.
Can pass while crawler policy blocks important resources.
Separates policy accessibility from discovery readiness.
Usually a one-time check.
Baseline comparison can identify crawler-policy regressions.
Reports file-level findings.
Produces scores and recommendations for release or content teams.
A file checker is useful validation. A discovery audit is more appropriate when AI accessibility is an operational objective.
A direct website preflight for AI crawlability and LLM discovery: URL in, crawler-policy and llms.txt evidence out, with regression monitoring available when needed. Returns structured, machine-readable results for repeatable research, monitoring, automation, Apify Tasks, API.
See the Actor