跳至正文
技术 SEO 与抓取索引 · Google SEO

How Long Does a Robots.txt Update Take? Cache, Refetch and Indexing

// / / 光算科技

Google generally caches robots.txt for up to 24 hours, but that is not a guaranteed deadline for a new rule—or a search-result change—to take effect. Google may keep the cached file longer when refreshing fails and may adjust its cache lifetime using Cache-Control max-age. A source-file edit, a public CDN response, Google's fetched copy and a page's index status are four separate things to verify.

First confirm that the public file has changed

Download the current file, save a rollback copy and edit it as UTF-8 plain text. Check the intended crawler groups and representative allowed and blocked URLs locally. Google's open-source robots.txt parser can be built for local tests; the Search Console report is not an upload-before-publishing editor.

Upload to the root of the correct host. Robots rules are scoped to protocol, host and port: a file on https://www.example.com/ does not automatically govern the apex host, another subdomain or HTTP. Refresh a relevant CDN cache if necessary, then fetch the public /robots.txt and compare its bytes or hash with the intended file. An origin-only check does not establish what Google receives.

Google's cache has its own refresh cycle

The robots specification describes a usual cache lifetime of up to 24 hours, with longer reuse possible after timeouts or server errors. It also says Google may increase or decrease the lifetime based on the max-age response header. A CDN purge does not directly clear Google's copy.

Google's Caching section stating up to 24 hours, longer reuse after refresh failures, and possible Cache-Control adjustments
Real excerpt from Google Search Central, How Google interprets the robots.txt specification, captured 21 October 2026. Google documentation text under CC BY 4.0, subject to site policies; cropped to the Caching section and converted to WebP.

For an urgent correction, Google's update instructions point to Request a recrawl in the robots.txt report. That requests a faster fetch; it is not a promised instant refresh. The report is available for Domain properties and host-level URL-prefix properties without a path, subject to account access.

Check the response status, especially during an outage

robots.txt responseDocumented Google behavior
2xxProcesses the file supplied by the server
3xxFollows at least five redirect hops; an unresolved chain is then treated as a missing robots.txt (404)
4xx other than 429Treats the file as absent and assumes no crawl restrictions; 401/403 are not crawl-rate controls
5xx, or a network/DNS failureInitially stops site crawling for 12 hours while retrying; then uses the last good version for the next 30 days while retrying. Without a cached version, assumes no restrictions in that subsequent stage
Failure still unresolved after 30 daysIf the site is generally available, behaves as if robots.txt is absent; if the site has general availability problems, stops site crawling. Keeps checking for a new file

These stages describe the robots fetch, not the status of an ordinary content URL. In particular, do not apply the “no restrictions” 4xx rule to 429, and do not interpret a redirect as an automatic failure if Google successfully reaches the file. Resolve delivery errors before trying to measure the effect of a rule.

Crawl permission and index removal are different

A blocked URL may still appear in search without its content being crawled. Robots.txt is therefore not a dependable removal method. For a public page that should stay available but leave search, return a supported noindex tag or response header and allow Google to fetch it.

For urgent hiding from Google Search, the Removals tool provides a temporary measure; implement the appropriate lasting change as well. Private information needs real access control rather than robots.txt. A robots cache refresh does not establish that a page has been recrawled, removed from the index or ranked differently.

Avoid file-format and testing mistakes

  • Encoding: use UTF-8 plain text. Google says it ignores the initial Unicode BOM; that is not a basis for claiming that the BOM necessarily drops the first rule. Invalid encoding or malformed lines are separate problems.
  • Size: Google ignores content after 500 KiB. Put essential controls within a valid, maintainable file rather than relying on truncated lines.
  • Unexpected whole-site block: test a matching Disallow: / deliberately before deploying. Test the actual crawler group, not just the wildcard group.
  • Unsupported fields: Google does not support crawl-delay or a noindex directive inside robots.txt.

The file-format rules are in the specification, and the creation guide places public testing after upload. Local parser checks, live availability checks and Search Console processing checks serve different purposes.

Close the loop with observable evidence

  1. Confirm the public response has the intended content and status.
  2. In the robots.txt report, inspect the fetched version, last fetch time and errors; request a recrawl when appropriate.
  3. Use URL Inspection for the affected page's current availability and separately its indexed information. Live success does not guarantee indexing.
  4. Review server logs for actual subsequent fetches; allowing a URL does not force an immediate request.

For page-level rules, see HTML versus X-Robots-Tag conflicts (Chinese); for an unread sitemap, use the Sitemaps report troubleshooting guide. Guangsuan's SEO service includes website technical work, with the particular investigation and delivery scope agreed for the project. 中文说明:robots缓存、重新获取与索引核验。

Portions of this page are reproduced from work created and shared by Google and used according to terms described in the Creative Commons 4.0 Attribution License.