Google does not need to index every URL on your website. Start by separating the important, distinct pages you want in search from redirects, duplicates, private pages and intentionally excluded URLs. For an important missing page, inspect the exact URL in Search Console, identify its current reported status, and compare Google's stored record with the page being served now.
Crawling, indexing and appearing for a particular query are different stages. A page can be reachable without being indexed, and indexed without ranking for the search you tried. A successful sitemap submission, HTTP 200 response or live URL test cannot guarantee all three.
Build the right list before calculating an indexing rate
Export the public pages you actually want indexed from the CMS or another reliable inventory. Record their intended canonical URLs and group them by purpose: articles, products, service pages or other useful sections. Compare this list with the sitemap and Search Console's Page indexing report. A URL Google does not know about may be absent from the report, so the report alone is not a complete inventory of your site.
Do not divide the number shown by a site: search by every URL a crawler can generate and call the result your indexing rate. Search operators are not an exhaustive index audit. Filter combinations, tracking parameters, internal search results and private account screens can inflate the denominator without representing distinct public pages that belong in search.
For a clearly labelled fictional inventory example, suppose a site has 120 intended public canonical pages and another 40 parameter aliases that repeat those pages. The owner inspects 20 of the 120 intended pages and finds 15 indexed at the observation time. That is 15/20, or 75%, of the inspected sample—not evidence that 75% of all 160 URLs, or of all 120 intended pages, is indexed. Keep uninspected URLs marked unknown; do not silently count them as failures.
Use the reported status to choose the next investigation
Open the Page indexing report and inspect representative URLs in the relevant reason group. Check the report's update date. Its example lists are limited, and a successful validation covers the issue being tracked rather than certifying that every page is indexed.
| Reported situation | What it establishes | Useful next check |
|---|---|---|
| Discovered – currently not indexed | Google knows the URL but has not crawled it yet in the reported record | Discovery paths, host availability, excess URL inventory and whether the page is worth fetching; the label alone does not identify one cause |
| Crawled – currently not indexed | The page was crawled but is not currently indexed; it may or may not be indexed later | The crawled content and date, useful distinct information, duplication and rendering; do not treat the label alone as a manual penalty or a definitive quality verdict |
| Excluded by noindex or blocked by robots.txt | An indexing instruction or crawl restriction is involved; these are different controls | Whether the control is intentional, and the actual HTML, headers or robots rule Google can access |
| Alternate or duplicate page | Another URL may represent the content | User-declared and Google-selected canonical URLs, actual similarity and conflicting signals |
| Page with redirect or not found (404) | The inspected URL redirects or was unavailable in that record | Whether that is the intended result, and whether any replacement or internal reference is correct |
| Server error or soft 404 | Google encountered a delivery failure or treated the delivered page as an error/empty page | Server logs, status, rendered main content and failed resources; HTTP 200 alone is insufficient |
Google's description of “Crawled – currently not indexed” explicitly says there is no need to resubmit that URL for crawling. Repeatedly pressing a submission button does not diagnose why a fetched page remains absent. Investigate and improve a real problem where you find one.
Compare the indexed record with a live test
In URL Inspection, record the last crawl date, fetch result, indexing permission and canonical information that are available. A stored record may describe a version from before your latest repair. Test the live URL to investigate its current accessibility and rendered content, and keep the two observations separate.
A live “URL is available to Google” result is not a promise of indexing. The live test does not predict Google's selected canonical and does not cover every possible indexing or search-display condition. Conversely, an indexed record does not guarantee that your page appears for every query, country or device.
For a fictional troubleshooting example, a store discovers that a deployment added noindex to otherwise public product pages. It removes the setting, checks the origin and public HTML plus response headers, and obtains a successful live test. If the stored crawl date is still from before the repair, the owner can document that the current fault is fixed while leaving later crawl and indexing outcomes pending. Reporting “all pages reindexed” at this point would be unsupported.
Fix delivery and indexing controls at their source
Check the final public URL with a real GET request and a browser. Look for unintended redirects, authentication, CDN challenges, unavailable resources and an empty or misleading error screen returned with status 200. Compare the source or database version with the origin response and the public response when caching is involved.
Inspect both HTML robots directives and X-Robots-Tag response headers. If noindex was accidental on a page intended for search, remove the unintended instruction at the layer generating it and verify the delivered result. Google's noindex guidance explains that Google must be able to fetch the page to see that instruction.
Robots.txt controls crawling; a blocked URL can still be indexed without its content. It is not a reliable way to remove an already known URL from search, and blocking a page can prevent Google from reading its noindex or canonical. Keep genuine private content behind appropriate access controls. For a recently changed file, use the robots.txt caching and refetch checklist to distinguish your public file from Google's cached copy.
For server failures or soft 404s, Google's crawling-error guide distinguishes a removed page, a moved page and a page whose content still exists but failed to render. A removed page without a suitable replacement should return a real 404 or 410. A genuine move can use a corresponding permanent redirect. If the page still exists, repair the missing main content or resources instead of redirecting everything to the homepage.
Check whether Google is indexing another version
An alternate URL being excluded can be the intended outcome. Compare its main content with Google's selected canonical, not just its title or URL. Tracking aliases may be duplicates; a complete language version, a later pagination page or a distinct product model needs its own assessment.
- Indicate the preferred URL version consistently through canonical tags, redirects, and sitemaps; the preferred-domain setting is no longer available in Search Console.
- Check that the preferred URL actually works, is indexable and serves the corresponding content.
- Inspect conflicting declarations from plugins, templates, headers and JavaScript rather than adding another tag blindly.
The canonical tag guide explains the HTML and HTTP-header implementations. A canonical is a preference signal, not a browser redirect or an order that forces Google's choice. Do not point every page to the homepage or use noindex as a substitute for selecting the preferred duplicate.
Make important pages discoverable without creating more noise
Link to an important page from relevant, reachable pages with a normal crawlable link and a descriptive anchor. Check categories, navigation and related articles where they genuinely help the reader. A small group that only links among its own members can remain disconnected from the rest of the site.
Compare a crawl with the CMS or sitemap inventory. A crawl cannot reliably enumerate pages it has no way to discover, and a result missing from a limited crawl is not automatically an orphan. Check the starting URL, allowed paths, rendering settings and crawl limits before drawing that conclusion.
Keep the sitemap aligned with intended public canonical URLs. Add new pages and update meaningful modification dates when their content changes. The sitemap update-frequency guide explains why repeatedly rebuilding unchanged XML or assigning every URL today's date does not set Google's crawl schedule.
If many known URLs remain uncrawled, inspect Crawl Stats, host availability and server logs for a pattern. Too many generated filter URLs, repeated errors or constrained serving capacity may deserve attention. Faster responses can help crawling, but there is no universal loading-time threshold that guarantees indexing. Do not infer Googlebot access merely from a successful request using a copied user-agent string.
Assess the page's useful contribution when technical access works
Compare representative unindexed pages with relevant indexed pages and with the actual task readers need to complete. Does a product page identify the model, specifications and availability? Does a guide supply working steps, prerequisites and important exceptions? Does a location page contain genuinely useful local information rather than only a city-name substitution?
Google's people-first content guidance emphasizes helpful, reliable and substantive information. There is no universal 300-, 800- or 1,500-word indexing requirement. Add the evidence and detail the question needs; padding, copied summaries or a changed date do not supply a missing answer.
Do not delete a useful page solely because it has not been indexed by an arbitrary deadline. Equally, do not preserve misleading or redundant pages simply to increase the URL count. Decide whether to improve, consolidate, redirect or remove based on the actual content and user need, then handle existing links and any genuine replacement carefully.
Check Search Console's Manual Actions and Security Issues reports if there is evidence of a broader problem. A specific notice or compromised content calls for its own investigation. An ordinary indexing-status label is not a reason to bulk-disavow links, invent a penalty diagnosis or promise recovery after a fixed “sandbox” period.
Request discovery appropriately, then verify what changed
For a small number of new or materially updated URLs, an eligible property owner or full user can request indexing through URL Inspection. For many URLs, maintain a suitable sitemap. Google's recrawl-request guidance says crawling can take from a few days to a few weeks, that repeated requests do not make it faster, and that inclusion is not guaranteed. This is guidance on expected processing, not a deadline for every page.
The Google Indexing API is limited to pages with JobPosting or BroadcastEvent embedded in VideoObject. It is not a general submission shortcut for ordinary articles, products or company pages. A plugin's successful API request is not evidence that an ineligible page is indexed or entitled to use that route.
Keep a short record for each affected group: intended URLs, reported reason and observation date, representative checks, exact repair and deployment time, live retest, and subsequent crawl/index status. Check remaining URLs before extrapolating from a sample. Track impressions, rankings and qualified enquiries separately after indexing; publication and indexing alone do not establish those outcomes.
If the investigation began after a rewrite, use the four checks for rankings falling after a content update to compare the release, the original query needs and concurrent search changes before choosing a rollback.