跳至正文
数据、工具与监测 · Google SEO

How to Use Screaming Frog for SEO: Crawl, Trace and Verify Repairs

// / / 光算科技

Screaming Frog SEO Spider helps you crawl a website, find technical inconsistencies and trace them to the pages that need editing. The useful outcome is a verified repair list, not the largest possible issue count. A flagged URL may be intentional, outside the crawl scope or affected by the crawler's settings.

Prepare the scope and install the current official version

Download the current version from Screaming Frog’s official site. The free SEO Spider crawls up to 500 URLs; saving crawls, advanced configuration and JavaScript rendering require a licence. Check the installed version and licence before planning a repeatable audit. A free crawl that hits its limit is an incomplete sample, not a full inventory of a larger site.

Identify the intended domain, subdomains, protocols and path scope. Exclude irrelevant search or parameter combinations where appropriate, and choose a crawl rate that the authorized target can handle. Do not bypass access controls just to make the crawl larger.

Run a small crawl first

Use Spider mode to discover URLs from a starting address, or List mode to check known URLs. Enter the intended canonical starting URL, then inspect sample responses, redirects and discovered links before expanding. With the relevant licensed features, save the crawl and configuration for later comparison. With the free version, record the start URL, version, settings and exported results; do not assume you can reopen a saved crawl or render JavaScript.

If only the homepage and a few script files appear, investigate link discovery and rendering before concluding the site has no other pages. If the URL count expands unexpectedly, examine query strings, calendars, faceted navigation and subdomain settings.

A magnifying glass highlights a 404 page connected to its source page in a small link map
AI-assisted original illustration: trace an affected URL back to the page containing its link. This is a teaching map, not a real crawl export.

Read the main reports as questions to investigate

Report areaUseful questionPossible action
Response CodesWhich internally linked pages fail or redirect unnecessarily?Fix the link or the intended destination; keep legitimate removals
Page Titles and Meta DescriptionAre important pages missing accurate, distinct descriptions?Update the actual page fields or their template source
H1 and contentDoes the visible heading describe the page's main task?Resolve template mistakes or misleading page copy
Canonicals and directivesAre indexing and canonical choices intentional and consistent?Correct conflicts after deciding the desired URL
ImagesAre meaningful images accessible and appropriately described?Repair broken resources, context and useful alternative text
Internal linksCan readers reach important answers through relevant paths?Add or repair links where they serve the source page

A long title is not automatically a ranking penalty. A decorative image can correctly have empty alt text. A canonical pointing elsewhere may be intentional. Review the page's purpose before changing every row in a filtered report.

Trace a broken URL back to its source links

Select an affected URL and inspect its Inlinks to identify the pages and anchor text that point to it. Export the relevant records for editing. When there is a genuine replacement, update links to that destination and apply appropriate redirects where needed. If no replacement exists, remove or revise the obsolete reference rather than sending every 404 to the homepage.

Check the final response after each repair. A target returning 200 can still contain the wrong page, a login screen or an error message.

Understand what an orphan-page check requires

A crawl cannot discover an orphan by following links from its starting page if no path reaches that URL. Screaming Frog’s orphan-page tutorial notes that orphan pages can still link to one another. Compare the crawl with an appropriate sitemap, CMS export or authorized analytics/search inventory. Normalize redirects and alternate URL forms, and review deliberately excluded areas before deciding a page is orphaned.

“No contextual link in article bodies” is narrower than “no internal links anywhere.” Archives, navigation and pagination may still make a page reachable. Add editorial links according to relevance, using the internal-link review method, rather than indiscriminately linking every candidate.

When JavaScript rendering is necessary

The official JavaScript rendering tutorial explains the licensed rendering workflow: choose JavaScript under Configuration → Spider → Rendering; when original and rendered HTML are needed for comparison, enable their storage under Extraction before crawling. Select a URL and inspect the rendered page, HTML and discovered links. Start with representative templates because rendering consumes more resources.

If the capture is incomplete, review blocked resources and the crawler's timeout rather than treating the tool's default as Google's indexing limit. Confirm important findings using the rendering comparison workflow.

A repair record you can actually use

Teaching example: a product guide links to a retired specification. Record the source guide, its anchor, the broken destination and the intended replacement document. Check that the replacement covers the same model and revision before updating the source link. If it does not, revise the sentence or remove the obsolete reference instead of choosing a merely accessible URL.

After the edit, recrawl both the source and destination. Confirm the old link has gone, the new response is correct, and the document still supports the source sentence. A smaller error count alone would miss a link redirected to an unrelated page.

Prioritize and verify the work

For each confirmed issue, record the affected URL or template, evidence, intended behavior, proposed edit and retest result. Fix access or unintended indexing barriers on important pages before spending time shortening every URL or chasing an arbitrary health score.

After deployment, rerun the affected scope with the saved configuration and read the live pages. Successful crawling is a technical milestone; indexing, rankings, clicks and qualified business outcomes remain separate observations.