Quick answer: Open the Crawl Errors report in Google Search Console, separate site-wide server problems from one-off not found URLs, fix the underlying cause (hosting, redirects, robots, templates), then mark errors as fixed only after Google can fetch the URL successfully. Do not obsess over every old 404 from a typo; prioritise errors on important URLs and patterns that affect many pages.
The Crawl Errors report (under Crawl in the Search Console interface most site owners still use day to day) lists URLs where Googlebot hit trouble. A parallel experience is rolling out in the newer Search Console, but the job is the same: find fetch failures, understand why, and repair what matters for indexing.
This how-to walks through a practical triage order so you do not waste a day on harmless noise.
What counts as a crawl error
Common buckets you will see:
- Server errors (5xx): Googlebot reached your host but got 500, 502, 503 or timed out.
- Not found (404): the URL does not exist (or your site returns 404).
- Access denied / robots: blocked by robots.txt or an HTTP 401/403.
- DNS / network issues: Googlebot could not resolve or connect (less common once a site is established).
- Soft 404: the URL returns 200 OK but looks like an empty or “not found” page to Google.
Not every line is an emergency. Scanners and old backlinks invent URLs constantly. Your job is pattern recognition.
For a broader process that includes crawl issues beside on-page and off-page checks, see our SEO audit overview and the technical SEO checklist.
Step 1: Confirm Search Console access and property type
Make sure you are looking at the correct property (domain or URL-prefix). If you recently moved to HTTPS, confirm the HTTPS property is verified and that you are not only cleaning the old HTTP property. After migrations, follow a proper SEO migration plan so redirects and Search Console both line up.
Step 2: Fix server errors first
Server errors on many URLs usually mean:
- Hosting overload or PHP crashes
- Plugin fatal errors
- Misconfigured reverse proxy or CDN
- Database connection failures
- Deployments that briefly broke the site
Actions:
- Note the time range of the spike in Crawl Errors and match it to hosting logs and deploys.
- Load a sample URL logged out; confirm whether the error is still happening.
- Check PHP error logs, plugin conflicts and resource limits.
- If you use aggressive caching or a firewall, ensure Googlebot is not rate-limited into failures.
- After stability returns, use Fetch as Google (or the newer URL inspection / fetch tools available to you) on key templates, then mark as fixed in Crawl Errors when appropriate.
A few 503s during a short maintenance window are normal. Days of 500s on product or article templates are not.
Step 3: Triage 404s by importance
Export or browse Not Found URLs and sort them mentally into:
A. Important URLs that should work
Deleted service pages, moved blog posts, old campaign landers still linked from ads or partners. Create 301 redirects to the best current equivalent. Do not 302 temporary redirects for permanent moves.
B. Soft mistakes on your own site
Broken internal links in navigation, footers, old content. Fix the link at the source so users and bots stop hitting the 404. Tools and browser extensions help, but a crawl of your own site is better.
C. Junk from the outside world
/wp-login.php probes on a static marketing site, invented WordPress paths, old phishing paths, typo domains pointing oddly. You do not need a redirect for every junk URL. A clean 404 page with search and key links is enough. Marking hundreds of irrelevant 404s as fixed without redirects achieves little.
D. Parameter noise
Facets, sort orders and tracking parameters can create endless URL variants. Fix with canonical tags, parameter handling in Search Console where relevant, and tighter internal linking, not endless one-off redirects.
Step 4: Robots and access denials
If Googlebot is blocked from CSS, JS or important sections, rendering and indexing suffer. Open your robots.txt and confirm you are not Disallowing assets the site needs. After HTTPS migration, check that you did not leave Disallow rules aimed at an old path structure.
Password-protecting a whole staging site is fine; password-protecting production by accident is not. Likewise, firewall rules that challenge every unknown bot can falsely flag Googlebot if misconfigured. Whitelist verified Googlebot ranges carefully (verify reverse DNS; do not trust User-Agent alone).
Step 5: Soft 404s
Soft 404s often appear when:
- Empty category or search result pages return 200 with “no results”
- Thin affiliate or filter pages show almost no content
- Custom 404 templates mistakenly send a 200 status
Return a real 404 or 410 for true missing content. For empty search pages, a 200 can be acceptable if the page is clearly a search UI with useful chrome, but thin doorway-like pages should be fixed or noindexed as appropriate to your strategy.
Step 6: Re-test and mark as fixed
In the Crawl Errors report you can mark examples as fixed after you believe Google will succeed. That is a signal to recrawl sooner; it is not a magic clean slate if the URL still fails. Use Fetch as Google on representative URLs:
- Homepage
- A key service or product template
- A blog post
- A URL you redirected
Confirm status codes with curl or browser dev tools as well. Search Console is not the only source of truth.
How this ties to indexing and rankings
Crawl errors block or delay discovery of pages. They do not automatically “penalise” a site in the cartoon sense, but Google cannot rank what it cannot fetch. Chronic server errors also hurt users. Fixing crawl health supports Google Search Console monitoring alongside Search Analytics.
If traffic dropped and crawl errors spiked together, treat hosting stability as incident response, then repair redirects and internal links. See also website traffic drop for a wider diagnostic map.
A weekly habit (keeps this manageable)
- Skim Crawl Errors for new server error spikes.
- Sample ten fresh 404s: redirect, fix internal link, or ignore junk.
- Confirm robots.txt has not changed unexpectedly.
- After major plugin or theme updates, fetch key URLs once.
Fifteen minutes a week beats a quarterly panic.
FAQ
Should every 404 become a 301?
No. Only redirect when there is a clear replacement users need. Redirecting everything to the homepage creates soft-404 style confusion and poor UX.
Do crawl errors mean I was hacked?
Not by themselves. Hacked sites can generate weird URLs and server errors, but most crawl error lists are boring broken links and bot noise. If you see injected spam URLs, investigate WordPress security separately.
Is the old Crawl Errors report going away?
Google has been evolving Search Console reports. Use whichever crawl/coverage tools your property shows today, but keep the same triage logic.
How fast will fixes show?
After fetches and recrawls, days to weeks depending on site size and importance. Marking as fixed can encourage sooner recrawl; it does not guarantee same-day clearance.
Next step
Stable crawling is table stakes for organic search. If you want a structured pass over Crawl Errors, indexing and on-page issues together, our get found work starts with the technical picture before content or links.
