What is a 404 Error?
A 404 error is a page on your website that is no longer on your server: it has been removed or not properly redirected when you made a change in your content management system (CMS).
Regularly resolving your sites 404 errors is a signal to Google that you are maintaining a healthy website and keeping your visitors happy by preventing them from landing on a broken page as well as helping Google from unecessarily crawling a page.
Below are some tools and tricks to effectively find and resolve 404 error pages.
GOOGLE WEBMASTER TOOLS (FREE)
Google Webmaster Tools (a.k.a. Search Console) is a resource Google provides for web developers to monitor for any website issues and have some control over how their sites appear in Google’s search results. Assuming your GWT is verified, Login> Select your Domain> Select Crawl on the Left Sidebar > Crawl Errors> Select ‘404 Errors’> Select Download to export as Excel/CSV file.
SCREAMING FROG (FREE & PAID VERSION)
Screaming Frog’s SEO Spider is the BEST tool to crawl your website for 404 errors—period. Enter your website in the search box, get-up and grab a snack, and your crawl should be ready to export. You can crawl up to 500 URLs with the free version of Screaming Frog; the paid version allows you to crawl your entire website (500+ URLs).
We’ve highlighted (below) instructions on gathering and exporting your 404 error pages from Screaming Frog.
YOAST SEO WORDPRESS (Free and Paid Version)
If your entire site or blog is hosted on WordPress, Yoast SEO (formerly known as WordPress SEO by Yoast) is a must download to help configure your pages for SEO success. This plugin allows you to edit your URLs, meta data, XML Sitemaps, as well as monitor for 404 errors via integration with Google Webmaster Tools.
Action: If you have Yoast Premium ($69/year for one site), download the Yoast SEO WordPress Plugin on WordPress.org and add the file to your directory or visit Plugins > Add New > Search for ‘Yoast’ in the left side-bar of your WordPress home screen. CRI has provided instructions to find your website’s 404 errors using Yoast SEO.
FILTER, FILTER, FILTER
You will encounter a number of ugly strings from your exports that are not files or directories (e.g., % for searches, ?strings and fonts?family) in your content management system. Use filtering techniques in Excel to delete the ugly URL strings.
COMBINE & REMOVE DUPLICATES
Highlight the two columns (URL and Error Code)> Visit ‘Data’ on the top navigation> Select ‘Remove Duplicates’.
ARE YOUR 404 ERRORS REALLY 404 ERRORS?
Since Google Webmaster Tools gathers 404 errors from an extended period of time, you may encounter old 404 errors that have already been automatically resolved by Google or was a completed technical support ticket.
Action: After combining your Google Webmaster Tools and Screaming Frog 404 Errors (and removing duplicates), download our favorite tool, SEO TOOLS FOR EXCEL and install it as an Excel add-on.
Instructions for Validating 404s with SEO Tools for Excel
HAVE A LOT OF 404S? PRIORITIZE THEM
If you have a large site with hundreds of 404 errors, the best option is to prioritize your efforts using traffic from Google Analytics (or your preferred web analytics platform). Google’s algorithm will resolve the majority of your 404 errors over time; however, you want control over where Google will redirect a visitor.
Action: Gather all 404 Errors using the steps above> Pull ALL traffic data for a pre-determined period of time> Use the excel function, VLOOKUP, to identify how much organic traffic each 404 page has received.
HTACCESS FILE AND REWRITE RULES
To automate the removal of 404 errors we suggest making use of your .Htaccess file in your CMS. The initial setup is determining which page you would like to be the replacement for a 404 error page—translated into an excel document for your developers.
WARNING: An .htaccess file is an extremely sensitive file in your CMS. We highly recommend your IT department sign-off on any edits to an .htaccess file.
Mention redirect and .Htaccess file to your developers and they will understand what is required.
KEEP MONITORING FOR 404 ERRORS
Continue to monitor your 404 errors using the above tools—Yoast SEO for WordPress, Google Webmaster Tools, and Screaming Frog—and keep performing the recommended steps in this post to maintain a happy and healthy site.
sUMMARY
404 errors should be resolved on your website but keep in mind that 404 errors are not the END OF THE WORLD. There’s nothing wrong with developing a custom 404 error page that informs a visitor that “these things happen” online, but here is what we are doing to help you out! You might find that there is no other page on your site that is relevant to the 404 error page, in which you would redirect to the homepage as the default. Also, don’t waste your time needing to know or resolving soft 404 errors—they don’t matter.