Google's Spiders are no Match for a Screaming Frog

Google’s Spiders are no Match for a Screaming Frog

Okay, so you have a website. You’ve identified the keywords that you want your website to rank for, and you or your SEO team have performed all of the SEO 101 basic tasks:

  • You’ve implemented your keywords into the page titles of your website
  • You’ve carefully worded the meta descriptions of your web pages so that they include some of your keywords, but you’re also able to convince the user to click on your web page in the search results because you didn’t write the meta description solely for search engines
  • You’ve implemented your keywords into your website’s content and H1/H2 headings
  • You’ve built interior links to your websites pages throughout the website
  • You’ve implemented a sitemap, added the website to Google Webmaster Tools, etc. etc.

You’re now likely operating under the assumption that your website is dully optimized from an SEO/on-page standpoint, which means that you can shift your focus to building links and authority to the website so that you can start dominating your competition.

But are you fully aware of all of the pages that are indexed on your website?

If you have some advanced knowledge, you know that you can do a “site:” search in Google and it will display all of the pages that are (publicly) indexed in their search results.

If you manage and monitor your website in Google Webmaster Tools, you know that you can see how many pages are indexed, whether or not there are any crawl errors, etc.

But if you’re performing SEO on your own and you completely rely on what Google tells you about your website, then you’re more naive than…well I’ll spare you the corny jokes, let’s just say that you’re far from a ninja.

SEOMoz has a paid service where you can add your website as a campaign, and it will crawl your website on a regular basis to tell you what pages are missing page titles or meta descriptions, which pages have duplicate on-page aspects, and other errors or warnings such as load errors or too many on-page links. But unless you’re performing SEO for multiple websites, it’s kind of hard to justify spending $99/month for this type of data.

That’s where the screaming frog steps in. (**Note- we are in no way affiliated with the makers of this tool, nor do we receive any sort of compensation for promoting it. We just think it’s a really cool free piece of screaming software**)

Screaming Frog SEO Spider is a free software tool that allows you to see every page that’s on your website, including commonly overlooked pages such as blog category pages, archive pages, tag pages, etc.

Not only does it show you every page of your website, but it also shows you various aspects of each page, such as:

  • Status (whether or not the page loads correctly, or if it’s been 301 redirected)
  • Page title (and page title length)
  • Meta description (and length)
  • Meta keywords (which aren’t very relevant unless you’re reading this in 2004)
  • H1 and H2 headings
  • Page-level canonical settings
  • Page size
  • Word Count
  • Level (from a folder standpoint)
  • Inlinks (how many links from other pages on the site)
  • Outlinks (how many links to other pages on the site)
  • External Outlinks (how many links to pages on other websites)

This is all valuable information for a number of reasons:

  1. If you have pages which do not have designated page titles or meta descriptions, then you should make sure you take the time to do so.
  2. If you have page titles or meta descriptions which are duplicate, then you should be making some changes. You can sort by meta description and page title in the tool which makes identifying possible duplicates easier.
  3. You want to make sure that your keywords are properly included in on-page aspects such as your page titles, meta descriptions, headings, and (if you are patient and can wait for the link juice to re-populate to a new page URL through a 301 redirect) URL.
  4. You can also use this tool to take a look at competitors websites and gather valuable information such as what pages they have included on their websites, internal link structure, etc. Basically, if you have a few websites that constantly outrank you for your keywords, then I don’t know about you, but I would like to know what they’re doing.
  5. Also, if you’re considering a website redesign, you’ll want to know every page on your website so that you can implement the necessary 301 redirects. This tool can help, and of course you can move everything over to an Excel spreadsheet (in the free version, simply select everything by dragging your mouse and copy it using command/control and c).

And if you use WordPress as your website’s content management system, there is a plugin called WordPress SEO by Yoast which can help you to identify how many times a designated keyword is implemented in that page’s title, meta description, heading, and URL.

The little green screaming frog may not necessarily catapult you to the top of Google, but it will help you to get a firmer grasp on what Google and other major search engines actually see when their spiders are crawling your website.