Yoast SEO Plugin Bug – was your website affected, and how can you recover?
In late May and early June, webmasters and SEO professionals starting noticing a drop in rankings and organic traffic following the version update of the extremely popular Yoast SEO WordPress plugin. The Yoast 7.0 update was technically launched in March, but obviously it would take time to notice a change in organic search traffic and rankings over time, assuming the plugin was updated immediately upon release, which is hardly ever the case.
We updated and/or installed a new version of the Yoast plugin on a small handful of client websites in late July, and noticed an alarming trend of organic traffic drops on the client websites that we updated the plugin on. The problems that the updated Yoast SEO plugin caused were so severe that Yoast’s founder issued a public apology.
According to BuiltWith, close to 8 million WordPress websites use the Yoast SEO plugin, so you could imagine not only how many websites were affected, but how many websites are still being affected yet the problem might still be undetected.
What kind of issues is the Yoast SEO plugin causing?
While it’s likely that Yoast 8.0 and later versions will not cause the same problems, the simple fact is that anyone who updated their Yoast SEO plugin to versions 7.0-7.2 has likely already been affected by the bug, with the majority of webmasters and website owners being unaware of the issues that were caused. It’s also very likely that many are unaware of how to recover from the drops in organic traffic and rankings caused by updating and/or installing that version of the plugin. Yoast announced that there was a patch that could be downloaded and installed in order to expedite the recovery process, but unfortunately there are additional steps that are necessary for a website to fully recover.
The Yoast SEO plugin 7.0 version added a new feature in which a new URL would be created for every media object on a website. Not only was this new feature introduced for no real apparent reason, but the Yoast developers also decided that it would be a good idea to enable the feature as the default setting, meaning that every time the plugin was installed or updated, it would automatically be set to create hundreds and thousands of bogus URL’s on the website, causing major SEO and indexing issues.
If the setting in the Yoast SEO plugin is set to “no”, than you likely have a big problem on your hands, because instead of keeping the actual URL of the media objects uploaded to your server, you now have a new page (with no content whatsoever) and a new URL created for every media object on your website. This is bad for SEO because these new pages are basically blank URL’s with no content, page titles, meta descriptions, etc. which causes a severe drop in organic search traffic and organic rankings, negatively affecting lead generation efforts on your site as a result.
For our client websites that were affected, it not only negatively affected their online marketing efforts, but it also put us in a tough position of having to take the time to diagnose the problem, explain to the clients that it was a result of the Yoast SEO plugin bug (which obviously most clients don’t stay updated on), put together a plan of action for recovery, and then take the time to manually fix the issue. On top of that, we were forced to set the expectations to the clients that it would take time for everything to improve, with it likely happening gradually over the coming weeks and/or months.
How do I figure out if my website was affected by the Yoast SEO plugin bug?
Another issue that we ran into was that the organic search traffic drop-offs that came as a result of the Yoast SEO plugin update occurred around the same time as the Google Medic update which was rolled out by Google in late July and early August, so identifying the Yoast SEO plugin as the culprit of severe drops in organic traffic and rankings took some time. We analyzed clients’ backlink profiles, server uptime (making sure the drop-off wasn’t caused by the site crashing for an extended period of time), crawl errors, new/lost backlinks, practically everything you could think of.
As we dove deeper into Google Webmaster Console data, we noticed a huge increase in valid and indexed pages in Google’s index around the same time as the drop off in rankings and organic traffic.
(to access the “valid pages” option in Google Webmaster Console, click on “coverage” on the left, and then click on the “valid” box as shown below)
Obviously this huge increase in indexed pages was a cause for concern.
Next, we downloaded all of the URL’s that Google specified as valid in their search engine index.
We created a Google spreadsheet of all URL’s that were indexed. In this case, as you can see, there were 1,146 pages that were indexed in Google’s search index, but not included in the sitemap. We manually went through this list and deleted any URL’s that were valid, and didn’t appear to redirect to an image.
Note: You obviously want to turn the Yoast feature to “yes”, but after you change this setting, the pages that were created as a result of the default setting in Yoast being set to “no” may now redirect to the original image files, but the URL’s will stay indexed in Google search index for weeks before being removed.
As I mentioned before, Yoast announced that there was a patch available to download and install which should expedite the process of these bogus URL’s being removed, but installing the patch and waiting for weeks and/or months for our paying clients’ organic search traffic and rankings to eventually recover simply wasn’t an option.
How can I recover from the Yoast SEO plugin bug penalty?
Once you’ve gone through the list of URL’s which you exported from Google Webmaster Console and eliminated all of the valid URL’s which should remain indexed in Google’s search index, the plan is to ask Google to remove the bogus image redirect URL’s from their index via their URL removal tool.
When we went through the list of close to 1,200 URL that we exported, we knew that it was quite possible for us to overlook one or two valid website pages that, if removed from Google’s index, would cause even more problems. So we doubled checked our final list of URL’s in a bulk URL checker.
There are a number of bulk URL checkers out there which will check a URL’s redirect status, but the one we chose to use was the Search Commander Bulk URL Checker which checks up to 50 URL’s at a time and shows you what URL it redirects to, along with the type of redirect. While putting 50 URL’s at a time might not seem like the most efficient option from a productivity standpoint, it was vital on our end to be sure we weren’t asking Google to remove valid website pages when we requested these bogus URL’s to be removed.
Once we submitted the final list of close to 900 bogus image URL’s to be removed to the bulk URL checker (50 at a time), we had our final list of URL’s that we wanted to ask Google to remove via the Google Webmaster Console URL removal tool.
However, the problem with the URL removal tool was that you could only submit one URL at a time, which would have taken us the entire week to complete.
Lucky for us, we found a great Google Chrome extension which allowed us to upload a .txt file of the URL’s we wanted to remove via bulk, versus one at a time. We downloaded the plugin, followed the installation instructions, copied all of the URL’s that we wanted to remove from the Google Sheet and pasted them into a .txt document.
Then, we went to the Google URL removal tool, clicked on “choose file”, uploaded our .txt file of URL’s we wanted to request to be removed, and watched the chrome extension go to work as it added each URL from the list one by one in the background.
In total, 857 URL’s were added to the URL removal request tool, and it took around 3-4 hours to upload the entire list.
We checked the URL removal tool two days later, and all of the client URL’s that we requested to be removed were marked as “removed”. We also, as an extra precaution, regenerated and resubmitted the client’s URL sitemap in Google Webmaster Console.
Moving forward, over the next few weeks, we will monitor the client’s organic search traffic in Google Analytics, organic search rankings, and indexed “valid” pages in Google Webmaster Console (which is not real-time and goes back around 10 days).
These were the steps that we took for ONE client who we noticed was likely penalized as a result of the Yoast SEO plugin update bug. As I mentioned before, we had a number of clients who were affected, and this took close to the entire day to identity, export the “valid” URL’s from Google Webmaster Console, manually go through a list of 1200 exported URL’s to eliminate valid pages, double check the final list of URL’s in the bulk URL checker, install the Chrome extension, upload the final list of URL’s in the .txt document and wait 3-4 hours for it to complete uploading. As I said, these extra steps are likely necessary to keep our paying clients happy, and are steps that we will continue to take for all of the client sites that were affected. It’s unfortunate that the Yoast plugin created all of this extra work and major issues with our client websites’ online search engine visibility and SEO efforts, but I believe that every step is necessary to ensure that the client websites fully recover from something that was completely out of their control.
Hopefully this post helps anyone out there who believes their organic search traffic and Google rankings have taken a major hit as a result of the Yoast SEO plugin bug penalty. If you’ve noticed a recent drop-off in organic search traffic or rankings, feel free to contact us to discuss how we can help identify what could be causing the problems on your website.