How To Identify True Search Competitors – SEO Competitive Analysis

At SMX Advanced back in June I attended a session where I heard about a really cool idea. The concept was that you could take a ton of keywords, and then map out what sites were showing up on those keywords to really get a good idea of who your real search competitors are.

The bad thing was that they said that they used their own developed software. I’m always bothered when I get excited about an idea, but then have no way of reproducing it. Well, I did some digging around and was able to find a way to do a similar type of report. Sure, it’s more labor intensive, but you can really learn some thing about who else is competing for your same keyword sets.

Below I’m going to outline the techniques I use to get the data, how to organize it, and what to do with it once you have the information. I admit that it may be a little choppy, so if you have any questions on it feel free to hit me up on Twitter: @dan_patterson

This technique was also mentioned by my friend Matt Siltala in his Pubcon presentation last week, which you can view here: Competitive Intelligence Pubcon Las Vegas.

Step 1 – Get a List of Keyword Sets

Since the goal of this whole exercise is to find other sites that are going after the same keyword sets as you, the first step in this process is to come up with a large list of keywords. A good place to start is to go through you analytics to find keywords that you’re already getting traffic from.

Once you have a list, break them up into topic sets. This way, you will be able to find which sites are full competitors or just partial competitors. Come up with a short name for all of these sets, and use this when you’re scraping results to identify which set that scrape belongs in.

Step 2 – Scraping the SERPs

There are plenty of tools out there that you can use to scrape the search results. Some of them use proxies and other tactics that the search engines aren’t fond of, so instead I’m going to go over two tools you can use that shouldn’t raise any of these problems.

The first is a handy little Firefox plugin called OutWit Hub. The second I’ll go over is the SEOmoz Pro Keyword Difficulty & SERP Analysis Tool for those of you that are already members of SEOmoz Pro.

Scraping SERPs with OutWit Hub

1- Download OutWit Hub

Go to http://www.outwit.com and download OutWit Hub (Free Firefox Plugin). Technically it’s a site content scraper tool, so we’re going to use it to scrape URLs from the SERPs.

OutWit Hub

2- Change Your Google Search Settings

In order to effectively use the plugin, you’re going to have to change a few of your Google search settings

  1. Turn off Google Instant by going to your account Search Settings, and then choosing “Do not use Google Instant”.
    Turn Off Google Instant
    Google Instand and Number of Results
  2. Decide how deep you want to look and set “Number of Results” to match. You can choose 10 (default), 20, 30, 50, or 100.
  3. Save your preferences and go back to Google Search

3- Scrape

Do a search for your first term. Once the SERPs come up, click on the OutWit Button in Firefox.

OutWit Button

This will give you the OutWit Hub Window. In the window, click on the ‘Guess’ option.

Guess Button

This will give you the info from the SERPs. We’re most interested in the “id” (rank) and “URL” columns. You can either export this info or just copy and paste it into Excel.

Ann Smarty did a post about OutWit a while back and also has a custom scraper you can use for Google results. The only problem I’ve found with this is that sometimes you’ll get URLs with spaces in them from breadcrumbs, which makes it a little harder to filter things down in Step 3. If you are in a niche that doesn’t have this problem, this can be a faster way to go.

4- Download Your Scrape and Clean It Up

One problem with OutWit Hub is that it can be inconsistent. Sometimes you get local listings in the export, sometimes you don’t. Sometimes you get paid listings in there. Somteims you don’t. So you have to watch what you’re scraping and make sure you’re actually getting the right info. They usually have a heading row, but you still have to do some filtering and cleanup work to get an accurate list. When you do this, make sure you also update the id (rank) column to reflect the real ranking you’re seeing.

You can either export the data to a CSV, or you can also just copy and paste it into Excel. I like the copy and paste option because if I see some paid ads at the top or bottom of the data, I can just not copy those rows.

5- Rinse and Repeat

This is unfortunately the labor intensive part of this whole process. You’ll have to repeat this process for all of the keywords you want to check. Again, there are other tools that do a little bit more brute force against Google, but OutWit Hub is a great FREE tool that will help you get the data you need if you’re willing to take the time.

No matter which method you use, make sure that you add a column at the beginning that includes your shortname for each set before the rank and URL of each scrape. This way you can identify which set the rankings and URLs belong to later.

Also, make sure you’re combining all of your data into one spreadsheet so we can do the comparison and filtering later. In Step 3 of this whole process I’ll show you what to do once you have all of your scraping done.

SEOmoz Keyword Difficulty & SERP Analysis Tool

SEOmoz Keyword Difficulty & SERP Analysis

In the long run, I think that using the SEOmoz tool is a lot easier and cleaner to use for this exercise. One nice thing about using the Keyword Difficulty & SERP Analysis Tool is that you can run up to 5 keywords at a time, and you don’t have the cleanup work that you have to do with OutWit. Once difference between the two is that with OutWit you can dig as deep as you want to set your Google Search settings. With SEOmoz you will get the top 25 and that’s it.

Here are the steps to getting the same data with the SEOmoz Keyword Difficulty & SERP Analysis Tool:

1- Run a Report (up to 5 at a time)

Sometimes I’ve found that the tool will time out if you run 4 or 5, so if you’re having that problem just run 3 and you’ll have an easier time.

2- CSV Export

Once the report loads, click on it and then choose the “Export to CSV” link down towards the bottom. It’s above the table with all the pretty greens and reds.

CSV Export - SEOmoz Tool

3- Rinse and Repeat

The only columns we need are ‘Rank’ and ‘URL’. If you want to start getting in to Domain and Page Authority comparisons you could use that data as well, but for this blog post I’m just going to keep it simple.

Just like with the OutWit Hub data, make sure you’re combining all of your CSV download into one master file so you can do the filtering you’ll need to do.

Step 3 – Filter Down to Just Domain Names

In order to really do the comparison, you need to filter your SERP scraping down to just the domain names. With a little Excel formula magic, this is easily done. Here are the basic steps to follow in Excel. Since there are so many different version of Excel and other spreadsheet programs, I’m just going to give you the basic steps and formulas here so you can do what you need to in the program/version you’re using.

Before you do the steps below, make sure to MAKE A COPY OF ALL YOUR SCRAPED DATA. We’re going to filter down to just the domain names you’ve scraped, but that’s only so we have a list of unique domains that we can then do some counting and averages on. You have to leave your original data so you can get the counts. So I repeat, make a copy of all your scraped data and do the steps below on the copy.

  1. Use ‘Text to Columns’ and delimit on ‘/’. This is probably the easiest way to break out the http: and any other folders in the URLs you’ve scraped. Delete all of the columns that don’t have just the domain name.
  2. Get rid of www. Since some of your scraping with have URLs with www and some won’t, we need to get rid of these. Sort your list of domain names alphabetically. Then, do another text to columns on the domains that have the www in them. You can do this the easiest by doing the ‘Fixed Width’ option since www. is always the same width. You may also have other subdomains in your list, but honestly I would just treat these as separate sites from the main.
  3. De-dupe. Now that you have your list of just domain names without the www and folders, de-dupe this list so that you have a list of unique domain names.

Step 4 – Count # of Results and Average Rank For All Unique Domains

Once again, for this part I’m just going to give you the steps rather than screenshots since it might vary a little bit from spreadsheet program to spreadsheet program.

To set up your spreadsheet matrix, you should have all of your unique domains down the left, and then across the top you’ll have a column for # Results and Avg Rank for each of your keyword set shortnames. Put these at the top of the two columns and merge over them if you want to make it a little prettier, and it will give you something to reference in your formulas.

Getting Number of Results Per Unique Domain

  1. This formula will vary a little bit based on how big your data set is and where your list of original domains is.
  2. For example, if your first unique domain is in cell B3, your full data set of URLs is from cell D120 to cell D293, the column in the data set with the keyword set short names is in cells A120 to A293, and your first short name column name is in cell C1 your formula would look like this: =COUNTIFS($D$120:$D$293,”*”&B3&”*”,$A$120:$A$293,$C$1)
  3. Notice the absolute references for the cell ranges. This is critical, otherwise you won’t get the correct count.
  4. The “*”&B3&”*” is a wild card that basically says match B3 with anything before or after it. So you’ll get www, non-www, home page, and any other page for that domain name.
  5. If your formula looks good, copy it down to all of your unique domains.
  6. Repeat this process or all of your keyword sets.

What this number tells you is the number of times that unique domain shows up in your scrapes for that keyword set. If they show up a lot, than that’s something they are going after, and can be higher if they have multiple listings as well. If they don’t show up very much, than it isn’t an important set for them.

Getting the Average Rank Per Unique Domain

  1. This formula will also vary a little bit based on how big your data set is, etc.
  2. Let’s use the same example cells as listed above, but your Rank data is in cells B120 to B293. Your formula in Excel would look like this: =AVERAGEIFS($B$120:$B$293,$D$120:$D$293,”*”&B2&”*”,$A$120:$A$293,$C$1)
  3. Again, notice the absolute references and make sure you have them in there.
  4. If your formula looks good, copy it down for all of your unique domains.
  5. Repeat this process or all of your keyword sets.

What this number tells you is the average rank for that domain name for that keyword set. Naturally, the lower the number the better they rank on average. The higher the number, the less of a threat they currently are, but it also shows that they are at least showing up for that set.

Step 5 – Organize and Analyze

Once you have all of your formulas down and you’re happy with what you see, I recommend copying and then pasting back the values for your matrix. This way you can sort the data any which way you want without it messing up the data (I made this mistake once and it wasn’t pretty).

Now you have a really cool matrix that will show you by keyword set which sites are going after different sets, and how important each set is to them based on how often they show up and what their average ranking is.

Have some fun sorting by different columns and even highlighting the numbers and sites that stand out to you. Here’s a screenshot sample of a matrix I did once to help.

Sample Data

What To Do With This Info

One of the problems with competitive analysis is that site owners and marketers only look at the companies they know, the major players in their space. Well, with this technique you will also see the affiliate sites that are competing that you may have overlooked, how big of a player sites like Wikipedia are in your space, etc.

If you run this every couple of months, you can also see the changes that are happening in the SERPs and better keep an eye on those sites that are becoming more of a threat.

As you identify new competitors, you also now have another site to analyze for marketing ideas, competitive links, etc.

Let Me Know What You Think

I really hope that this has been a helpful post for you to learn a technique to identify more of the true competitors in your space, and then what to do with that information. I’m sure that there are other ways to get this information, and if you have any additional tips please share them in the comments below.

Taylor Pratt on Competitive Analysis #seochat

On Thursday, November 18, 2010 Search Marketing Weekly will be hosting an #seochat on Twitter. Taylor Pratt (@RavenPratt) will be our guest answering questions about competitive analysis.

Details:

Starts at 7:00pm Mountain Time
About 1 hour long
Use hashtag #seochat
Host: Greg Shuey
Transcript
Questions Answered

  1. What are the most important backlink metrics to look at when doing a competitive analysis?
  2. When doing an on page analysis, where are some major areas you tend to look for problems?
  3. How often should you perform a competitive analysis?
  4. How does a competitive analysis help you set the overall strategy for an SEO campaign?
  5. How does a competitive analysis help you set the overall strategy for an SEO campaign?

About Taylor Pratt

Taylor Pratt is the Product Marketing Manager of Raven Internet Marketing Tools, a multi-user Web-based application for managing SEO and social media campaigns. Prior to joining Raven, Pratt was a Senior Search Specialist at nFusion where he specialized in search engine optimization (SEO), conversion and Web analytics, and promotes the use of conversion-focused, analytics-based SEO.

Pratt began his Internet marketing career as a search analyst at LunaMetrics, a Web analytics consulting firm. While working at LunaMetrics and nFusion, his client roster ranged from small businesses to Fortune 500 companies.

In addition to helping companies win online, Pratt contributes to Search Engine Journal and Search Engine People covering the latest tools and techniques in social media marketing.

You can catch him on Twitter at twitter.com/RavenPratt. Pratt is a frequent at both small business and search marketing conferences, including PubCon, SMX Advanced, SMX: East & West and PubCon South.

Competitive Analysis From Taylor Pratt

I love analyzing competitors who rank for any given keyword. Through a competitive analysis, you can get a ton of great information. When done right, it should give you a pretty good road map of how your competitors began ranking (and continue ranking) and what you can do too to get there yourself to the top three.

This past week, I came across two really good resources from Taylor Pratt from Raven Tools. One is a slide show presentation from SMX East and two, a competitive analysis checklist to help you analyze your competitors. You should definitely check both of them out. Here they are…

Download the Competitive Analysis  PDF Checklist here.

  • porn movies wacth online free
  • nude amateur home camcorder videos
  • young porn stars little caprice
  • free online tits videos
  • big boobs free online movies
  • video
  • buy japanese black pine
  • online black porn community
  • sexy couples holloween costumes
  • video
  • asian mailorder wife
  • sex virtual reality devices on sale
  • video
  • full length sex toy videos online
  • wife deepthroating online video
  • free online erotic videos
  • video
  • movies halloween sex costume
  • video
  • video
  • online amateur porn
  • monster porn movies and buy
  • watch free online movies softcore
  • teen sex videos online free
  • black girl online forums
  • teen girls pussy for sale
  • video
  • free ebony adult online movies wmv
  • longest free porn videos online
  • free online video cartoon porn
  • naked men pics online
  • pictures of girls costume
  • olsen sex video online
  • watch ebony porn free online
  • mature men circle jerk buy pics
  • free online porn movies large
  • old and young teen online video
  • college girls sexfor sale
  • video
  • video
  • border patrol find girls to fuck
  • japan sex movies online
  • black porn watch online
  • free hardcore sex videos online
  • free anal movies online
  • free online orgasmic porn videos
  • Girls having sex online
  • wives online free porn videos
  • sex buy horny women oral
  • girl fucked in holloween costume porn
  • video
  • little caprice videos anal
  • dreamgirls naked on campus
  • online porn megavideo
  • backstabbing bitch online porn video galleries
  • order a japanies wife
  • order sexy soccer babes calendar
  • porn site buy pictures
  • video
  • sexy costumes on young girlspics
  • young buys naked
  • video
  • video
  • video of online sex toys
  • ray j sex video free online
  • video
  • video
  • sex horror movies online
  • free online sexy fuck videos
  • video
  • share and watch porn movies online
  • free online homemade sex videos
  • video
  • free online amateur hardcore porn
  • couples seduce teens in costumes
  • topless pics of katie price jordon
  • video
  • video
  • ladyboy porn videos view online
  • erotic online videos for women
  • online live webcam girls
  • asian sex on megavideo online
  • picture of japan border
  • free black adult online movies wmv
  • kick ass video sales
  • free online hardcore sex illustrated stories
  • buy wholesale erotic comics porn magazines
  • punkrock girl naked online videos
  • anal sex prices in china
  • penis pussy costumes
  • free porn movies online sexy wife
  • online porn video on demand
  • video
  • natural breasts videos mgp
  • online asian porn
  • online black amateur porn community
  • cartoon porn free online videos
  • touch a girls boobs online
  • video
  • girl teen mobster costume
  • indian movie online naked
  • huge nipples farmgirls
  • video
  • video
  • costume party sex
  • anal vids mgp
  • young ethnic homebuyers
  • sexy virtual fuck buy
  • anima sex warrior the movie online
  • average couples having sex online
  • free porn videos of katie prices
  • great anal sex toys for cheap
  • petite long evening gowns for sale
  • video
  • free online young porn video
  • pictures of halloween costumes for couples
  • free porn online rincon girls
  • free public nudity videos online
  • online video sex live free
  • erotic free online videos
  • free online anal sex
  • online lesbian sex stories
  • lomg male orgasm video
  • la petite academy online payment
  • watch free online porn movie
  • amateur anal video for sale
  • asian porn watch online
  • sex movies for sale
  • cheap porn movies
  • online mature hollywood movies
  • video
  • halloween costumes whores naked pictures young
  • video
  • sex videos at no cost
  • costumes for teenage girls
  • free online hairy mature porno videos
  • movie black moon foresale
  • teens webcam online post
  • free amateur wives online posts
  • video
  • free online sex pictures
  • Asian teen mgp
  • sexy online girls
  • video
  • free erotic movies online
  • free sex anime videos online
  • amateur submitted sex video online
  • video
  • dance your ass off video buy
  • sex fuck buy women florida oral
  • video
  • video
  • college girls sexfor sale
  • free online hardcore pussy licking videos
  • free fuck team five videos online
  • video
  • video
  • free online screaming pain porn movies
  • video
  • video
  • lesbian sex porn online
  • video
  • ass parade free online videos
  • lesbian costume sex
  • anoushka hardcore videos online
  • video
  • Teen online video
  • video
  • time traveler wife free online movie
  • free online crying fat porn movies
  • free online porn videos trini
  • free online chicks having sex
  • costume porn pictures
  • free online teen amimal sex
  • amateur sex tapes online free
  • free online hardcore lesbian porn movies
  • free anal sex films online
  • video
  • free vintage porn videos online
  • see free xxx sex online free
  • video
  • juicyonline amateur vids
  • videosale blowjob and handjob
  • free online pictures of topless girls
  • online best blowjob by black
  • black homemade sex videos online
  • asian cheap webcam
  • rihanna online free porn videos
  • asian models in sexy costumes
  • video
  • anal sex prices
  • free online wildest blacks porn videos
  • online interracial porn stories
  • video
  • black online video sex sharing
  • free female orgasm video online
  • free malayalam sex movies online
  • online sex tv movies
  • video
  • video
  • video
  • online porn video sharing
  • video
  • free group public sex videos thumger
  • free online t teen pussy videos
  • online gystyle sex videos
  • watch full length erotic movies online
  • free online lesbians sex videos
  • video
  • average couples having sex online
  • video
  • video
  • free online holly madison porn videos
  • free porn movie watch online
  • petite womens polo shirts for sale
  • threesome online sex videos
  • video
  • streaming sex online porn amateur
  • katie price naked pics
  • free online hardcore anal sex
  • mature gaping anal pay buy join
  • free online hairy mature porno videos
  • free online anal stretching videos
  • video
  • softcore movies online
  • black porn free online
  • free online s with naked teens
  • world party tour girl mgp
  • free sex luve videos online
  • video
  • sex in cowgirl costume
  • fuck machine anal online
  • shemale anime online video free
  • first time lesbian free online porn
  • video
  • online porn videos
  • erotic literture and video online
  • video
  • watch erotic movies online for free
  • video
  • costume porn free pics
  • Sex porn video online
  • video
  • free trini porn videos online
  • video
  • sex videos for free online watching
  • watch the movie teen witch online
  • video
  • free online young litte sex videos
  • milla monroe online sex videos
  • free online streaming sex videos
  • kick ass hit girl costume
  • petite southwestern apparel online
  • online shemale lesbian porn videos
  • black porn watch online
  • video
  • free xxx porn movies wacth online
  • mallu softcore movies online
  • male on shemale free movie online
  • black sex movies online
  • online girlfriend naked
  • pictures of girls with eating disorders
  • teen online streams sex
  • buy wholesale erotic comics porn magazines
  • watch bondage video sex online free
  • girls gone wild movies online
  • video
  • video
  • free suicide girls online videos
  • video
  • online movies that dont suck
  • sexy peasant girl costumes
  • online little young sex
  • sex porn xxx group online free
  • free online femdom handjob videos
  • video
  • asian sex sale
  • sex scenes from young adam online
  • girls sucking dick at clubs online
  • hardcore sex free online
  • facial abuse free online videos
  • video girl ai online comics
  • cheap erotic dvd movies
  • video
  • video
  • couples sexy costumes
  • naked pictures for sale
  • free safe comic erotic sexy online
  • video
  • hollywood sex scenes movie online
  • free online hardcore teen porn
  • free online videos of porn stars
  • free colassic porn movies online
  • teen hot sexy online videos
  • couples having sex online live free
  • verry cheap teen girl clothes
  • lesbian mgp girls milfs
  • video
  • californias asian dreamgirls calendar
  • online dating sites adult young teens
  • free online blonde porn
  • free video little caprice cumshots
  • cost benefit analysis of young carers
  • video
  • video
  • petite clothing antique riding jacket sale
  • pornstar jamie lynn videos for sale
  • costa mesa hot chicks gallery
  • watch pirates movie porn online
  • best underground porn videos online
  • video
  • free online pink pussy
  • free online lesbian asians having sex
  • pics of my fuck buy
  • sexy babes in halloween costumes
  • costa mesa hot chicks gallery
  • max hardcore sale schoolgirls euro edition
  • black widow free online hentai video
  • online sex reality show playboy
  • video
  • free online young females videos
  • online video prurient porn
  • meet for free online sexy girls
  • sex and the city fuck buy
  • online fuck sex videos free
  • freeonline family porn movies
  • free online erotic massage video
  • free xxx hardcore videos online
  • girls and teen accessories wholesale
  • e beckinsale naked pics
  • online shopping for teenage girls
  • online photos sexy mature woman
  • casual sex western australia fuck buy
  • scary girl costume ideas for teens
  • Lesbian online videos
  • video
  • free online grandpa sex movies
  • video
  • video
  • video
  • best watching sex video online
  • free online adult xxx porn vidoes
  • Teen fuck online
  • video
  • free movies mgp sex fucking
  • stories short online romantic erotic sexy
  • harem girls video for sale
  • mature video sales
  • free online hardcore videos
  • imgsrc naked girl
  • sex boobs online
  • cheap sex videos for sale
  • buy asian movie dvds
  • sexy costumes beer girl xl
  • video
  • Watch sex and the city full movie online
  • japanese school girl uniform wholesale
  • free sex anime videos online
  • free online erotic massages pictures
  • free muslimgirls pussy
  • hardcore anime movies online
  • watch porno girl time online
  • ea out online free porn videos
  • video
  • girls gone wild online videos
  • free online shemale lesbian sex stories
  • free online prn movies for girls
  • video
  • hardcore brutal sex stories online
  • free online hardcore sex voluptuous women
  • teen online sex movies
  • watch porn videos for free online
  • tracy lords porn movies online
  • free online sexy fuck videos
  • video
  • free online mature sex tales
  • pussy world order
  • family sex movies online
  • young boy fuck buy
  • free henti porn online movies
  • video
  • online amateur spanking video
  • sexy beauty belle costume
  • video
  • free online group sex photos movies
  • border girl fire sexy dance
  • video
  • katie price pussy upskirt
  • naked pictures of mariska law order
  • video
  • free online porn galleries for teens
  • hardcore lesbian sex online
  • online full length porn movie
  • free online porn videos on
  • video
  • full length porn movies online free
  • kate beckinsale sexy bikini pics
  • watch amateur porn movies online
  • Watch hindi sex movies online
  • video