How To Identify True Search Competitors – SEO Competitive Analysis

At SMX Advanced back in June I attended a session where I heard about a really cool idea. The concept was that you could take a ton of keywords, and then map out what sites were showing up on those keywords to really get a good idea of who your real search competitors are.

The bad thing was that they said that they used their own developed software. I’m always bothered when I get excited about an idea, but then have no way of reproducing it. Well, I did some digging around and was able to find a way to do a similar type of report. Sure, it’s more labor intensive, but you can really learn some thing about who else is competing for your same keyword sets.

Below I’m going to outline the techniques I use to get the data, how to organize it, and what to do with it once you have the information. I admit that it may be a little choppy, so if you have any questions on it feel free to hit me up on Twitter: @dan_patterson

This technique was also mentioned by my friend Matt Siltala in his Pubcon presentation last week, which you can view here: Competitive Intelligence Pubcon Las Vegas.

Step 1 – Get a List of Keyword Sets

Since the goal of this whole exercise is to find other sites that are going after the same keyword sets as you, the first step in this process is to come up with a large list of keywords. A good place to start is to go through you analytics to find keywords that you’re already getting traffic from.

Once you have a list, break them up into topic sets. This way, you will be able to find which sites are full competitors or just partial competitors. Come up with a short name for all of these sets, and use this when you’re scraping results to identify which set that scrape belongs in.

Step 2 – Scraping the SERPs

There are plenty of tools out there that you can use to scrape the search results. Some of them use proxies and other tactics that the search engines aren’t fond of, so instead I’m going to go over two tools you can use that shouldn’t raise any of these problems.

The first is a handy little Firefox plugin called OutWit Hub. The second I’ll go over is the SEOmoz Pro Keyword Difficulty & SERP Analysis Tool for those of you that are already members of SEOmoz Pro.

Scraping SERPs with OutWit Hub

1- Download OutWit Hub

Go to http://www.outwit.com and download OutWit Hub (Free Firefox Plugin). Technically it’s a site content scraper tool, so we’re going to use it to scrape URLs from the SERPs.

OutWit Hub

2- Change Your Google Search Settings

In order to effectively use the plugin, you’re going to have to change a few of your Google search settings

  1. Turn off Google Instant by going to your account Search Settings, and then choosing “Do not use Google Instant”.
    Turn Off Google Instant
    Google Instand and Number of Results
  2. Decide how deep you want to look and set “Number of Results” to match. You can choose 10 (default), 20, 30, 50, or 100.
  3. Save your preferences and go back to Google Search

3- Scrape

Do a search for your first term. Once the SERPs come up, click on the OutWit Button in Firefox.

OutWit Button

This will give you the OutWit Hub Window. In the window, click on the ‘Guess’ option.

Guess Button

This will give you the info from the SERPs. We’re most interested in the “id” (rank) and “URL” columns. You can either export this info or just copy and paste it into Excel.

Ann Smarty did a post about OutWit a while back and also has a custom scraper you can use for Google results. The only problem I’ve found with this is that sometimes you’ll get URLs with spaces in them from breadcrumbs, which makes it a little harder to filter things down in Step 3. If you are in a niche that doesn’t have this problem, this can be a faster way to go.

4- Download Your Scrape and Clean It Up

One problem with OutWit Hub is that it can be inconsistent. Sometimes you get local listings in the export, sometimes you don’t. Sometimes you get paid listings in there. Somteims you don’t. So you have to watch what you’re scraping and make sure you’re actually getting the right info. They usually have a heading row, but you still have to do some filtering and cleanup work to get an accurate list. When you do this, make sure you also update the id (rank) column to reflect the real ranking you’re seeing.

You can either export the data to a CSV, or you can also just copy and paste it into Excel. I like the copy and paste option because if I see some paid ads at the top or bottom of the data, I can just not copy those rows.

5- Rinse and Repeat

This is unfortunately the labor intensive part of this whole process. You’ll have to repeat this process for all of the keywords you want to check. Again, there are other tools that do a little bit more brute force against Google, but OutWit Hub is a great FREE tool that will help you get the data you need if you’re willing to take the time.

No matter which method you use, make sure that you add a column at the beginning that includes your shortname for each set before the rank and URL of each scrape. This way you can identify which set the rankings and URLs belong to later.

Also, make sure you’re combining all of your data into one spreadsheet so we can do the comparison and filtering later. In Step 3 of this whole process I’ll show you what to do once you have all of your scraping done.

SEOmoz Keyword Difficulty & SERP Analysis Tool

SEOmoz Keyword Difficulty & SERP Analysis

In the long run, I think that using the SEOmoz tool is a lot easier and cleaner to use for this exercise. One nice thing about using the Keyword Difficulty & SERP Analysis Tool is that you can run up to 5 keywords at a time, and you don’t have the cleanup work that you have to do with OutWit. Once difference between the two is that with OutWit you can dig as deep as you want to set your Google Search settings. With SEOmoz you will get the top 25 and that’s it.

Here are the steps to getting the same data with the SEOmoz Keyword Difficulty & SERP Analysis Tool:

1- Run a Report (up to 5 at a time)

Sometimes I’ve found that the tool will time out if you run 4 or 5, so if you’re having that problem just run 3 and you’ll have an easier time.

2- CSV Export

Once the report loads, click on it and then choose the “Export to CSV” link down towards the bottom. It’s above the table with all the pretty greens and reds.

CSV Export - SEOmoz Tool

3- Rinse and Repeat

The only columns we need are ‘Rank’ and ‘URL’. If you want to start getting in to Domain and Page Authority comparisons you could use that data as well, but for this blog post I’m just going to keep it simple.

Just like with the OutWit Hub data, make sure you’re combining all of your CSV download into one master file so you can do the filtering you’ll need to do.

Step 3 – Filter Down to Just Domain Names

In order to really do the comparison, you need to filter your SERP scraping down to just the domain names. With a little Excel formula magic, this is easily done. Here are the basic steps to follow in Excel. Since there are so many different version of Excel and other spreadsheet programs, I’m just going to give you the basic steps and formulas here so you can do what you need to in the program/version you’re using.

Before you do the steps below, make sure to MAKE A COPY OF ALL YOUR SCRAPED DATA. We’re going to filter down to just the domain names you’ve scraped, but that’s only so we have a list of unique domains that we can then do some counting and averages on. You have to leave your original data so you can get the counts. So I repeat, make a copy of all your scraped data and do the steps below on the copy.

  1. Use ‘Text to Columns’ and delimit on ‘/’. This is probably the easiest way to break out the http: and any other folders in the URLs you’ve scraped. Delete all of the columns that don’t have just the domain name.
  2. Get rid of www. Since some of your scraping with have URLs with www and some won’t, we need to get rid of these. Sort your list of domain names alphabetically. Then, do another text to columns on the domains that have the www in them. You can do this the easiest by doing the ‘Fixed Width’ option since www. is always the same width. You may also have other subdomains in your list, but honestly I would just treat these as separate sites from the main.
  3. De-dupe. Now that you have your list of just domain names without the www and folders, de-dupe this list so that you have a list of unique domain names.

Step 4 – Count # of Results and Average Rank For All Unique Domains

Once again, for this part I’m just going to give you the steps rather than screenshots since it might vary a little bit from spreadsheet program to spreadsheet program.

To set up your spreadsheet matrix, you should have all of your unique domains down the left, and then across the top you’ll have a column for # Results and Avg Rank for each of your keyword set shortnames. Put these at the top of the two columns and merge over them if you want to make it a little prettier, and it will give you something to reference in your formulas.

Getting Number of Results Per Unique Domain

  1. This formula will vary a little bit based on how big your data set is and where your list of original domains is.
  2. For example, if your first unique domain is in cell B3, your full data set of URLs is from cell D120 to cell D293, the column in the data set with the keyword set short names is in cells A120 to A293, and your first short name column name is in cell C1 your formula would look like this: =COUNTIFS($D$120:$D$293,”*”&B3&”*”,$A$120:$A$293,$C$1)
  3. Notice the absolute references for the cell ranges. This is critical, otherwise you won’t get the correct count.
  4. The “*”&B3&”*” is a wild card that basically says match B3 with anything before or after it. So you’ll get www, non-www, home page, and any other page for that domain name.
  5. If your formula looks good, copy it down to all of your unique domains.
  6. Repeat this process or all of your keyword sets.

What this number tells you is the number of times that unique domain shows up in your scrapes for that keyword set. If they show up a lot, than that’s something they are going after, and can be higher if they have multiple listings as well. If they don’t show up very much, than it isn’t an important set for them.

Getting the Average Rank Per Unique Domain

  1. This formula will also vary a little bit based on how big your data set is, etc.
  2. Let’s use the same example cells as listed above, but your Rank data is in cells B120 to B293. Your formula in Excel would look like this: =AVERAGEIFS($B$120:$B$293,$D$120:$D$293,”*”&B2&”*”,$A$120:$A$293,$C$1)
  3. Again, notice the absolute references and make sure you have them in there.
  4. If your formula looks good, copy it down for all of your unique domains.
  5. Repeat this process or all of your keyword sets.

What this number tells you is the average rank for that domain name for that keyword set. Naturally, the lower the number the better they rank on average. The higher the number, the less of a threat they currently are, but it also shows that they are at least showing up for that set.

Step 5 – Organize and Analyze

Once you have all of your formulas down and you’re happy with what you see, I recommend copying and then pasting back the values for your matrix. This way you can sort the data any which way you want without it messing up the data (I made this mistake once and it wasn’t pretty).

Now you have a really cool matrix that will show you by keyword set which sites are going after different sets, and how important each set is to them based on how often they show up and what their average ranking is.

Have some fun sorting by different columns and even highlighting the numbers and sites that stand out to you. Here’s a screenshot sample of a matrix I did once to help.

Sample Data

What To Do With This Info

One of the problems with competitive analysis is that site owners and marketers only look at the companies they know, the major players in their space. Well, with this technique you will also see the affiliate sites that are competing that you may have overlooked, how big of a player sites like Wikipedia are in your space, etc.

If you run this every couple of months, you can also see the changes that are happening in the SERPs and better keep an eye on those sites that are becoming more of a threat.

As you identify new competitors, you also now have another site to analyze for marketing ideas, competitive links, etc.

Let Me Know What You Think

I really hope that this has been a helpful post for you to learn a technique to identify more of the true competitors in your space, and then what to do with that information. I’m sure that there are other ways to get this information, and if you have any additional tips please share them in the comments below.

Taylor Pratt on Competitive Analysis #seochat

On Thursday, November 18, 2010 Search Marketing Weekly will be hosting an #seochat on Twitter. Taylor Pratt (@RavenPratt) will be our guest answering questions about competitive analysis.

Details:

Starts at 7:00pm Mountain Time
About 1 hour long
Use hashtag #seochat
Host: Greg Shuey
Transcript
Questions Answered

  1. What are the most important backlink metrics to look at when doing a competitive analysis?
  2. When doing an on page analysis, where are some major areas you tend to look for problems?
  3. How often should you perform a competitive analysis?
  4. How does a competitive analysis help you set the overall strategy for an SEO campaign?
  5. How does a competitive analysis help you set the overall strategy for an SEO campaign?

About Taylor Pratt

Taylor Pratt is the Product Marketing Manager of Raven Internet Marketing Tools, a multi-user Web-based application for managing SEO and social media campaigns. Prior to joining Raven, Pratt was a Senior Search Specialist at nFusion where he specialized in search engine optimization (SEO), conversion and Web analytics, and promotes the use of conversion-focused, analytics-based SEO.

Pratt began his Internet marketing career as a search analyst at LunaMetrics, a Web analytics consulting firm. While working at LunaMetrics and nFusion, his client roster ranged from small businesses to Fortune 500 companies.

In addition to helping companies win online, Pratt contributes to Search Engine Journal and Search Engine People covering the latest tools and techniques in social media marketing.

You can catch him on Twitter at twitter.com/RavenPratt. Pratt is a frequent at both small business and search marketing conferences, including PubCon, SMX Advanced, SMX: East & West and PubCon South.

Competitive Analysis From Taylor Pratt

I love analyzing competitors who rank for any given keyword. Through a competitive analysis, you can get a ton of great information. When done right, it should give you a pretty good road map of how your competitors began ranking (and continue ranking) and what you can do too to get there yourself to the top three.

This past week, I came across two really good resources from Taylor Pratt from Raven Tools. One is a slide show presentation from SMX East and two, a competitive analysis checklist to help you analyze your competitors. You should definitely check both of them out. Here they are…

Download the Competitive Analysis  PDF Checklist here.

  • mind control movies erotic
  • black pornstar olivia
  • home amateur girls
  • Free hot sexy nude teens
  • young black and stacked
  • orgasm girl squirter videos
  • pinky pornstar videos
  • pussy hole closeup
  • Sex with hot aunties
  • many girls one boy sex
  • ass lap dance porn videos
  • porn picture of teens
  • big cock shemale sex picture galleries
  • mature classy pics
  • angel eyes tits indian black ebony
  • free young interracial sex
  • free female fingering orgasm video amateur
  • men sucking lactating woman breasts pictures
  • twistys teen video
  • Saltine the girl is hot
  • funny sexy pussy pps
  • Hot college pussy
  • anal ass mouth
  • cash for teen pussy
  • Constance marie nude pic
  • Burmese sexy girl
  • notorious movie sex scene
  • brooke skye facial video
  • japanese girl in vacuum bag
  • bakugan animated porn videos
  • my dick in your ass
  • free videos of girls playing porn
  • couples talk sex dvd
  • spanking sex gallerys
  • hairy coeds homemade wife sex video
  • free fuck movie video sleep fucking
  • emily horne and porn movies
  • girls bending over showing off ass
  • oral sex polls
  • mature sexy hung men
  • Girls like fuck
  • free porn vids tube
  • do girls like to say pussy
  • older lady showes pussy lips
  • sexy teen striping
  • quality porn vids
  • caramel pornstar ebony dailymotion
  • teacher anal porn picture archive
  • very very young naked girle
  • camel toe bikini sex
  • Hot lesbian teen
  • naked young teenie girls
  • kinky girls plays with boobs
  • teen ebony gymnast thumbnail gallery
  • anapola mushkadiz sex video
  • blonde pubic pics
  • free black butt videos
  • Super ass fuck
  • free first fuck teen porn
  • shemale tristin moore video
  • amateur sex ads
  • black xxxvideos raw gonzo
  • Indian boy nude
  • torrent kick ass movie
  • mother daughter threesome video
  • cute pics of girl hugging boy
  • inspirational video fuck it
  • closeup cumshots vids
  • free rough force sex movie
  • le petite rens clothes
  • free chinese porn pictures
  • free african interracial porn
  • free amatuer anal sex videos
  • lady ga ga naked pics
  • big girl has sex
  • mature american housewives pussy
  • movies about cheerleaders who like sex
  • stories of lesbian sex
  • porn mom fuck daughter watches
  • redtube teen hardcore
  • teen scene porn pics
  • pale blonde nude free pictures
  • amateur girls soles
  • wife blowjob and swallow vids
  • Blonde lesbians nude
  • Very little nude girl
  • young women forced sex videos
  • Nepali sex videos
  • dangerous beauty sex scenes
  • pictures of hot girls stripping
  • free italian porn gallery
  • the bt erotic movie downloads
  • make pictures naked
  • fuck anal fat girl
  • Sexy asian girl photos
  • hot babe shows off pussy
  • first time blowjob movies
  • anal butts videos
  • la petite famille preschool
  • tiffany fallon sex videos
  • mature women seduces young women
  • busty babes anal gangbang
  • Hairy teen pussy movie
  • Homemade teen movies
  • long ffm sex vids
  • pitch black movie sunglasses
  • Jesse jane sex movies
  • hand job cum on boobs pictures
  • Young cum covered teens
  • teen girl fucks cnfm teen boy
  • emilys bod naked pics
  • free teen video post upskirt
  • sexiest female nipples
  • hs musical girl naked
  • sexe hard mature
  • free wet masterbating pussy
  • naked couples sex videos
  • young japanise sex
  • anal orgasm versus vaginal
  • lesbian water hose in the ass
  • Black sex pay per view
  • free ebony granny pictures
  • amateur kissing flashing girls
  • free horny girl pics
  • Teen ass tit
  • Girl on girl orgies
  • Korean models naked
  • porn celeb videos
  • hentai suck dick
  • Beautiful pussy porn
  • Nude stripper videos
  • milf slavin sex movies
  • video hiphop chicks
  • make a realistic pussy
  • free naked young slut pictures
  • Nigeria girls porn
  • teenage boy nake naked legal
  • lichelle marie free porn videos
  • xxx black bitches hardcore porn
  • Big juicy boobs
  • amateur ffm video
  • sexy ebony bbw
  • Totally nude girls
  • Mofo sex videos
  • girls naked pissing
  • naked girls sexy shoes
  • Pics of hot college girls
  • free hardcore asian video porn
  • strong female orgasm videos
  • Student sex porn
  • death porn porn videos
  • pictures of america ferria tits
  • euro trash anal blonde
  • nude photograph of young girl
  • desi free sex video
  • free adult sex pictures twilight
  • girl with tight blouse video
  • sex robots on webcam
  • sexy chicks in bikinis
  • girls naked on trucks
  • big black teens
  • teen cybersex rooms
  • teen ebony movies
  • annuaire du sexe black
  • young girls nude fingering
  • extreme porn blacks
  • sexy ass girls with big tits
  • busty teen ellen videos
  • african girl fuck
  • pornstar envy vids
  • the black nudist girls
  • cadence cross free porn movies
  • real average looking wives porn
  • Pics of naked hunks
  • granny asian anal
  • Eating out girls porn
  • pictures of japanese girls
  • xvideos pussy closeup
  • Basshunter girl naked
  • Amy lindsay sex scenes
  • black berries ebony king
  • vhs movies for sale mature amature
  • free chubby blond porn videos
  • young girls getting gang bang
  • mexican babes free pics
  • petite nude latin girl
  • asian booty movies
  • younger teens girls
  • naughty japanese girls
  • amateur couple home sex tape
  • sexy babes jackin guys
  • sexy teens in skirts
  • old lady porn movie galleries
  • xxx erect nipples
  • college amateur coed pics
  • john c holmes pornstar pictures
  • explicit how to perform anal sex
  • painful first time anal video cries
  • gemma massey sex video
  • black anal gangbang
  • fuck movies kim kardashian
  • Actress lesbian kiss
  • interaical porn fuck
  • big tit teen sex movie
  • amateur teen tricked into fucking
  • Indian with big boobs
  • Hijab girl sex
  • up the butt porn
  • boobs lesbians videos
  • free teen sex seduction videos
  • cheerleader lesbian porn
  • hugh dick porn
  • cute teen lesbians gangbang
  • old people porn video
  • dallas cowboy cheerleader naked video
  • vancouver ex girlfriend nude porn
  • sweet cute young girls nude
  • coed bikini models pictures
  • Mass efect sex scene
  • white wife breed by black men
  • school girl pantyhose asian
  • interracial sex in nc
  • porn babes blow job
  • dirty sex full video
  • family naked on a boat video
  • twisted metal black blacks movie
  • nude young girl anime
  • teen girls and boy video
  • old young suck
  • girl geets a huge anal dick
  • mature ladies suck young boys
  • erotic public pictures
  • amature anal sex movies
  • swingers party miami
  • petite red blouse
  • free selena gomez bikini pics
  • asian man and black woman
  • young naked girls ilegel
  • college drunk pussy fucking
  • fre amatuer sex pics
  • big black shaved pussy
  • porn movies with real sisters
  • anime gallery japan magna
  • sex asian video legal
  • naked women masturbating videos
  • ebony girls big booty pictures
  • young teen lesbeins
  • big black butts videos
  • very sexy mature women
  • erotic photography gallery
  • free naked guy gallery
  • mario and peach naked with sex
  • high school pussy shots
  • ct group sex
  • wife fuck party
  • Hot teen claire
  • mother fuck movies
  • pantyhose lick pussy free movie
  • coed pussy videos
  • young girl cummed in video
  • beefy naked guys hardcore sex
  • tight ass jeans sex
  • sexy breasts porn
  • hot sexy blowjob slutload
  • Lesbian first kiss
  • Harry pussy pics
  • china sex porn video
  • Sexy girls pictures in pakistan
  • stacy london bikini pics
  • teen extreme anal
  • girls orgasm pictures
  • girl taking dick hard
  • pussy tube porn tube
  • asian girls take huge cock
  • sweet ass pictures
  • teenage girls room
  • sex comics videos
  • homemade family sex movies
  • shaved nude men pics
  • shemale black movie
  • mature female sexy biceps
  • shemale sol video
  • lesbian blonde milf mpeg
  • naked hunting girls with guns
  • hot blonde amature sex
  • young teen like it big
  • Big natural tit teens
  • meaty pussy compilation
  • Beauty hot girls
  • lesbian porn little girls
  • asian girl with perfect tits
  • grandpa fuck teen woman
  • sexy ink girls
  • best movie sex scenes free
  • amateur black teen gang bang
  • fuck that little redheads ass videos
  • hardcore partying images
  • amuertur ebony sex on couch
  • video viril young man jerking off
  • free pictures of ebony girls fucking
  • asian schoolgirls torrent
  • fuck black teen
  • define sexy girls
  • teens on webcam videos
  • davidson beauty supply hickory nc
  • porn star stacey cash photo pics
  • Hot tub sex scenes
  • sex swingers in boardman oregon
  • amateur candy girls
  • teens smelly pussy
  • free big natural tits lesbians
  • dick to big to fuck
  • Fine ass video
  • rusian fuck movies
  • karen kane petite clothing
  • long sex videos licking pussy
  • sex life wife
  • big ass chubby girls
  • Girls showing off their ass
  • asian girls on
  • laure phoenix pornstar free picture gallery
  • blonde anal fuck
  • Naked filipino models
  • sexo con teens virgenes
  • wife cries painful sex video
  • amateur s girl
  • petite women posing naked
  • old young sex stories
  • dwarf oral sex
  • best way to finger a pussy
  • sexy babe young gallery nude