How To Identify True Search Competitors – SEO Competitive Analysis

At SMX Advanced back in June I attended a session where I heard about a really cool idea. The concept was that you could take a ton of keywords, and then map out what sites were showing up on those keywords to really get a good idea of who your real search competitors are.

The bad thing was that they said that they used their own developed software. I’m always bothered when I get excited about an idea, but then have no way of reproducing it. Well, I did some digging around and was able to find a way to do a similar type of report. Sure, it’s more labor intensive, but you can really learn some thing about who else is competing for your same keyword sets.

Below I’m going to outline the techniques I use to get the data, how to organize it, and what to do with it once you have the information. I admit that it may be a little choppy, so if you have any questions on it feel free to hit me up on Twitter: @dan_patterson

This technique was also mentioned by my friend Matt Siltala in his Pubcon presentation last week, which you can view here: Competitive Intelligence Pubcon Las Vegas.

Step 1 – Get a List of Keyword Sets

Since the goal of this whole exercise is to find other sites that are going after the same keyword sets as you, the first step in this process is to come up with a large list of keywords. A good place to start is to go through you analytics to find keywords that you’re already getting traffic from.

Once you have a list, break them up into topic sets. This way, you will be able to find which sites are full competitors or just partial competitors. Come up with a short name for all of these sets, and use this when you’re scraping results to identify which set that scrape belongs in.

Step 2 – Scraping the SERPs

There are plenty of tools out there that you can use to scrape the search results. Some of them use proxies and other tactics that the search engines aren’t fond of, so instead I’m going to go over two tools you can use that shouldn’t raise any of these problems.

The first is a handy little Firefox plugin called OutWit Hub. The second I’ll go over is the SEOmoz Pro Keyword Difficulty & SERP Analysis Tool for those of you that are already members of SEOmoz Pro.

Scraping SERPs with OutWit Hub

1- Download OutWit Hub

Go to http://www.outwit.com and download OutWit Hub (Free Firefox Plugin). Technically it’s a site content scraper tool, so we’re going to use it to scrape URLs from the SERPs.

OutWit Hub

2- Change Your Google Search Settings

In order to effectively use the plugin, you’re going to have to change a few of your Google search settings

  1. Turn off Google Instant by going to your account Search Settings, and then choosing “Do not use Google Instant”.
    Turn Off Google Instant
    Google Instand and Number of Results
  2. Decide how deep you want to look and set “Number of Results” to match. You can choose 10 (default), 20, 30, 50, or 100.
  3. Save your preferences and go back to Google Search

3- Scrape

Do a search for your first term. Once the SERPs come up, click on the OutWit Button in Firefox.

OutWit Button

This will give you the OutWit Hub Window. In the window, click on the ‘Guess’ option.

Guess Button

This will give you the info from the SERPs. We’re most interested in the “id” (rank) and “URL” columns. You can either export this info or just copy and paste it into Excel.

Ann Smarty did a post about OutWit a while back and also has a custom scraper you can use for Google results. The only problem I’ve found with this is that sometimes you’ll get URLs with spaces in them from breadcrumbs, which makes it a little harder to filter things down in Step 3. If you are in a niche that doesn’t have this problem, this can be a faster way to go.

4- Download Your Scrape and Clean It Up

One problem with OutWit Hub is that it can be inconsistent. Sometimes you get local listings in the export, sometimes you don’t. Sometimes you get paid listings in there. Somteims you don’t. So you have to watch what you’re scraping and make sure you’re actually getting the right info. They usually have a heading row, but you still have to do some filtering and cleanup work to get an accurate list. When you do this, make sure you also update the id (rank) column to reflect the real ranking you’re seeing.

You can either export the data to a CSV, or you can also just copy and paste it into Excel. I like the copy and paste option because if I see some paid ads at the top or bottom of the data, I can just not copy those rows.

5- Rinse and Repeat

This is unfortunately the labor intensive part of this whole process. You’ll have to repeat this process for all of the keywords you want to check. Again, there are other tools that do a little bit more brute force against Google, but OutWit Hub is a great FREE tool that will help you get the data you need if you’re willing to take the time.

No matter which method you use, make sure that you add a column at the beginning that includes your shortname for each set before the rank and URL of each scrape. This way you can identify which set the rankings and URLs belong to later.

Also, make sure you’re combining all of your data into one spreadsheet so we can do the comparison and filtering later. In Step 3 of this whole process I’ll show you what to do once you have all of your scraping done.

SEOmoz Keyword Difficulty & SERP Analysis Tool

SEOmoz Keyword Difficulty & SERP Analysis

In the long run, I think that using the SEOmoz tool is a lot easier and cleaner to use for this exercise. One nice thing about using the Keyword Difficulty & SERP Analysis Tool is that you can run up to 5 keywords at a time, and you don’t have the cleanup work that you have to do with OutWit. Once difference between the two is that with OutWit you can dig as deep as you want to set your Google Search settings. With SEOmoz you will get the top 25 and that’s it.

Here are the steps to getting the same data with the SEOmoz Keyword Difficulty & SERP Analysis Tool:

1- Run a Report (up to 5 at a time)

Sometimes I’ve found that the tool will time out if you run 4 or 5, so if you’re having that problem just run 3 and you’ll have an easier time.

2- CSV Export

Once the report loads, click on it and then choose the “Export to CSV” link down towards the bottom. It’s above the table with all the pretty greens and reds.

CSV Export - SEOmoz Tool

3- Rinse and Repeat

The only columns we need are ‘Rank’ and ‘URL’. If you want to start getting in to Domain and Page Authority comparisons you could use that data as well, but for this blog post I’m just going to keep it simple.

Just like with the OutWit Hub data, make sure you’re combining all of your CSV download into one master file so you can do the filtering you’ll need to do.

Step 3 – Filter Down to Just Domain Names

In order to really do the comparison, you need to filter your SERP scraping down to just the domain names. With a little Excel formula magic, this is easily done. Here are the basic steps to follow in Excel. Since there are so many different version of Excel and other spreadsheet programs, I’m just going to give you the basic steps and formulas here so you can do what you need to in the program/version you’re using.

Before you do the steps below, make sure to MAKE A COPY OF ALL YOUR SCRAPED DATA. We’re going to filter down to just the domain names you’ve scraped, but that’s only so we have a list of unique domains that we can then do some counting and averages on. You have to leave your original data so you can get the counts. So I repeat, make a copy of all your scraped data and do the steps below on the copy.

  1. Use ‘Text to Columns’ and delimit on ‘/’. This is probably the easiest way to break out the http: and any other folders in the URLs you’ve scraped. Delete all of the columns that don’t have just the domain name.
  2. Get rid of www. Since some of your scraping with have URLs with www and some won’t, we need to get rid of these. Sort your list of domain names alphabetically. Then, do another text to columns on the domains that have the www in them. You can do this the easiest by doing the ‘Fixed Width’ option since www. is always the same width. You may also have other subdomains in your list, but honestly I would just treat these as separate sites from the main.
  3. De-dupe. Now that you have your list of just domain names without the www and folders, de-dupe this list so that you have a list of unique domain names.

Step 4 – Count # of Results and Average Rank For All Unique Domains

Once again, for this part I’m just going to give you the steps rather than screenshots since it might vary a little bit from spreadsheet program to spreadsheet program.

To set up your spreadsheet matrix, you should have all of your unique domains down the left, and then across the top you’ll have a column for # Results and Avg Rank for each of your keyword set shortnames. Put these at the top of the two columns and merge over them if you want to make it a little prettier, and it will give you something to reference in your formulas.

Getting Number of Results Per Unique Domain

  1. This formula will vary a little bit based on how big your data set is and where your list of original domains is.
  2. For example, if your first unique domain is in cell B3, your full data set of URLs is from cell D120 to cell D293, the column in the data set with the keyword set short names is in cells A120 to A293, and your first short name column name is in cell C1 your formula would look like this: =COUNTIFS($D$120:$D$293,”*”&B3&”*”,$A$120:$A$293,$C$1)
  3. Notice the absolute references for the cell ranges. This is critical, otherwise you won’t get the correct count.
  4. The “*”&B3&”*” is a wild card that basically says match B3 with anything before or after it. So you’ll get www, non-www, home page, and any other page for that domain name.
  5. If your formula looks good, copy it down to all of your unique domains.
  6. Repeat this process or all of your keyword sets.

What this number tells you is the number of times that unique domain shows up in your scrapes for that keyword set. If they show up a lot, than that’s something they are going after, and can be higher if they have multiple listings as well. If they don’t show up very much, than it isn’t an important set for them.

Getting the Average Rank Per Unique Domain

  1. This formula will also vary a little bit based on how big your data set is, etc.
  2. Let’s use the same example cells as listed above, but your Rank data is in cells B120 to B293. Your formula in Excel would look like this: =AVERAGEIFS($B$120:$B$293,$D$120:$D$293,”*”&B2&”*”,$A$120:$A$293,$C$1)
  3. Again, notice the absolute references and make sure you have them in there.
  4. If your formula looks good, copy it down for all of your unique domains.
  5. Repeat this process or all of your keyword sets.

What this number tells you is the average rank for that domain name for that keyword set. Naturally, the lower the number the better they rank on average. The higher the number, the less of a threat they currently are, but it also shows that they are at least showing up for that set.

Step 5 – Organize and Analyze

Once you have all of your formulas down and you’re happy with what you see, I recommend copying and then pasting back the values for your matrix. This way you can sort the data any which way you want without it messing up the data (I made this mistake once and it wasn’t pretty).

Now you have a really cool matrix that will show you by keyword set which sites are going after different sets, and how important each set is to them based on how often they show up and what their average ranking is.

Have some fun sorting by different columns and even highlighting the numbers and sites that stand out to you. Here’s a screenshot sample of a matrix I did once to help.

Sample Data

What To Do With This Info

One of the problems with competitive analysis is that site owners and marketers only look at the companies they know, the major players in their space. Well, with this technique you will also see the affiliate sites that are competing that you may have overlooked, how big of a player sites like Wikipedia are in your space, etc.

If you run this every couple of months, you can also see the changes that are happening in the SERPs and better keep an eye on those sites that are becoming more of a threat.

As you identify new competitors, you also now have another site to analyze for marketing ideas, competitive links, etc.

Let Me Know What You Think

I really hope that this has been a helpful post for you to learn a technique to identify more of the true competitors in your space, and then what to do with that information. I’m sure that there are other ways to get this information, and if you have any additional tips please share them in the comments below.

Taylor Pratt on Competitive Analysis #seochat

On Thursday, November 18, 2010 Search Marketing Weekly will be hosting an #seochat on Twitter. Taylor Pratt (@RavenPratt) will be our guest answering questions about competitive analysis.

Details:

Starts at 7:00pm Mountain Time
About 1 hour long
Use hashtag #seochat
Host: Greg Shuey
Transcript
Questions Answered

  1. What are the most important backlink metrics to look at when doing a competitive analysis?
  2. When doing an on page analysis, where are some major areas you tend to look for problems?
  3. How often should you perform a competitive analysis?
  4. How does a competitive analysis help you set the overall strategy for an SEO campaign?
  5. How does a competitive analysis help you set the overall strategy for an SEO campaign?

About Taylor Pratt

Taylor Pratt is the Product Marketing Manager of Raven Internet Marketing Tools, a multi-user Web-based application for managing SEO and social media campaigns. Prior to joining Raven, Pratt was a Senior Search Specialist at nFusion where he specialized in search engine optimization (SEO), conversion and Web analytics, and promotes the use of conversion-focused, analytics-based SEO.

Pratt began his Internet marketing career as a search analyst at LunaMetrics, a Web analytics consulting firm. While working at LunaMetrics and nFusion, his client roster ranged from small businesses to Fortune 500 companies.

In addition to helping companies win online, Pratt contributes to Search Engine Journal and Search Engine People covering the latest tools and techniques in social media marketing.

You can catch him on Twitter at twitter.com/RavenPratt. Pratt is a frequent at both small business and search marketing conferences, including PubCon, SMX Advanced, SMX: East & West and PubCon South.

Competitive Analysis From Taylor Pratt

I love analyzing competitors who rank for any given keyword. Through a competitive analysis, you can get a ton of great information. When done right, it should give you a pretty good road map of how your competitors began ranking (and continue ranking) and what you can do too to get there yourself to the top three.

This past week, I came across two really good resources from Taylor Pratt from Raven Tools. One is a slide show presentation from SMX East and two, a competitive analysis checklist to help you analyze your competitors. You should definitely check both of them out. Here they are…

Download the Competitive Analysis  PDF Checklist here.

  • ebony milf porn
  • mature foot lesbians
  • brutal anal movie
  • foresed sex free vids
  • ebony porn riding skills
  • kelly madison pussy ff
  • sex vids forced
  • naked woman having sex on camera
  • stroke the pussy
  • beautiful porn star free gallery
  • sexy erotic surprises for men
  • hardcore crying porn
  • To hot girls
  • mandy wife black cock
  • black spider with a white butt
  • myspace interracial wife
  • sexy naked pics
  • suck her little pussy and tits
  • sexy blonde cant stop eating
  • free video big dicks small chicks
  • free sex weing group sex photos
  • asian girls bare feet
  • Girls with big boobs fucking
  • mature pussy in hose
  • pussy pay per view
  • im kardashian pussy fucked
  • girl drinks cum porn video
  • Lesbian smelling feet
  • Emma watson fake nude pictures
  • lesbian erotic photos
  • nasty black girls fucking girls
  • Teens tied up and fucked
  • pics of big latina ass
  • sexy littel white girl nude
  • Solo oral sex
  • teen girls screw s
  • girl fingering girl video
  • Zac efron nude video
  • fee amature hme porn movies
  • cute bikini teen
  • picture of girl putting on deodorant
  • amatur teen anal
  • female ball busting sex video free
  • girl with tight breasts
  • aqua teen hunger force movie tabs
  • mature latino galleries
  • carmella bing big tits movies
  • cumming in my girlfriends ass
  • armry girls showing ass
  • Sexy brazilian girls naked
  • pattaya pussy galore bar girl video
  • erotic erotic art movies
  • Kylie minogue sex video
  • wife in stocking threesome
  • girls one guy porn tube
  • shemale xxx tube
  • Diane lane naked pics
  • Meagan good nude video
  • lesbian girls shower nude
  • youngest girl bikini video
  • interracial sex party
  • black porno sex
  • young deutch girls nude
  • free very young sex vidoes
  • lesbian moms having sex video
  • pictures of a girls boobs
  • tiny pussy tina
  • deepthroat face fuck free video
  • naked rainbow party sex
  • chicks harley davidson
  • fuckable chines teen video
  • innocent teen girl porn
  • free pics of nz girls
  • Maryse ouellet nude pictures
  • movie auditions teens
  • pictures of young sunflower
  • free toon fuck pics
  • biggest butt porn
  • sexy nonude young models
  • Diya mirza nude photos
  • porn movies for for women
  • how to fuck your teenage daughter
  • free black buty sex videos
  • estelle bingham naked plympics
  • free asian fisting videos
  • mature twink amateur galleries
  • xxx ebony tube free
  • home video huge boobs
  • huge shemale tits and cock videos
  • girls naked young anal
  • male sex threesome
  • free asian school girl videos
  • hairy anal sex videos
  • teen lesbian naked solo
  • nimbus video vibrator porn
  • anal sex with cat
  • women with objects in pussy
  • Girls locker room porn
  • hi res erotic porn pics
  • cuckold white wife want jamaican video
  • anal movie thumb
  • big tits free pics bikini
  • amateur videos college
  • free naked college girls home videos
  • teen party washington pa
  • Unexpected cum in mouth
  • ass groping videos
  • anime sex movie thumbs
  • anal butt rectal
  • Manisha koirala nude videos
  • hairy asian pussys
  • school girl fuck videos
  • hardcore porn movies twilight
  • skinny black shemale tube videos
  • wife sharing pics gallerys
  • lesbian forced to orgasm
  • very young nude girls ice
  • Hot naked sex videos
  • blonde teenagers black dick
  • black women long nipples
  • teen black pussy upclose
  • Hot ass thongs
  • celebrity and movie star porn
  • wding lost camera honeymoon naked sex
  • Cum in ass video
  • xxx young girl and boy
  • free red teen bush videos
  • ebony adult nude black men
  • shemale only videos
  • Teen asian ass
  • hard anal sex gallery
  • Sex ed movies
  • mini girl model pics
  • Hot girl booty shake
  • Lesbians eating girls out
  • black amateur booty bouning
  • hot girls giving a blowjob
  • Wet teen vagina
  • tits anal movie
  • teen ex girlfriend pictures
  • d cup sexy blonde lesbian
  • Tiffany teen nude pics
  • hardcore triple xxx porn in rome
  • free lactating big tits movie
  • free sex jizz tube video
  • sexy young boys art
  • teen home made x videos
  • fuck your friends sister videos
  • naked butch girl
  • stunning mature movie galleries
  • interracial young girl gangbangs
  • free teen blowjob movies
  • legal flat top teen pics
  • free hot girls vids
  • throat gager pics
  • deep throat amateaur videos
  • Elisha cuthbert nude photo
  • black nurse mom freak huge ass
  • swingers video sex
  • ir wife vids
  • tattoos of naked girls
  • girl ejaculates video
  • Hot tamil ass
  • Teen dildo pic
  • free overseas teen gallery
  • tight little black girls getting fucked
  • japanese bizzare sex
  • porn tube video download
  • s japanese modern girls
  • teen real amateur
  • young animai sex vids
  • amateur teen interracial videos
  • amateur sex video married couples
  • chicks on chick porn
  • muscular porn mpeg galleries
  • girl ass fingered
  • gang bang horny teen videos
  • sex videos divx
  • japanese erotic force stories
  • perfect young volelyball babes
  • pics of black pearl pirate ship
  • girls under galleries thumbs
  • Black ass and titties
  • teen romanian pussy
  • vintage movies porn
  • cute chubby girls galleries
  • single girl erotic massage kona island
  • homo thug big black dicks
  • cheating fucking friends wife stories blackmail
  • Big boob seducing
  • Teens bedroom wall decals
  • youngest naked girls ever
  • keri russell erotic pics
  • teen sluts pussy skirt
  • home video amateur free
  • furries sex videos
  • mature women who hunt young lesbians
  • hd young teen sex videos
  • wife pussy flashing
  • why does a girls pussy swell
  • mcdonalds girl flashing
  • free internet sex amateur
  • sexy drawing of girls
  • sex vid of milk into ass
  • kick his ass video
  • teenie fuck spy cam
  • naked womenwet pussy
  • free young young porn movies
  • free high quality teen pics
  • pictures of black leaders
  • Bulgarian sex movie
  • free hardcore sexy girls
  • free pics public sex outdoors beach
  • Daniela denby ashe nude pics
  • black teen tight
  • white men and asian women porn
  • bushy pussy three way fuck
  • sex dolls fuck video
  • dildo chicks sex
  • The best pussy pics
  • likes girls pussyu
  • emmy black tits
  • girl fuck camera
  • free missonary sex guide video
  • step s force girls porn
  • What makes a girl sexy
  • gtanny sex videos
  • orgasm with me movie
  • amateur lesbian ass play videos
  • tightess pussy by race
  • young little lesbian girls pussy
  • black school girls getting fuck
  • girls costume long black gloves
  • free videos mature lesbian bosses
  • sex porn mature tube asian movies
  • teenie home fuck
  • nakednews naked videos
  • Big ass little waist
  • free full sex sleeping movies
  • thick oily black ass
  • upskirt youngest japan street
  • free amateur videos hardcore
  • Naked strip porn
  • wool stockings pussy
  • gaping milf pussy pics
  • did ham have a black wife
  • video of britney fornshell naked
  • softcore couples sex
  • ask porn interracial
  • friendswood pussy ass
  • males strippers sex at bachelorette party
  • white slave mile east fuck movie
  • pictures of blonde haired movie stars
  • dirty talk handjob video
  • can a snake fuck a pussy
  • nina hartley teaches eat pussy
  • Anne archer nude pictures
  • Bikini hot ass
  • naked lesbian sistas
  • teen sluts group sex pics
  • free young girls teens porn movies
  • little teen suck big cock
  • teenie young porn
  • interracial double three black facial cumshots
  • wife party fucks
  • xxx group sex videos
  • amateur nude gallery archives
  • pictures pregnancy breasts changes
  • sexy blonde tight white tee
  • petite strip ibp spec
  • young lil ass
  • girls and boys free fucking videos
  • Nude photos of vida guerra
  • watch fuck machine movies
  • Nude thin girls
  • free sex and violence videos
  • young pron pictures of girls
  • petite jean national forest
  • Pics of sex and the city
  • chinese girl black man
  • Super sexy lesbian
  • giant dick anal sex tape
  • Girls sexy nipples
  • ebony teen backdoor
  • coach fucks young teen
  • my fucked my wife videos
  • Anal sex nude
  • tiny naked teens
  • huge boobs that want huge dick
  • big black ass nurse
  • nuns pics free erotic
  • school nurse movie big nipples
  • anal licking movies
  • tween porn vids hoe made
  • college girls naughty videos
  • young woman doin anal
  • black anal moms
  • free sex tv porn movies
  • girl get sex in public place
  • white wife loves blackmans sperm
  • school girl sex free
  • shitty gapping anal hardcore
  • mature homemade sex free
  • naked mature haveing sex
  • Big tit dragon
  • cock panties porn video
  • young and anal allysin chaynes
  • pussy blonde fuck slutload
  • lela star video orgasm
  • girls showing breasts at golf tournaments
  • abhay deol sex scene
  • amateur sex hunters
  • wet young lesbian
  • ass butt sexy hot
  • young skinny schoolgirl free porn
  • nikara naked pics
  • hairy teen ebony pussy
  • hanover nh petite womens clothing
  • amateur pornography kim and daniel
  • Naked sexy penis
  • free fat hairy ebony pics
  • hairy vintage xxx porn galleries
  • wet pussy teen panty
  • asian lesbian foot fetish
  • free sex military men video
  • mature shemale pussy
  • orgasm during filming porn
  • gy style teen video sex
  • hot girls nipples
  • bed lounge petite
  • Plump big tits
  • Full exploited black teens
  • milf young girls
  • ebony cappuchino porn
  • young jello tits
  • r ygold pussy
  • hot bride to be fuck blackmen
  • girlfriend amateur blowjob
  • black boy white wife
  • amateur milf anal fuck videos
  • topless woman pics
  • voyeur sexe amateur
  • lacey turner sex video free
  • blonde slut big ass porn
  • free hairy wet pussy outside
  • Daughter caught masturbating
  • flat chested porn videos
  • pussy pierced movies free
  • two dicks in the ass galleries
  • u tube drunk college girls sex
  • hardcore black fucking galleries
  • Asian anal sex movies
  • free teen pantyhose videos
  • female free online stripping movies porn
  • homemade teen threesome porn
  • tranny lesbos pussy
  • girls sexy pools baby oil
  • big natural tits sex
  • hot nude babes video
  • young black girls images
  • giant juggs blowjob movies
  • mother son fuck movies
  • sharuk khan movie wife dies
  • fuck my sex
  • professional porn movies
  • fuck trixie teen video
  • naked office pics
  • young teen females nude
  • Jo garcia nude videos
  • sexy scene girls clothes
  • young nude russian nudist girls
  • and girl anal
  • girl gets pregnant video
  • hentai teen boyfriend fuck
  • Naked nude lesbians
  • black cum in asian
  • erotic brother and sister sex stories
  • hosewifes used as suck slaves
  • teen worker blowjob
  • fuck my italian pussy
  • illegal nude pics young girls
  • asian planet video store
  • black girls with sexy legs
  • free busty redhead hardcore videos
  • dans free fuck galleries
  • Hot girl big tits
  • older lesbian fucks young girl
  • m f teen st anal
  • petite cashmere sweater
  • amateur pregnant pics
  • evil girls having sex
  • angus young as devil pics
  • free ebony teen sex vids
  • pussy face fucking movies
  • white amature wives fuck black cock
  • sex flix of naked women
  • blacks inside blonde
  • wife upskirt and housewife pantyhose sex
  • youngest amateur nudity
  • free prostate orgasm videos