How can scraped data be used for keyword and advertising research?

ProcrustesPR

Scraped data can help you compile the keywords competitors rank for, the article titles and metadata they use, and the paid search ads they run. Comparing this information with data from your own website can inform keyword research and content planning. The aim is to gather information for research, not to copy articles or ads. The following is based on an article published on neilpatel.com in 2018. Its descriptions of tool interfaces, plugin features, and usage requirements reflect that time; the original steps may no longer work today.

ProcrustesPRProcrustesPRDo I still need SEO if I already run Google Ads?

First, decide which data to research

The article suggests researching organic search keywords, article metadata, and PPC (pay-per-click) ads. Organizing the data into three categories helps keep information for different purposes separate:

  • Organic search keywords: Identify the terms competitors rank for as a starting point for finding candidate keywords.
  • Titles and metadata: Examine how competitors describe their article topics to inform content planning, rather than copying their wording directly.
  • Paid search data: Research which keywords competitors pay for and what types of ads they run.

The article also recommends scraping your own website, not just collecting information about competitors. For example, compiling article titles and metadata from both your site and competitors’ sites makes it easier to compare them later.

Compile paid keywords, sort by cost, and export

The 2018 article demonstrates how to get a list of paid keywords in SEMrush. First, open Positions under Advertising Research and enter a domain to see the keywords it paid for through the service then known as AdWords. Then sort by the CPC column and export the list as a spreadsheet.

The point of this example is to organize paid keywords into a list you can track over time. Sorting by CPC only compares the cost information in that column. The article provides no conversion or profitability data, so it cannot establish which keywords are most worth bidding on. Keywords a competitor has paid for are not necessarily suitable for your own business.

Consider Excel if you already have data from compatible tools

If you already use compatible tools for keyword research, SEOGadget for Excel is an option described in the article. At the time, the plugin was described as a way to import data from tools such as SEMrush and Moz into Excel and fill in fields automatically, reducing the need to switch between tools.

This approach requires compatible data sources and access to the data you need. Although the article described the plugin as free, it still required API keys for services such as SEMrush. That does not mean the entire process of obtaining data was free. The article alone cannot confirm whether the plugin still works today or provide current plan or pricing information.

Use the data for research, not to republish others’ content

The article clearly distinguishes research from plagiarism: keywords, titles, metadata, and advertising information can be extracted to improve marketing, but others’ articles should not be republished word for word or slightly modified and claimed as original. The author also warns that plagiarism harms relationships with other blogs. The focus here is on the distinction between research and plagiarism; the author’s claims about search penalties are not being presented as current official Google rules.

ProcrustesPRProcrustesPRHow is organic traffic different from paid traffic?

Scraped data for keyword and advertising research: perspectives from across the web

How can search suggestions expand a list of candidate keywords?

The 2018 article demonstrates collecting candidate terms from sources such as Google Suggest and YouTube search suggestions, then removing duplicates from the resulting list. This can help broaden the scope of initial research, but the terms on the list remain only candidates. The material provides insufficient data to assess advertising performance and does not confirm whether the tools and methods described at the time are still usable today.

Can you still research advertising data when websites block crawlers?

It may still be possible, but the 2018 article alone cannot confirm whether the advertising data you need is accessible today. The author argued that AdWords advertising data was less likely to be blocked at the time than website keywords or blog metadata, and used SEMrush as an example of a tool for researching paid advertising keywords. This was a comparison made at the time, not a guarantee that all advertising data could be accessed.

ProcrustesPRHow do cookies work, and what are their uses and privacy risks?

References

  1. How to Use Content Scrapers to Automate these 7 SEO Hacksneilpatel.com · 2018-02-16T15:00:03+00:00 — Discusses: How can search suggestions expand a list of candidate keywords?; Can you still research advertising data when websites block crawlers?

Related articles

Do I still need SEO if I already run Google Ads?How is organic traffic different from paid traffic?How do cookies work, and what are their uses and privacy risks?Can AdSense Be Used for SEO? Ad Monetization Is Not Ranking OptimizationWhat Value Does SEO Offer the Travel Industry?How Do SEO and PPC Differ? Speed, Costs, Benefits, and LimitationsHow Are SEO, SEM, and Backlinks Related?How Are SEO, Paid Search, and SEM Related?
ProcrustesPR
ProcrustesPR