DeepResearch in Action: Building a Competitor Analysis Engine

Understanding your competitors’ digital strategy is crucial for success. What keywords are they ranking for? What new landing pages are they launching? What messaging are they using to attract customers? Answering these questions manually is a time-consuming, relentless task.

This is where DeepResearch shines. In this use-case guide, we’ll show you how to build a powerful, automated competitor analysis engine. This engine will leverage the DeepResearch methodology to provide you with continuous, actionable insights into your competitors’ online activities.

The Goal: Automated Competitor Intelligence

Our objective is to create a system that can:

1. Monitor SERP Presence: Track a competitor’s ranking for key industry terms.
2. Discover Strategic Pages: Automatically identify their most important product pages, landing pages, and blog posts.
3. Extract Key Messaging: Scrape titles, headings, and meta descriptions to understand their marketing angles.
4. Run Continuously: Operate on a schedule to provide up-to-date intelligence.

Architectural Overview

This engine will be an extension of the Python agent we built previously. We will modify it to focus specifically on a single competitor’s domain and extract marketing-related content.

Here’s the high-level architecture:

Component	Role	Technology
Seed Keyword List	A list of high-intent keywords your target audience uses.	Plain text file
SERP API	Queries Google for each seed keyword to find competitor pages.	SERPpost API
DeepResearch Agent	Crawls the competitor’s website starting from the SERP-discovered URLs.	Python, BeautifulSoup
Data Extractor	Scrapes specific marketing copy (H1s, meta descriptions).	BeautifulSoup
Data Store	A simple CSV file to store the findings for analysis.	Python csv module

Step 1: Defining the Scope

First, identify your competitor and the keywords you want to monitor. For this example, let’s assume our competitor is competitor.com and we want to track them for AI-related software terms.

Create a file keywords.txt:

AI content generation tool
best AI writer for SEO
AI-powered marketing copy

Step 2: Modifying the DeepResearch Agent

We’ll adapt our previous Python script. The key changes are:

• Read keywords from keywords.txt.
• Filter SERP results to only include our target competitor.
• Modify the scraper to only follow links within the competitor’s domain.
• Extract specific marketing text: title, meta description, and H1.

Here is the modified script, competitor_analyzer.py:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import time
import csv

# --- Configuration ---
API_KEY = 'YOUR_SERPPOST_API_KEY'
BASE_URL = 'https://serppost.com/api/search'
COMPETITOR_DOMAIN = 'competitor.com' # Replace with your target

visited_urls = set()

def search_competitor_pages(keywords):
    """Finds competitor pages from SERP for a list of keywords."""
    print(f"🔍 Searching for {COMPETITOR_DOMAIN} across {len(keywords)} keywords...")
    competitor_urls = set()
    for query in keywords:
        params = {'s': query, 't': 'google'}
        headers = {'Authorization': f'Bearer {API_KEY}'}
        try:
            response = requests.get(BASE_URL, params=params, headers=headers)
            response.raise_for_status()
            data = response.json()
            for result in data.get('organic_results', []):
                if COMPETITOR_DOMAIN in result['link']:
                    competitor_urls.add(result['link'])
            time.sleep(1) # Be respectful to the API
        except requests.exceptions.RequestException as e:
            print(f"❌ SERP API request failed for '{query}': {e}")
    print(f"✅ Found {len(competitor_urls)} unique competitor URLs from SERP.")
    return list(competitor_urls)

def scrape_competitor_site(url, depth=2):
    """Recursively scrapes a competitor's site for marketing insights."""
    if url in visited_urls or depth <= 0:
        return [], []

    print(f"{'  ' * (3 - depth)}↳ Analyzing Page (Depth {depth}): {url}")
    visited_urls.add(url)

    try:
        response = requests.get(url, headers={'User-Agent': 'CompetitorAnalysisAgent/1.0'}, timeout=10)
        soup = BeautifulSoup(response.text, 'html.parser')

        # Extract marketing-focused data
        title = soup.title.string.strip() if soup.title else ''
        h1 = soup.find('h1').get_text().strip() if soup.find('h1') else ''
        meta_desc = soup.find('meta', attrs={'name': 'description'})['content'] if soup.find('meta', attrs={'name': 'description'}) else ''
        
        scraped_data = {'url': url, 'title': title, 'h1': h1, 'meta_description': meta_desc}

        # Discover new URLs within the same domain
        new_urls = []
        for a_tag in soup.find_all('a', href=True):
            link = urljoin(url, a_tag['href'])
            if urlparse(link).netloc == COMPETITOR_DOMAIN and link not in visited_urls:
                new_urls.append(link)
        
        return [scraped_data], new_urls
    except Exception as e:
        print(f"❌ Failed to analyze {url}: {e}")
        return [], []

def run_analysis(keywords_file, output_csv, max_depth=2):
    """Main function to run the competitor analysis."""
    with open(keywords_file, 'r') as f:
        keywords = [line.strip() for line in f.readlines()]

    initial_urls = search_competitor_pages(keywords)
    
    all_insights = []
    queue = [(url, max_depth) for url in initial_urls]

    while queue:
        current_url, current_depth = queue.pop(0)
        insights, new_urls = scrape_competitor_site(current_url, current_depth)
        all_insights.extend(insights)
        if current_depth > 1:
            for new_url in new_urls:
                queue.append((new_url, current_depth - 1))
        time.sleep(0.5)

    # Save to CSV
    with open(output_csv, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=['url', 'title', 'h1', 'meta_description'])
        writer.writeheader()
        writer.writerows(all_insights)

    print(f"\n--- Analysis Complete ---")
    print(f"✅ Insights from {len(all_insights)} pages saved to {output_csv}")

if __name__ == '__main__':
    run_analysis('keywords.txt', 'competitor_insights.csv', max_depth=2)

💡 Note: This script now saves its findings to a competitor_insights.csv file, creating a structured dataset for you to analyze.

Step 3: Running the Engine and Analyzing Results

To run the engine, simply execute the Python script:

python competitor_analyzer.py

The script will perform the DeepResearch process and generate a competitor_insights.csv file. When you open this file, you’ll have a spreadsheet view of your competitor’s key pages and their core on-page SEO and marketing messages.

url	title	h1	meta_description
https://competitor.com/ai-writer	AI Writer for SEO	The Best AI Writer…	Generate SEO-optimized content…
https://competitor.com/pricing	Pricing Plans	Simple, Affordable Pricing	Explore our pricing plans…
…	…	…	…

By running this script weekly, you can track changes in their titles and descriptions, and discover new landing pages as soon as they start ranking on Google.

Conclusion: From Data to Strategy

This use case demonstrates the true power of DeepResearch. You’ve moved beyond simple scraping to build an automated intelligence system. This engine provides a continuous stream of structured data about your competitors’ strategies, directly from the source.

With this data, you can answer critical business questions:

• Are they launching new features? (Look for new product pages)
• Are they changing their target audience? (Analyze changes in H1s and meta descriptions)
• What new content marketing topics are they focusing on? (See new blog posts)

This is the strategic value of DeepResearch. It’s not just about data; it’s about automated, actionable insight.

Ready to build your own marketing intelligence tools? It all starts with a reliable SERP API.

Try SERPpost for free → and start uncovering what your competitors are doing right now.

Related Resources

• How to Build a DeepResearch Agent in Python
• Extracting Competitor Data with a SERP API
• Building SEO Tools with a SERP API: A Guide
• SERP API for Marketing and SEO Agencies