Build an AI Research Agent with Google, Bing & URL-to-Markdown APIs

An AI research workflow can use two separate steps: first request live Google or Bing results for a query, then choose a result URL in your application and convert that page into Markdown. This keeps search discovery and page reading explicit, so you can inspect the source material before passing it to an LLM or another downstream system.

This tutorial uses the current SERPpost search and URL Extraction endpoints. It does not claim that a search result is factual, that every URL can be extracted the same way, or that one API is a replacement for every web-search service. Test the requests with your own inputs before relying on them in production.

The workflow in plain language

1. Send a topic to the SERP API with t set to google or bing.
2. Apply your own selection logic to the returned results. For example, you may choose a trusted domain, a result with a complete URL, or a source your application is allowed to use.
3. Send that selected URL to the URL Extraction API with t set to url.
4. Store the source URL alongside the Markdown and pass the material to the next stage of your workflow.

Use the search step when the application needs fresh discovery. Use URL Extraction when it already knows the target page and needs a clean Markdown-ready representation of that page.

Before you start

You need a SERPpost API key from your dashboard. Keep it in a server-side environment variable; do not put a bearer token into client-side JavaScript or commit it to a repository.

New users can validate the interface with 100 free credits without a credit card. Check the current request parameters and response format in the API documentation before production use. The current credit model lists one credit per SERP request and two per URL Extraction request; check pricing again if you are planning a paid workload.

A minimal Python example

The example below mirrors the current documented endpoints. It searches one engine, deliberately chooses the first result only as a demo, and then asks the URL endpoint for Markdown. Replace that demonstration selection rule with the criteria your application actually needs.

import os
from urllib.parse import urlparse

import requests


API_KEY = os.environ["SERPPOST_API_KEY"]
SEARCH_URL = "https://serppost.com/api/search"
URL_EXTRACTION_URL = "https://serppost.com/api/url"
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}


def post_json(url, payload):
    response = requests.post(url, headers=HEADERS, json=payload, timeout=30)
    response.raise_for_status()
    body = response.json()
    if body.get("code") != 0:
        raise RuntimeError(body.get("msg") or "SERPpost API returned an error")
    return body.get("data")


def search(query, engine="google"):
    if engine not in {"google", "bing"}:
        raise ValueError("engine must be 'google' or 'bing'")
    return post_json(
        SEARCH_URL,
        {"s": query, "t": engine, "d": 5000, "p": 1},
    )


def extract_markdown(target_url):
    parsed = urlparse(target_url)
    if parsed.scheme not in {"https", "http"} or not parsed.netloc:
        raise ValueError("selected result does not contain a usable URL")
    return post_json(
        URL_EXTRACTION_URL,
        {"s": target_url, "t": "url", "w": 3000, "d": 20000, "b": True},
    )


results = search("AI agent web research workflow", engine="google")
if not isinstance(results, list) or not results:
    raise RuntimeError("no selectable search results were returned")

# Demonstration only: production code should use an explicit relevance,
# domain, freshness, and policy-aware selection rule.
selected_url = results[0].get("url")
markdown = extract_markdown(selected_url)

print({"source_url": selected_url, "markdown_characters": len(markdown)})

Choose the search engine explicitly

The search request uses the same endpoint and changes only the t parameter:

Do not treat this switch as a promise that both engines return the same URLs or that one is always better for a particular audience. Send representative queries through the API Playground and compare the result fields your application needs.

Select a source before extracting it

An AI agent should not blindly extract every result it sees. Decide what makes a result usable for your task before making an extraction call. A minimal selection policy might check:

• the URL is present and uses HTTP or HTTPS;
• the domain is appropriate for the research task;
• the application can retain the source URL for citation or review;
• the result has the fields your downstream step expects; and
• the workflow handles an empty list, an error response, or a page that needs different rendering settings.

The API gives your application a request/response boundary. Your application remains responsible for source selection, reviewing output, handling retries, and complying with the terms that apply to its target URLs.

A note for legacy Bing Search API users

Microsoft announced that Bing Search APIs retired on August 11, 2025. If your existing application used that legacy interface, first write down the exact inputs, output fields, locations, rate behavior, and downstream assumptions it needs. Then test a representative SERPpost request in the Playground before changing production code.

This is a workflow-evaluation step, not a claim that one service can substitute for every Microsoft or Azure feature without testing the required behavior. See the Microsoft Lifecycle announcement for the retirement notice.

Move from a demo to a production workflow

Before connecting extracted text to an LLM, add the safeguards your application needs:

1. Keep API keys on the server and log only non-sensitive request context.
2. Store the selected URL, request time, and any metadata needed for review.
3. Set timeouts and retries deliberately; do not retry an input forever.
4. Validate that required fields exist before the LLM or another worker consumes them.
5. Start with the Playground, then use 100 free credits for a lightweight end-to-end test.
6. If the observed interface, request capacity, and credit model match the workload, review pricing before buying a pack.

Frequently asked questions

Is the Bing Search API still available?

Microsoft states that Bing Search APIs retired on August 11, 2025. That retirement does not determine which current web-search workflow is right for your application; test the exact API behavior, output, and operating requirements you need.

Does every AI agent need both search and URL Extraction?

No. Use search when the agent needs fresh discovery. Use URL Extraction when it already has a target page and needs Markdown-ready content. A workflow may use one or both steps depending on its job.

Can I send the first result directly to an LLM?

You can design that behavior, but it is safer to apply an explicit source-selection and validation rule first. The demonstration code uses the first result only to show the request sequence; it is not a production recommendation.

How can I test the workflow before buying credits?

Use the API Playground to inspect sample requests, then register for 100 free credits without a credit card. Confirm the current endpoint rules and credit model in the documentation and pricing pages.

Sources and review date

Reviewed August 25, 2026. API details and pricing can change, so verify the current SERPpost documentation and pricing before a production decision.

• SERPpost API documentation
• SERPpost API Playground
• SERPpost pricing
• Microsoft Lifecycle: Bing Search APIs retiring on August 11, 2025