How to Build Web Tools for AI Agents That Actually Work

Every agent framework — LangChain, CrewAI, AutoGen, Claude's tool use — lets agents call external tools. The most requested tool category? Web browsing. Agents need to look things up, verify information, gather data, and interact with websites. But as we covered in why AI agents need a better web scraping API, most existing tools weren't designed for autonomous agent use.

But most "web browsing" tools available to agents are terrible. Here's how to build ones that work.

What agents actually need from web tools

An agent calling a web tool needs reliability (works every time, on every site), structure (machine-readable output, not a wall of text), and speed (slow tools kill multi-step reasoning flows).

What most web tools give you instead: raw HTML that forces the agent to parse a DOM, rendered text that's still unstructured and noisy, or screenshots that can't be processed programmatically. None of these help the agent make a decision.

The structured tool pattern

The best web tools for agents return structured data that maps directly to the agent's task. If the agent is comparing prices, the tool should return prices as numbers. If it's gathering contact info, the tool should return names, emails, and phone numbers as typed fields.

# Bad: agent gets a wall of text
def browse_url(url: str) -> str:
    """Fetch a URL and return the page content as text."""
    return requests.get(url).text

# Good: agent gets exactly what it needs
def get_product_info(url: str) -> dict:
    """Extract structured product information from a URL."""
    return crawler.json(url, schema={
        "name": "string",
        "price": "number",
        "currency": "string",
        "description": "string",
        "in_stock": "boolean"
    })

The structured version is better for the agent in every way: less tokens consumed, no parsing needed, typed data it can reason about directly. This is exactly the pattern behind a URL-to-JSON API — define a schema, get structured data back.

Schema-per-task, not one generic tool

A common mistake is building one generic "browse the web" tool. Instead, build task-specific tools with schemas tailored to what the agent needs for each task.

tools = [
    Tool(
        name="get_pricing",
        description="Get pricing info from a company's website",
        schema={"plans": [{"name": "str", "price": "str", "features": ["str"]}]}
    ),
    Tool(
        name="get_company_info",
        description="Get basic company information",
        schema={"name": "str", "founded": "int", "hq": "str", "employees": "str"}
    ),
    Tool(
        name="get_article_summary",
        description="Get the main content of an article",
        schema={"title": "str", "author": "str", "date": "str", "content": "str"}
    ),
]

Each tool has a clear purpose, a descriptive name, and a typed output. The agent can choose the right tool for the job and know exactly what data it'll get back.

Error handling matters more than you think

When an agent's tool fails, the agent needs to understand why and decide what to do. "Internal server error" is useless. "Page requires authentication" is actionable.

Return structured errors with enough context for the agent to reason about:

{
  "success": false,
  "error": "page_requires_auth",
  "message": "This page requires login. Try the public pricing page instead.",
  "suggestion": "https://example.com/pricing"
}

An agent can work with this. It can try the suggested URL, ask the user for credentials, or skip this source entirely. It can't do any of that with a stack trace.

Caching is an agent's best friend

Agents often revisit the same URLs within a single reasoning chain. Without caching, each visit is a fresh network request — slow and wasteful. Cache extracted data with a reasonable TTL and your agent gets instant responses for repeated lookups.

This is especially important for multi-step tasks where the agent gathers data from several pages, then goes back to verify or cross-reference.

FAQ

How do I build a web tool for an AI agent?

Define a function with a clear name, description, and typed output schema. Instead of returning raw HTML or plain text, use structured extraction so the agent gets machine-readable data it can reason with directly. Each tool should be task-specific — get_pricing is better than a generic browse_url.

Why does AI agent tool use work better with structured data?

Agents burn context tokens parsing unstructured text. When a tool returns typed, structured data, the agent can immediately use the values in its reasoning without spending tokens interpreting a wall of HTML or prose. Structured output also makes errors explicit and machine-readable rather than buried in text.

Should I build one generic web browsing tool or multiple specific ones?

Multiple specific ones. A generic "browse the web" tool forces the agent to handle all the parsing and interpretation itself. Task-specific tools with schemas tailored to each use case are faster, consume fewer tokens, and give the agent actionable data instead of raw content to figure out.

The takeaway

Don't give your agents raw web access and hope for the best. Build typed, task-specific web tools that return the data the agent actually needs. Your agents will be faster, more reliable, and waste fewer tokens parsing noise. For an implementation walkthrough, see our guide on how to feed live web data to your LLM agent.