Building Datasets for AI Agents: Extracting Structured Web Data at Scale
0
0
Ask a team running a production AI agent what breaks first, and the answer is rarely the model. It’s the data feeding it.
Most of the interesting failure modes in agentic systems — hallucinated company facts, stale pricing, contact details that bounce, “current” information that is months old — trace back to the same root cause. The dataset behind the agent wasn’t built the way a dataset needs to be built. It was scraped once, dropped into a vector store, and left to rot.
For data scientists and ML engineers, this is a familiar problem wearing a new hat. Model quality has always been bounded by data quality. What’s changed is the shape of the data an agent needs: not a static training corpus, but a live, structured, queryable slice of the web — refreshed on a schedule, consistent in shape, and traceable back to its source.
Why “just scrape it” stops working at scale
A one-off pull of a single company page or profile is trivial. A dataset of tens of thousands of companies, refreshed weekly, feeding a research or sales-intelligence agent, is a different engineering problem. Four requirements show up immediately once you’re past a prototype, and none of them are optional.
Structure. An agent, or a downstream model reasoning over the data, needs the same fields in the same shape regardless of which page produced them. A company record pulled from a directory listing and one pulled from a filings database need to resolve to compatible schemas before they’re useful in the same query. Raw HTML doesn’t give you that. Every page is its own arbitrary tree, and “parsing the DOM” means writing bespoke logic per source, per layout.
Freshness. Headcount, funding stage, pricing, review scores, job openings — most of what makes web data valuable is time-sensitive. A dataset collected once and never revisited becomes a liability the moment an agent starts reasoning over it as if it were current. Production pipelines need incremental collection: re-fetch what changed, skip what didn’t, and track a cursor so a nightly run doesn’t re-download the whole dataset from scratch.
Pagination and volume. Real datasets aren’t ten rows. Pulling reviews for a location, posts for a profile, or filings for a company means paging through hundreds or thousands of records, with rate limits in play the whole way. Naive approaches either load everything into the agent’s context window — expensive, and reasoning quality degrades past a certain size — or hand-roll cursor logic per source, which is exactly the kind of brittle glue code that breaks on the next site redesign.
Provenance. Every record needs a timestamp and a source reference. Without it, you can’t debug why an agent cited a number that’s since changed, and you have no basis for any trust signal in the outputs a downstream user sees.
The mechanics of a real pipeline
Picture a dataset that combines professional profiles, company and funding data, and local business listings into one queryable source — the kind of composite dataset a research or go-to-market agent needs. In practice, that means chaining several distinct source types: a people layer (professional profiles, work history), a company layer (funding rounds, filings, headcount), and a places layer (listings, reviews, hours) — each with its own page structure, its own pagination scheme, and its own rate limit.
A schema contract for a dataset like that might look as simple as this, regardless of which source populated a given record:
company:
name: string
domain: string
employee_count: int
funding_total_usd: int
last_updated: timestamp
source_url: string
The interesting fields aren’t the business ones. They’re
last_updated
and
source_url
— carried on every record, populated the same way no matter which underlying source supplied the data, so an agent’s claim (or a human auditing its output) can always be walked back to where and when it came from.
Handling that by hand means three or four separate scraping codebases, three or four different failure modes, and a maintenance job that never really ends, because every one of those sites redesigns its markup periodically, and a selector-based scraper breaks the moment it does. This is the actual argument for treating web data access as infrastructure rather than a script: a layer that abstracts each source behind a consistent API, returns the same JSON shape regardless of the page underneath, and detects and adapts when the underlying markup changes instead of quietly returning empty fields. Anysite is one example of infrastructure built specifically for this — a structured API layer over sources including LinkedIn, Crunchbase, Google Maps, SEC filings, and Y Combinator’s company data, so a dataset assembly job calls one consistent interface instead of maintaining a parser per source.
Filtering before it hits the model
There’s a second-order problem that only shows up once you’re feeding an agent instead of a spreadsheet: token economics. If your extraction layer returns everything and expects the LLM to filter it, you’re paying a language model to do what a database should be doing. A dataset of ten thousand profiles, dumped raw into context so the agent can find the forty that match a filter, is an expensive and slow way to run a query.
The fix is server-side filtering and aggregation, done before anything reaches the model: fetch broadly, cache the results, then filter, sort, and aggregate against the cache — count, average, group-by — and pass only the narrowed result set into the agent’s context. It’s the same principle as pushing a
WHERE
clause down to the database instead of pulling every row into application memory. Treat the extraction layer as structured web-data extraction infrastructure, not raw HTML retrieval, and this becomes a query problem you can optimize, rather than a context-window problem you have to work around.
Provenance and graceful degradation
The last piece is less glamorous than schema design and more important than it sounds: what happens when a source can’t be parsed cleanly. A pipeline that silently returns partial or malformed data is worse than one that fails loudly, because a silent failure propagates into whatever the agent concludes downstream. The pattern worth building toward, whether you’re rolling this yourself or evaluating vendor infrastructure, is graceful degradation: detect the structural break, attempt an automatic recovery, and if that fails, flag the record for review instead of shipping an incomplete row as though it were complete.
Treat it as an engineering discipline
None of this is exotic. It’s the same discipline data engineers have applied to ETL pipelines for a decade — schema contracts, incremental loads, provenance metadata, monitoring for drift — applied to a source that happens to be the open web instead of an internal database. The teams getting reliable results out of AI agents are rarely the ones with the most elaborate prompts. They’re the ones who stopped treating the dataset behind the agent as an afterthought, and started engineering it with the same rigor as everything else in the stack.
Artificial Intelligence – The Data Scientist