Early Thoughts on Agentic Data Environments

The Data, Agents, Processes LAB (DAPLab, dap.cs.columbia.edu) is a new lab that vertically integrates HCI, AI, Data, Systems, and Algorithms together to tackle safety and task quality limitations towards widespread agent automation. This talk presents some early work on the data systems part of the lab, and was presented at the NOVAS and DashSYS workshops at the Very Large DataBases (VLDB) 2026 conference.

LLM are text-in text-out, which can be interpreted as any kind of content

These are all content generation, where its value is predominantly the immediate content of the output.

Text can be interpreted as code that is executed, and is the basis of automation loops. This has the potential to transform how work is done.

Code does not execute in a vacuum. It runs within a system -- an execution substrate. For the database community, the dominant substrate would be the database management system, and the program would be SQL.

But the power of LLMs and agents is in its expansive capabilities to execute and automate _any_ digital system. This goes well beyond a database and encompasses the entire digital environment -- the OS, agent, harness, web services, etc.

All of these systems contain and manipulate data, and that's where the action is. So data management should expand its world view to systems with state.

The problem with automation is that LLMs are also unreliable -- and not in terms of failing fast nor easy-to-detect corruptions, but in terms of semantic and correctness issues that are subtle to detect. And this then corrupts the -semantic correctness- of anything that consumes and depends on the LLM's outputs.

Which eventually includes the computing environment. Incidents like the openai hugging face incident are top of mind, but subtle errors arise in any run-of-the-mill task as well.

The same way we don't rely on end-users to ensure their inputs and actions are safe and correct, we should not rely on LLMs and agents to be correct either. The Data Environment should absorb responsibility for enabling higher quality task outcomes and safety, and provide new capabilities to do so.

How do existing agent systems increase the likelihood of good task outcomes? Since the trajectory is just a sample, most approaches draw many samples. If a good outcome exists, this increases the chances.

But, if the trajectories manipulate the data environment, then the environment must support exploration. A basic way is to support branch & restore.

The downside to sampling trajectories is that it is also more likely to sample unsafe/wrong trajectories!

It's likely impossible to eliminate all unsafe trajectories. But we should deterministically eliminate entire classes of unsafe trajectories/behavior. It is important for the user's psychological safety to know that a task is within a "safe region".

This is likely a reason read-only and fully-sandboxed tasks are widespread and acceptable today.

In addition, the set of good, safe trajectories may be quite small, and agents are not good at efficiently finding them. Agents ultimately choose trajectories based on the information they are given. A large subset of this information is data related to the task, the environment, the agent's actions and outcomes, etc. Giving agents better information and feedback can help agents sample better trajectories.

This is akin to web forms quickly notifying the user of invalid form entries, so the user can also "sample" better form inputs.

Let's start with Branch & Restore.

Back in early Fall 2025, it was clear agents would be using database systems, and there were many products that claimed to support branching. But did they work? What functionalities and performance profiles should they have?

Rather than build yet another system, our PhD Elaine Ang, along with my colleague Kostis Kaffes decided to build a TPC-like benchmark called BranchBench, but focusing on agent exploration over databases.

We were motivated by DBMS application workflows that didn't yet exist but that we expected would eventually exist: software development where agents create code edits and continuously run testing, trying to identify a past change to reproduce a failure during agentic debugging, data cleaning pipelines where how a piece of data is cleaned affects downstream actions, test-time exploration algorithms like MCTS, and simulation.

The benchmark is initialized with TPC-C and -H data, and executes an exploration tree. Each node in the tree corresponds to branching from its parent's state; mutating the database contents (rows, schema), configuration, physical design (indexes, partitions, etc); running queries/programs to evaluate the resulting database, and then deciding whether or not to stop.

An important workload are queries _across_ branches, which are often used to estimate a state's quality or to compare outcomes.

At a high level, our results from late Fall 2025 were as follows.

Most systems take very very long to create even a single branch and/or do not support concurrent branches.

Neon has good query performance because it allocates compute for each branch; internally, branching is fast because it is just a pointer to the WAL, but concurrent branches are limited due to resource costs per branch.

Dolt branches are very fast and supports arbitrary number of concurrent branches. Its use of a content-addressable tree as the storage layer and less mature query optimizer led to less-than-ideal query performance; the latter can be improved over time.

As a concrete example, the simulation workflow runs 1000 simulations, each in a separate branch, and needs to preserve all of the branches to compare across them at the end. Neon completes many simulations quickly but is limited to 30 branches (simulations). Dolt chugs along but times out due to slow query performance.

It's been great to see responses from some branchable databases, such as Dolt and Bauplan about using the benchmark.

We are revising the paper and putting together a leaderboard this fall (2026).

Branchable databases are nice, but they are only one piece of state. Most agents interact with the rest of the computer, and in particular, the terminal. While many containers, like Docker and Podman, support file-system checkpoint/restore, that's not enough.

Suppose the agent changes directory and creates a file.

This work is led by Jiakai (Alex) Xu.

then checkpoint and immediately restores. (in principle, this should be a noop)

but reading the file fails because the active directory was not checkpointed!

Systems don't need to branch _all state_, but do need to provide observation equivalence: from the agent's perspective, they should see the same action outcomes (state) as if each trajectory happened in isolation from the same starting state.

That's what Waypoint does! It's a container that branches process trees, the terminal, memory, file system, and channels within the container.

https://daplab.cs.columbia.edu/projects/waypoint/

We work with a startup called Veris to simulate webservices outside of Waypoint.

If the branching system doesn't support observational equivalence (like Docker and Podman+CRIU), then it's worse than not branching at all and just running from scratch 20 times.

More recently, we have preliminary results on 11 random tasks in Terminal-bench 2 that shows adding test-time search using a fairly basic critic pushes the accuracy-cost pareto frontier!

Another exciting use case is to run legacy web applications, such as a Shopify environment that Zhou Yu's group built, and "magically" support branching for users and/or web agents.

So far, we’ve only set up the ability to explore, but if it does an action that’s a bit sketchy (pink), and then a bad action (red), what can we do about it?

The agent that only sees tool inputs and outputs can’t reason about safety or policy following, but the data environment, which encompasses the harness, the prompts, memory, tool execution state, database, data flows, and the rest of the system, sees everything! It can often enforce policies deterministically. In the Data-flow Control work that our MS student Prajwal presented at the NOVAS workshop, we show that if you can trust the environment to deterministically enforce safety policies, then the environment can also steer the agent towards safe and successful actions.

Praj is also on the market!

The key result is summarized in this Agent Dyn benchmark. Most existing safety approaches try to filter tools, construct a "safe plan" from the task before execution, or detect safety violations in the prompt. They all give up safety for task accuracy.

In these benchmarks, observing and enforcing policies over data flows is sufficient to basically eliminate safety violations and improve the task success rates above not even having defenses at all!

As an example, if the user asks to re-purchase a shirt, data flows should only go from the orders table to the shopping cart. If this were enforced, then any prompt injections that try to purchase other items would be harmless, detected, and prevented.

The job of a data environment is to actively manage, retrieve, analyze, and identify data that the agent should have.

There's a lot of active work on agent memory, context management, NL2SQL, data agents, and so on.

Given that the conference is VLDB, we wanted to share two directions that are under-emphasized in the research today, and that we think are difficult to "train a model to solve".

The first is LakeQA.
The database community has worked for decades on data discovery -- finding tables that can add rows or attributes to an existing table. The use case is to "fill in" or "extend" a partial table, or to improve an ML model with more features/samples. The problem is a lack of adoption.

Jerry and Haonan (Peter) instead started with what people want -- question answering, which is a major use case for chat agents -- and see where the limitations are when we have a datalake with tens of millions of documents and tables.

Unfortunately, web search has never worked well for tabular data, QA agents assume the relevant data is directly provided, and data discovery systems are not directly relevant for QA.

In fact, it's not even clear how to measure progress!

So they created the LakeQA benchmark (lakeqa-bench.github.io)

The tasks are difficult because they _require_ the agent to 1) plan a sequence of subplans that must be answered, and each subplan requires the agent to 2) formulate search queries to find data with the needed evidence and 3) analyze the search results to compute the subplan's answer.

The tasks must correctly find half a dozen data sources, and are design to _require_ a competent search system. Even though the data lake is crawled from wikipedia and data.gov, the questions (and even subplans) cannot be answered by the model weights. Getting this right was tricky.

A major finding is that
1) almost 50% of the search results did not contain the document/table it needed and
2) even if the right data was retrieved, the agent would often fail to look at it.

This subsumes Text2SQL tasks because
1) each search query returns a new database the agent needs to reason about
2) the text "query" relies on the agent decomposing the task correctly and identify the right evidence it needs.

Another issue is that most benchmarks simply report the end-to-end results, but it's not clear whether the results are due to failures in the agent's ability to plan, the search system and the agent's ability to use search, and/or its ability to analyze data. Designing a benchmark system that can carefully ablate each component is tricky.

At the DashSys workshop, our undergrad Austin Wijaya presented an ablation system called SANA for data lake QA benchmarks and applied it to LakeQA and KramaBench. He's on the market!

The second project is based on our belief that tool outputs -- such as SQL generated by NL2SQL systems -- is not enough. The entire state of the data environment, such as policies, constraints, safety rules, and performance characteristics all matter for an agentic task, but they all hide "below the fold" and are not immediately visible to the agent. We saw this with data-flow control, and the same is for any consequential data task.

There were so many articles about database failures and issues related to data agents, that Peter and Jerry decided to survey nearly 600 blogs and forum posts to understand all of the properties that practitioners want in a "production ready" database beyond correct queries. The most common properties were packaged in a new CARE-ful benchmark that extends agent querying tasks to test these properties.

Here's one example of a property.

Jerry's row is deleted from the database,

but due to foreign-key constraints, the projects he works on are also CASCADE deleted!

The database state is correct and consistent, but the user is unhappy :(

The benchmark tests whether the agent can recognize this risk, and instead prompt the user about what to do.

A simulated user in the benchmark will tell the agent to assign the projects to peter.

Which then allows Jerry to be safely removed.

As data agents shift from demos, analysis, and side-projects towards serious applications, we expect that the level of expectations, requirements, and trade-offs will only grow in complexity.

Rather than ask the data agent to magically become better, there is a lot of complexity that the agent can offload to the data environment!

Agents are like a brain that’s smart but a bit unhinged.

This talk introduced some exploration and safety capabilities in a data environment, and ways to evaluate progress for important data tasks.

In addition, we have many many more projects in evaluation and data enviroments.

But just focusing on data is not enough for widespread trust of agent automation. Ultimately, automation is a _quality_ problem, that no individual discipline can solve. Instead, it requires vertical integration of the technology and usability stack to control quality end-to-end.

That's the thesis that the DAPLab is designed around.

That's also why the lab also includes applications and user interfaces; agent and model designs, RL/post-training, and optimizations; as well as cloud-level software infrastructure.

As a data management researcher, these are very exciting times. Traditionally, technologists design products that dictate how users should use them, and workloads mature and evolve at a slow enough rate that different CS disciplines can abstract away the application/user and design against that abstraction. For instance, database research abstracts the application as a SQL workload.

Agents have the capability to do _anything_ in digital space! And end-users are leveraging these capabilities. This may be the first time in technology where end-users have been given such a powerful, yet easy-to-use technology that the use cases they are dreaming up _and implementing_ is moving so fast that technology is being dragged behind it.

What this means is that technology -- such as data environments -- need to think beyond traditional assumptions and catch up to the user.

1) The state that needs to be managed goes well-beyond the DBMS, to the OS, application, browser, Python, and anything else that the agent interacts with,
2) but the management challenges are still just as difficult if not moreso
3) agents can in principle generate anything, including the systems it runs on. They do not because generating clean, reliable abstractions is still hard, and increasingly more valuable.

But most importantly, users need to know when they can fully rely on agents to complete tasks in order to _feel_ psychologically safe enough to adopt. This is a semantic guarantee that the value of automation will be high and far outweigh any of its potential risks.