Baidu introduces an agent benchmark focused on finished work

Measuring whether the work gets delivered

Baidu's DuMateBench evaluates agents on finished office and productivity work rather than on a conversational answer alone. An agent receives a workspace and tools, carries out a task and produces an artifact that an evaluator can inspect. The research paper specifies 200 tasks spanning eight broad scenarios and 17 finer categories.

Those counts correct the earlier brief's description of more than 200 tasks in six categories. The paper, submitted on August 27, is the more precise reference for the benchmark's composition. Its examples originate in anonymized, privacy-screened production sessions and are checked by humans before inclusion.

Keeping the context around a request

The researchers preserve relevant conversation history, persistent configuration and workspace material that existed before a task was solved. This makes the starting condition more substantial than an isolated instruction pasted into a fresh chat. A work request can depend on what was already discussed and on files that the user expects the assistant to understand.

The benchmark introduces three kinds of difficulty: insufficient information, unstable execution conditions and noisy material. It uses isolated Docker environments and combines deterministic checks with model-based judging. The paper evaluates five agent frameworks paired with four language models and reports a gap between partial progress and strict task completion.

Why the agent harness matters

The released software explicitly distinguishes a model from the surrounding agent program. A harness converts model responses into tool actions; a runner supplies the task container; an evaluator scores the resulting work. Changing any of these components can change the observed result even when the model name stays the same.

That distinction is particularly consequential for file-based tasks. A model can describe a correct spreadsheet operation yet fail to save a usable workbook. Conversely, a strong tool integration can make an otherwise ordinary reasoning step reliable. DuMateBench's artifact checks expose this gap between stating an intention and leaving the requested output in the workspace.

A download is not yet a valid benchmark run

The public dataset is about 3.5 GB and is distributed separately from the repository's runnable template. Its task packages must be supplied with the documented container environment before execution. The project requires Python and Docker Compose, while some evaluators also need tools such as LibreOffice, FFmpeg and PDF utilities. A failure while building that environment is therefore not evidence that the model failed the underlying office task.

The maintainers also distinguish local experiments from formal leaderboard submissions. A formal run uses their pinned Harbor dataset revision; a locally exported task collection is not accepted as equivalent evidence. That restriction helps preserve comparability when task packaging or evaluators change after release.

What the public release contains

Baidu publishes the runner, evaluators, command-line tooling and Harbor integration in its repository, while distributing the complete dataset separately through Hugging Face. The checked-in development tasks are a small test set, not the full 200-task evaluation. The repository also provides a public leaderboard link.

A deterministic smoke test exercises the infrastructure with a fixed local agent. It is not evidence of language-model quality. Actual scored runs must reach the task evaluator and produce a reward file. This separation helps readers understand what a successful installation proves and what still requires an agent run.

The benchmark's contribution is a concrete definition of delivery: the requested artifact must survive inspection under the supplied task conditions. It makes failures in execution and recovery visible alongside reasoning failures, which are often hidden when evaluation stops at a plausible final message.

The public leaderboard distinguishes a weighted final score from strict completion. It gives 30% weight to deterministic requirement coverage and 70% to artifact judging, then averages across tasks. Missing expected artifacts receive zero in the judging component. A displayed final score is consequently not the percentage of tasks completed perfectly; the separate metrics preserve information about partial work and output quality.