A benchmark for multimodal agents

VisSearch-Bench

Evaluating Long Horizon Visual Search
in Multimodal Agents

ZhengXian Wu1,2,*,‡Qianhao Nie1,2,*Zhuohong Chen1,*Junjie Gao2Yuyao Zhang2,3Yunyao Yu1Hangrui Xu1Cheng Cheng2Guiyu Ma2Jin Zhang2Xingzhong Xu2Kai Yang2,†Haoqian Wang1,†

1 Tsinghua University2 Ant Group3 Renmin University of China

* Equal contribution   † Corresponding authors   ‡ Work done during an internship at Ant Group

Can an agent keep track of what it sees
as the search gets longer?

204 human-written tasks that require agents to connect visual evidence,
object identities, and external facts across the open web.

Hugging Face Explore the tasks
ONE IMAGE. A CHAIN OF DEPENDENT QUESTIONS.FIGURE 2 · TASK 126
Sequential visual search

From a building photo
to a step-free entrance.

Each answer changes what the agent needs to find next.

  1. 01Identify the buildingMatch the image to the right wing.
  2. 02Trace its historyFind the designer, year, and original use.
  3. 03Find the makerspaceVerify how to visit and whether to book.
  4. 04Locate the accessible entranceConnect the evidence to a street name.
See. Search. Connect. Verify.A task illustration from the benchmark
204Human-written tasks
5Search families
7 / 25Primary / secondary domains
26.47%Best reported accuracy
The research question

Seeing is only
the beginning.

Long searches demand more than a correct first impression. An agent must organize, connect, and use visual information as objects, candidates, and search targets change.

Visual context management

Keep the evidence connected.

A visual observation must stay linked to the right object, the right source, and the right next step.

◉Visual evidenceAppearance · local details · layout
⌘Connected contextObjects · facts · candidates
◎The next searchCompare · filter · follow · verify
01 / COVERAGE

Find every required item.

A convincing partial list can still miss the task. The agent must inspect the complete candidate set and preserve object-level associations.

02 / CONTINUITY

Carry the right context forward.

An earlier visual decision can define later searches. A wrong identity or lost constraint can redirect the entire chain.

03 / VERIFICATION

Return to the visual evidence.

Textual matches are not enough when the answer depends on a local detail, a spatial relation, or an exact visual identity.

Inside the benchmark

Seven tasks.
Five ways to search.

Explore the seven examples in Figure 2. Each one turns visual evidence into a different search challenge.

FIGURE 2 · 01 / 07

A taxonomy of visual search.

Task families describe the structure of the search, independently of its content domain.

Human-written. Evidence-grounded.

Built around
verifiable search.

Public sources come first. Questions are built around traceable evidence, fixed reference answers, and explicit visual requirements.

01

Curate public sources

Select bounded entity sets, reliable image links, and independently verifiable facts.

02

Connect the evidence

Organize entities, images, candidate sets, and facts into traceable evidence structures.

03

Write the task

Apply a search-family structure, then lock the reference answer and solution.

04

Check and review

Verify coverage, accessibility, and shortcuts; human reviewers inspect every retained task.

Breadth meets depth

Seven domains.
Many kinds of visual evidence.

From museum collections and scientific specimens to buildings, devices, and everyday objects.

187Tasks with initial images
17Text-initiated visual searches
17.6Mean reference-solution steps
61Maximum reference-solution steps

Reference-solution steps include planning, checks, and answer preparation. They are not tool-call counts.

Main results

Search helps.
The challenge remains.

The strongest reported system fully solves 26.47% of tasks. Partial progress and complete answers reveal different strengths.

Best complete-answer accuracy26.47%

Kimi-K3 · ReAct

Best process score49.52%

GPT-5.5 · ReAct

Best direct-answer accuracy6.37%

GPT-5.5 / Gemini 3.5 Flash

Explore the paper results

16 models · 26 evaluated configurations · Pass@1

RankModel / backboneSettingAccuracy (%)Process Score (%)
Source: paper Tables 3–4 · Higher is better

Overall scores are micro-averages over 204 tasks. Only evaluated model–setting combinations are shown.

See the paper’s model comparison figure
The role of visual access

What changes when
agents can inspect images?

The paper compares four settings to examine the contribution of retrieved visual evidence.

Same tasks. Different access.

All four settings retain the task’s initial images.

  1. Direct Answer

    Question and initial images, without external tools.

  2. Text

    ReAct with text web search and page reading.

  3. Caption

    Retrieved images are represented by VLM-generated descriptions.

  4. Full

    ReAct can inspect retrieved images and crop visual details.

Scores over all 204 tasks (%). Gains are percentage points, computed before rounding in paper Table 5.

Explore tool use and efficiency in the paper

The paper analyzes when agents use visual tools and how performance relates to tool calls and runtime.

Inside the search process

Finding evidence is one thing.
Using it is another.

17 appendix case studies show successful searches and the points where visual context breaks down.

Same task · Different outcomes

The correct makerspace: found, then lost.

In task 126, one run connects the building to its step-free entrance. Another finds the right makerspace, then selects a different entity.

Successful caseFIGURE S3

Preserve the connection.

The run carries the building identity through historical records, makerspace information, and entrance verification.

GPT-5.5 · ReAct
Failure caseFIGURE S6

Lose the right intermediate entity.

The run encounters the correct makerspace and conflicting evidence, but its final response chooses the wrong entity.

Qwen3.6-35B-A3B · DeerFlow

A qualitative comparison of two reported runs. The backbones and frameworks differ; this is not a controlled framework comparison.

Read all 17 case studies

Selected steps and outcomes reproduced from the paper.

Evaluate and build

From a benchmark
to your next experiment.

Download the English release, inspect the reference evidence, and evaluate complete answers alongside partial progress.

Acc.

Complete-answer accuracy

Does the complete answer satisfy the task requirements? All and only the required content must be present, under task-specific matching rules.

PS

Process Score

How correct and complete are the individual results and subquestion answers? Scoring follows the rules provided for each task.

The English release

Ready to explore
on Hugging Face.

204 tasks in the test split, with initial images, reference answers, reference solutions, and English evaluation prompts.

Open the dataset ↗

17 tasks start without an image and require visual evidence found during search.

Python · datasets
from datasets import load_dataset

dataset = load_dataset(
    "antins-labs/VisSearch-Bench-v1",
    split="test",
)

task = dataset[0]
print(task["question_en"])
images = task["images"]
reference_answer = task["answer"]
What is included in each task?
FieldContent
idStable task identifier.
category_name / domainSearch family and primary content domain.
question_enComplete English question and output requirements.
imagesOrdered initial input images; an empty list for text-initiated tasks.
answerReference answer.
gold_trajectoryReference solution with traceable evidence.
accuracy_promptEnglish prompt for complete-answer grading.
Process ScoreEnglish prompt for task-specific partial-credit grading.

The two grading fields contain prompt text, not numerical model scores. The reference solution is separate from the model trajectories illustrated in the paper.

A few useful details.

What makes these tasks visually grounded?

Critical decisions depend on visual evidence: object appearance, local details, candidate comparisons, or image structure. Construction checks are designed to reject tasks that can bypass these requirements through text-only shortcuts.

Why do some tasks have no initial image?

They begin with a text instruction that defines what to search. The agent still needs to find and inspect images during the search. There are 17 such tasks in the 204-task release.

How are the evaluated settings different?

Direct Answer uses the question and initial images without external tools. ReAct and DeerFlow are agent frameworks evaluated with selected model backbones. The paper reports only the model–setting combinations actually evaluated. See the paper for each framework’s tools and budgets.

Read the paper

VisSearch-Bench

Evaluating Long Horizon Visual Search in Multimodal Agents

Suggested citation · manuscript
@misc{wu2026vissearchbench,
  title = {VisSearch-Bench: Evaluating Long Horizon Visual Search
           in Multimodal Agents},
  author = {Wu, ZhengXian and Nie, Qianhao and Chen, Zhuohong
            and Gao, Junjie and Zhang, Yuyao and Yu, Yunyao
            and Xu, Hangrui and Cheng, Cheng and Ma, Guiyu
            and Zhang, Jin and Xu, Xingzhong and Yang, Kai
            and Wang, Haoqian},
  year = {2026},
  howpublished = {Manuscript},
  url = {https://huggingface.co/datasets/antins-labs/VisSearch-Bench-v1}
}
BENCHMARK TASK

Scroll to inspect the figure. Press Escape to close.