Centered Image

ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents

Jie-Jing Shao1*, Bo-Wen Zhang1,2*, Xiao-Wen Yang1,3*, Bai-Zhi Chen1, Si-Yu Han1,3, Jing-Hao Pang1,2, Wen-Da Wei1,3, Guohao Cai4, Zhenhua Dong4, Lan-Zhe Guo1,2†, Yu-Feng Li1,3†
1State Key Laboratory of Novel Software Technology, Nanjing University, China
2School of Intelligence Science and Technology, Nanjing University, China
3School of Artificial Intelligence, Nanjing University, China
4Noah's Ark Lab, Huawei, China
*Equal Contribution, †Corresponding Authors
arXiv Paper Poster Code Data Leaderboard

Overview of ChinaTravel. Given a query, language agents employ various tools to gather information and plan a multi-day multi-POI itinerary. The agents are expected to provide a feasible plan while satisfying the logical constraints and preference requirements. Crucially, ChinaTravel is designed to facilitate the shift from rigid slot-filling paradigms to open-ended natural language interaction, challenging agents to handle the diverse, compositional logic inherent in authentic human intent. To provide convenience for global researchers, we provide an English translation here.

Abstract

Travel planning stands out among real-world applications of Language Agents because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce ChinaTravel, with four key contributions: 1) a practical sandbox aligned with multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison, 3) an open-ended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis revealing the potential of neuro-symbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10× improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios.

IJCAI 2025 Travel Planning Challenge Review

Building on ChinaTravel, we organized the IJCAI 2025 Travel Planning Challenge to benchmark LLM agents on open-ended travel planning with compositional constraints. A co-first-authored review paper was accepted by Frontiers of Computer Science: A Review of the IJCAI 2025 Travel Planning Challenge: Bench-marking LLM Agents with Compositional Constraints .

The review complements the ChinaTravel benchmark by documenting the community challenge setting and its benchmarking perspective for travel-planning agents.

ChinaTravel Sandbox

Overview of ChinaTravel Sandbox Environment. Our sandbox incorporates travel information from 10 of the most popular cities in China, offering comprehensive information on attractions, accommodations, and restaurants essential for travel planning. Here is the visualization of information from Beijing and Nanjing.


Environment Constraints

Environmental constraints act as a feasibility metric, ensuring that the generated plans are both valid and effective. For example, POIs in the plan must exist in the designated city, transportation options must be viable, and time information must remain accurate. The following table summarizes the environmental constraints in ChinaTravel.



Domain-Specific Language

We design the ChinaTravel's Domain-Specific Language (DSL) to provide the automatic evaluation for logical constrains and preference optimization.

Benchmark Characteristics

ChinaTravel poses unprecedented challenges rooted in authentic human requests, distinguishing it from prior benchmarks through context-rich long-horizon planning, combinatorial constraint diversity, and open contextual grounding.

1. Context-Rich Long-Horizon Planning

Processing over 1,200 candidate POIs per query and requiring massive input/output token contexts, challenging standard text-based planning paradigms.


2. Diversity and Openness of Travel Requirements

TravelPlanner Constraints

ChinaTravel Constraints

Our compositional DSL supports scaling from 15 synthetic constraints to over 100 unique human constraints, capturing natural Zipf's law distributions of combinations not seen in previous benchmarks.

Neuro-Symbolic Planning Baseline

To tackle the multi-day, multi-POI challenges and open-ended verification, we present an interactive neuro-symbolic pipeline utilizing Reflexion for accurate DSL translation and symbolic depth-first search for constraint-aware itinerary generation.

Empirical Results

Open Challenges in ChinaTravel

Planning with Preferences

Beyond strict logical constraints, agents must optimize soft preferences (e.g., minimizing costs, maximizing attractions). We explore LLM-guided heuristic search (PEQ) vs symbolic optimization (PDS).

Ablation on Preference Ranking

Pareto Win Rates (Multi-Objective)



BibTeX

@inproceedings{shao2026chinatravel,
      title={ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents}, 
      author={Jie-Jing Shao and Bo-Wen Zhang and Xiao-Wen Yang and Bai-Zhi Chen and Si-Yu Han and Jing-Hao Pang and Wen-Da Wei and Guohao Cai and Zhenhua Dong and Lan-Zhe Guo and Yu-feng Li},
      booktitle={The Fourteenth International Conference on Learning Representations},
      year={2026},
      url={https://openreview.net/forum?id=0YRVlxY9BH}
}