Language-Embedded 6D Pose Estimation
for Tool Manipulation

Yuyang Tu, Yunlong Wang, Hui Zhang, Wenkai Chen, Jianwei Zhang

University of Hamburg

IEEE Robotics and Automation Letters (RA-L), 2025

A tool can serve different tasks through different functional parts. We combine natural language instructions with a 3D point cloud to estimate the 6D pose of the part that matters for the task.

Language identifies a tool and functional part; a diffusion model combines its semantic embedding with a point cloud to estimate a task-relevant 6D pose. — From a language instruction to the pose of a functional part, and then to robot tool manipulation.

Core idea

1. Identify the part

Interpret the instruction to identify the tool, its function, and the part needed for the task.

2. Estimate its pose

Condition a diffusion-based pose estimator on language embeddings and the observed 3D point cloud.

3. Use the tool

Use the estimated functional-part pose as a reference for downstream manipulation.

Real-robot demonstrations

The same idea supports different task-relevant parts, including a mug handle and a wrench head.

Hanging a mug. The handle provides the task-relevant pose.

Locating a wrench. The wrench head defines the interaction pose.

Data and evaluation

The work introduces a synthetic dataset with annotated 6D poses of tool functional parts. Experiments evaluate category-level pose estimation and demonstrate task execution on a real robot. The dataset page provides access information and usage terms.

Publication

Yuyang Tu, Yunlong Wang, Hui Zhang, Wenkai Chen, and Jianwei Zhang. “Language-Embedded 6D Pose Estimation for Tool Manipulation.” IEEE Robotics and Automation Letters, 10(9), 8618–8625, 2025. doi:10.1109/LRA.2025.3587559.