As embodied intelligence continues to advance toward real-world physical interaction, tactile perception is emerging as another critical information modality alongside vision.

However, unlike the vision domain, which has already developed relatively unified data systems and evaluation paradigms, tactile learning remains highly fragmented. Different hardware platforms, tasks, and data protocols are often isolated from one another, making it difficult for models to develop stable and generalizable representations across scenarios and devices. This leaves a fundamental question unanswered: what kind of tactile representation can achieve generalization across tasks, scenarios, and embodiments?

At the current stage, building a universal benchmark that completely resolves all sensor and embodiment-specific differences remains unrealistic. Rather than attempting to directly create such a comprehensive benchmark, we explore a scalable and promising direction for future development: combining egocentric vision with full-hand tactile data, which represents one of the most promising technical pathways for embodied intelligence. This work serves as a first step toward this direction.

Based on this vision, we propose a unified evaluation framework for dexterous hand tactile perception:

This work is built around two core objectives: unified evaluation and unified representation. We construct the large-scale tactile benchmark HT-Bench and propose HandTouch, a vision-tactile unified encoding model, achieving systematic improvements in multi-task learning, multi-scenario adaptation, and out-of-distribution (OOD) generalization.

Why Is Tactile Learning Difficult to Generalize?

Current tactile learning faces three major structural bottlenecks:

(1) Highly fragmented data and sensors:
Different tactile sensing systems (finger arrays, full-hand tactile sensing, and soft tactile sensors) exhibit substantial variations in data structures and distributions, making it difficult for models to learn shared representation spaces.

(2) Disconnected task definitions:
Most existing approaches focus on individual tasks, such as grasp classification, contact prediction, or tactile reconstruction, while lacking unified evaluation protocols.

(3) Insufficient OOD (Out-of-Distribution) evaluation:
Models often perform well only on previously seen objects and scenarios, making it difficult to verify their true generalization capabilities in real-world settings.

Therefore, a fundamental gap remains: does there exist a unified tactile representation that can simultaneously support structural understanding, cross-modal alignment, and temporal prediction?

HT-Bench

“The First Unified Multi-Task and Multi-Scenario Benchmark for Dexterous Full-Hand Tactile Perception”

To address the lack of standardized evaluation protocols in tactile learning, we introduce HT-Bench, a large-scale multi-task benchmark designed for dexterous full-hand tactile perception.

The benchmark contains over 10 million RGB visual frames and 7.8 million tactile frames, covering diverse real-world robotic interaction scenarios. In addition, we design an Out-of-Distribution (OOD) test split based on scenario-level separation to evaluate model generalization to unseen tasks.

Rather than relying on a single evaluation metric, HT-Bench systematically evaluates tactile representation capabilities from four complementary perspectives:

• Structural retrieval
• Tactile completion
• Vision-to-tactile generation
• Temporal tactile prediction

These tasks correspond to different aspects of tactile intelligence, including structural consistency, local reconstruction capability, cross-modal reasoning ability, and dynamic interaction modeling, forming an evaluation framework closer to the requirements of real-world robotic manipulation.

HandTouch

“A Unified Tactile Representation Learning Framework”

Based on HT-Bench, we further propose HandTouch, a unified representation learning framework for dexterous hand tactile perception.

The core objective of HandTouch is to map continuous and heterogeneous tactile signals into a shared discrete representation space, enabling unified modeling across different devices and tasks.

The overall framework adopts a three-stage training strategy to progressively learn tactile representations.

Stage 1: Discrete tactile representation learning

In the first stage, the model leverages a Self-Attention Transformer and vector quantization mechanism to model tactile signals. Continuous tactile data are discretized into token representations, allowing the model to learn the spatial topology of tactile information and bring heterogeneous tactile sources into a unified representation space.

Stage 2: Vision-guided cross-modal alignment

In the second stage, visual information is introduced as a cross-modal condition. Through a masked reconstruction task, the model learns vision-guided tactile completion capabilities.

Visual features are injected into the tactile reconstruction process through a cross-attention mechanism, enabling the model to recover contact structures from incomplete tactile observations using visual semantic information, thereby improving cross-modal alignment.

Stage 3: Temporal tactile dynamics modeling

In the third stage, temporal modeling mechanisms are introduced, allowing the model to predict current and future tactile states based on historical visual and tactile sequences.

This extends tactile understanding from static perception to dynamic interaction modeling, enabling the learning of contact evolution processes.

Experimental Results: Validation of Cross-Task and OOD Generalization

We conduct a systematic evaluation of HandTouch on HT-Bench, comparing it with commonly used tactile encoding approaches, including CNN, ResNet, VAE, and ViT, which are widely adopted as tactile encoders in VTLA-based models.

Experimental results demonstrate that HandTouch achieves consistent improvements across multiple tasks. In the structural retrieval task, HandTouch improves Recall@5 from 74.65% to 85.23%, indicating stronger tactile structural representation capability. In the tactile completion task, the model reduces RMSE to 0.010 while improving the structural consistency metric cIoU to 0.911, outperforming all baseline methods. In the vision-to-tactile generation task, HandTouch maintains strong performance under Out-of-Distribution (OOD) scenarios, achieving a cIoU of 0.705, demonstrating robust cross-modal generalization capability.

Overall, HandTouch exhibits consistent advantages in structural understanding, cross-modal alignment, and OOD generalization capability.

From Task-Specific Learning to Unified Representation

The core significance of HandTouch lies not merely in improving performance on individual tasks, but in addressing a more fundamental question:

Does there exist a unified representation space for dexterous hand tactile perception that can be shared across tasks, scenarios, and hardware platforms?

To explore this question, HT-Bench provides a systematic evaluation benchmark, enabling different approaches to be compared under a unified dataset and task framework. Meanwhile, HandTouch explores a feasible pathway toward unified tactile representation learning through a three-stage learning paradigm, consisting of structural modeling, cross-modal alignment, and temporal prediction.

Specifically, the model maps heterogeneous tactile signals into a unified representation space through discrete representations, achieving alignment at the structural level. By introducing visual information as a cross-modal anchor, it enhances the capability to recover missing tactile information. Through temporal modeling, the framework further extends its understanding of dynamic interaction processes.

Overall, this work points toward a long-term goal: building a shared and transferable tactile representation space that provides dexterous hands with more robust physical perception capabilities.

In the future, this direction may be further extended to more complex dexterous manipulation tasks, longer-horizon real-world interactions, unified representations across diverse robotic embodiments, and deeper integration with VLA models and world models.

Original Paper Link

https://arxiv.org/pdf/2606.19161v1

Citation

If you use this work in research or engineering projects, please cite it as follows:

@article{huang2026htbench,
  author = {Yuzhe Huang and Jiaping Wu and Jiaming Jiang and Hezhe Lin and Aikebaier Aierken and Yunlong Wang and Kun Cheng and Ziyuan Jiao and Yuanxin Zhong},
  title = {HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision},
  year = {2026},
  howpublished = {arXiv preprint arXiv:2606.19161},
  note = {HandTouch: A vector-quantized vision-tactile encoder for unified tactile representation learning}
}