Tactile Prediction and Real-Time Action Correction in World-Action Models
for Contact-Rich Manipulation
δΈζ Β· Installation Β· Training Β· Inference
TacPAC is a tactile world-action model for contact-rich robot manipulation. A world-action model can predict the contact that an action chunk is expected to produce, but that prediction is fixed before execution. TacPAC makes it actionable: while the chunk is running, a tactile expert reads each newly observed tactile image against the predicted contact and corrects the unexecuted part of the plan in real time.
The key is a reusable, layer-wise tactile-action KV cache. It stores the predicted tactile contact and the action representation conditioned on that prediction. Each correction is a single pass over this cache, so tactile feedback can update an active plan without regenerating the full action chunk.
The paper is available on arXiv. Datasets and checkpoints are being prepared for public release. This repository currently contains the model, training, preprocessing, deployment, and test code.
- Prediction-grounded correction. Current tactile feedback is interpreted against the contact anticipated by the plan, rather than as an isolated reactive signal.
- Asynchronous closed-loop execution. The base model plans once per chunk; new tactile frames update only the unexecuted suffix while the robot keeps moving.
- Efficient cache reuse. A tactile correction takes 30.4 ms (32.9 Hz) in our setup, which is 20.7Γ faster than regenerating a chunk.
- Real-robot results. Across five contact-rich tasks, TacPAC improves average success from 22% for the vision-only base model to 64%, outperforming the strongest evaluated baseline by 16 percentage points.
- Matched train/deploy tactile processing. Raw, frame-residual, and stress representations are shared by the dataloader and inference server, with modality-aware image resizing.
TacPAC is trained in two stages:
- Tactile-predictive base model. A video expert predicts future visual and tactile observations, while an action expert jointly denoises an action chunk through Mixture-of-Transformers attention.
- Tactile expert. The base model is frozen. For each planned chunk, TacPAC caches the clean tactile and action keys/values. Training samples execution offsets and supervises a delta action on the still-unexecuted suffix.
At inference time, predict_action() returns the planned chunk first,
prefill_tactile_cache() builds the reusable cache, and every correct_action() call applies one
new tactile observation to the active plan.
Each method was evaluated over 20 real-world trials per task. TacPAC reaches the best success rate on all five tasks and averages 64%. The component study below shows that tactile prediction and reactive correction are complementary; direct access to the predicted tactile cache provides the largest final gain.
| Variant | Tactile prediction | Online correction | Tactile cache | Plug | Fruit | Chip | Bottle | Card | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Vision only | β | β | β | 15 | 30 | 60 | 5 | 0 | 22 |
| Without tactile expert | β | β | β | 35 | 50 | 65 | 20 | 15 | 37 |
| Without tactile prediction | β | β | β | 40 | 25 | 45 | 30 | 25 | 33 |
| Without tactile cache | β | β | β | 40 | 50 | 75 | 40 | 30 | 47 |
| TacPAC | β | β | β | 80 | 65 | 90 | 40 | 45 | 64 |
The five tasks cover charger-plug insertion, multi-object fruit transfer, fragile potato-chip transfer, empty-bottle uprighting, and expansion-card insertion.
| Component | Location |
|---|---|
| TacPAC framework and cache/correction flow | starVLA/model/framework/WM4A/WanMoTJointTacExpert.py |
| Single-pass tactile expert | starVLA/model/modules/wan_mot/tactile_expert.py |
| World-action Mixture-of-Transformers | starVLA/model/modules/wan_mot/ |
| Tactile preprocessing shared by train and inference | starVLA/dataloader/vla/tactile_stress.py |
| LeRobot tactile data loading and execution offsets | starVLA/dataloader/vla/dataset/ |
| TacPAC training configuration | starVLA/config/training/vla/starvla_wam.yaml |
Stateful prepare / refine inference protocol |
deployment/model_server/server_infersystem.py |
| Unit and protocol tests | scripts/test/test_tactile_*.py, scripts/test/test_infersystem_*.py |
The code is tested with Python 3.10 and CUDA-capable PyTorch environments.
git clone git@github.com:LogosRoboticsGroup/TacPAC.git
cd TacPAC
conda create -n tacpac python=3.10 -y
conda activate tacpac
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
pip install -e .Download the Wan2.2 TI2V backbone and update the local model paths in
starVLA/config/training/vla/starvla_wam.yaml. In particular, check dit_path, vae_path,
text_encoder_path, and tokenizer_path before training.
TacPAC expects LeRobot-format demonstrations. A typical Flexiv sample exposes two RGB views and two tactile views:
observation.images.third_view
observation.images.left_wrist_view
observation.images.left_wrist_left_tactile
observation.images.left_wrist_right_tactile
Register local datasets and their video_keys in
starVLA/dataloader/vla/mixtures.py. Keys containing tactile are treated as tactile views, and
stage-2 training emits tactile_now / tactile_offset samples through
datasets.vla_data.tactile_offsets_per_sample.
The private datasets used in the paper are not stored in this repository. Until their release,
replace the example data_root entries with paths to your own LeRobot datasets.
Precompute the text embeddings used by both training stages:
python scripts/vla/precompute_text_embeds.py \
--config_yaml starVLA/config/training/vla/starvla_wam.yaml \
--datasets.vla_data.data_mix flexiv_plug_4views \
--datasets.vla_data.text_embedding_cache_dir data/text_embeds_cache/flexiv_plug_4viewsActivate the environment first, then launch the scripts from any directory. The default scripts
use eight processes; set NPROC_PER_NODE for another GPU count. Hydra-style overrides can be
appended to either command. Set TEXT_EMBEDDING_CACHE_DIR if the cache is not stored under the
default data/text_embeds_cache/<data_mix> path.
NPROC_PER_NODE=8 bash scripts/vla/train_WanMoTJoint.sh \
flexiv_plug_4views \
stage1NPROC_PER_NODE=8 bash scripts/vla/train_WanMoTJoint-TacExpert.sh \
flexiv_plug_4views \
results/Checkpoints/vla/<stage1-run>/final_model/pytorch_model.ptStage 2 freezes the base MoT and proprioceptive encoder, initializes the tactile expert from the action expert, and trains residual corrections against plans produced by the frozen stage-1 model.
Start the InferSystem server with a TacPAC checkpoint:
CKPT=results/Checkpoints/vla/<tacpac-run>/final_model/pytorch_model.pt \
PORT=5556 \
bash deployment/local_infer-wan-tac.shThe server exposes a stateful tactile protocol:
prepare: generate a new action chunk and establish aplan_id;refine: send the latest tactile views, execution offset, and executed action prefix to correct the remaining actions of that plan.
See deployment/model_server/README.md and
scripts/test/test_infersystem_stateful_tactile.py for the request schema and reference flow.
CPU unit tests cover tactile preprocessing, cache geometry, image fitting, and the stateful server protocol:
python -m unittest \
scripts.test.test_tactile_stress \
scripts.test.test_tactile_expert \
scripts.test.test_tactile_view_size \
scripts.test.test_infersystem_stateful_tactile \
scripts.test.test_infersystem_prefill_orderGPU end-to-end smoke tests are available in scripts/test/smoke_tac_expert_gpu.py and
scripts/test/smoke_stateful_tactile_server_gpu.py.
TacPAC is greatly inspired by the following outstanding contributions to the open-source community: StarVLA, FastWAM, T-Rex, and Dream-Tac.
If you find TacPAC useful, please cite our paper.
@article{ma2026tacpac,
title = {TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation},
author = {Ma, Zipei and Wei, Xiaofei and Jiang, Junzhe and Lu, Shunlin and Zhang, Li},
year = {2026},
journal = {arXiv preprint arXiv:2609.05266},
url = {https://arxiv.org/abs/2609.05266}
}Please also cite the upstream framework and backbone models used in your experiments.
This repository is released under the MIT License. Third-party models, datasets, robot drivers, and assets remain subject to their respective licenses.