Tactile Guidance of the TOMM Mobile Manipulator: Robot Skin vs Joystick

Overview

Can a robot covered in artificial skin be controlled by touching it, and is that better than a joystick? I studied this question during an Erasmus+ month at the Institute for Cognitive Systems (ICS), Technical University of Munich, in Prof. Gordon Cheng's team. We built a whole-body gesture interface on TOMM, ICS's dual-arm mobile manipulator, and compared it with a joystick in a within-subject study with 12 first-time users ^[1].

A participant pushing the skin-covered arms of TOMM with both hands, and the robot carrying a box between its hands A participant moves TOMM from Table 1 to Table 2 while it holds the box; rotating the base is a persistent touch on two separate skin areas.

System

Component	Specification
Robot	TOMM: two UR5 arms on a holonomic omnidirectional base, about 15 DoF in total
Robot skin	Large-area modular artificial skin on the arms; each cell senses proximity $p$ and force $f$
Contact intensity	$I = 0.7p + 0.3f$ , normalized to $[0,1]$ by the firmware; contacts clustered into centroids and wrenches
Rates	Skin at 100 Hz, control stack at 200 Hz
Controller	Admittance $M\ddot{\mathbf{x}} + D\dot{\mathbf{x}} + K\mathbf{x} = \mathbf{F}_{tactile}$ with $M = I_{6\times6}$ , $D = 2I_{6\times6}$ , $K = 0$
Bimanual coupling	A virtual box frame between the end-effectors; both hands follow it through fixed transforms and keep facing each other

Gesture vocabulary

Discrete events toggle controller states; sustained contacts become continuous wrenches that drive the admittance dynamics.

Gesture	Where	Effect
Double-tap	Shoulder	Toggle between base mode and manipulation mode
Double-tap	Forearm	Grasp / release
Sustained single contact	Any skin patch	Translation only (base: $F_x, F_y$ with $\tau_z = 0$ ; manipulation: box frame in X, Y, Z)
Sustained two contacts	Two separate patches	Base yaw only ( $F_x = F_y = 0$ )

Magnitude gating, persistence requirements, refractory periods and spatio-temporal double-tap windows reject accidental touches.

Joystick baseline

The joystick drove the same admittance controller and the same discrete logic, so only the interface changed. One difference matters: the joystick axes allow simultaneous translation and yaw of the base, whereas the tactile interface keeps them in mutually exclusive gestures.

Method

Participants: N = 12 (6 female, 6 male), age 28.6 ± 8.4 years, predominantly STEM backgrounds; 8 had prior robot experience.
Design: within-subject, counterbalanced order (joystick vs tactile), no separate training phase, verbal instructions before each session.
Task: drive to Table 1, grasp a box with both hands, carry it to Table 2 (about 180° of base reorientation) and release it.
Phases annotated from video: going to Table 1, picking up the box, going to Table 2, putting down the box, plus "Other" (recovery or idle).
Measures: phase durations and interaction errors; participant path length from two RealSense D435i cameras with MediaPipe pose and multi-view 3D fusion; system logs; NASA-TLX; 7-point questionnaires on control, predictability, comfort, trust and enjoyment; a post-experiment command-recall check.

Results

On first exposure the joystick was faster and less demanding. The tactile penalty concentrated in the pickup and transport phases.

Median duration of the two phases with the largest tactile penalty, and of the whole task. Interquartile ranges (joystick vs tactile): picking up 16.8–26.1 vs 22.7–61.9 s, going to Table 2 33.9–45.4 vs 53.3–83.7 s, total 81.6–105.7 vs 141.1–177.2 s. Total time: Wilcoxon W=0, p<.001 (Holm-corrected p=0.004), r_rb=1.00.

Participants walked a median of 44.6 m with touch against 15.5 m with the joystick (p_Holm = 0.007), moving around the robot to reach the right patch.

Two heatmaps of where participants stood: concentrated in one area for the joystick, spread around the robot for touch Cumulative human position during the task: centralized with the joystick (left), widely distributed with tactile guidance (right).

Median NASA-TLX subscale scores (0–100, higher is worse). Only mental demand (Holm-corrected p=0.037), physical demand (0.044) and frustration (0.016) remain significant after correction; temporal demand, performance and effort do not (0.178, 0.141, 0.178). IQRs (joystick vs tactile): mental 5.0–25.0 vs 20.0–56.2; physical 0.0–12.5 vs 5.0–55.0; temporal 0.0–21.2 vs 11.2–41.2; performance 0.0–12.5 vs 17.5–37.5; effort 7.5–21.2 vs 10.0–37.5; frustration 0.0–15.0 vs 13.8–51.2.

The effect sizes separate the large differences from the null ones: control, frustration, mental demand and the timing measures move strongly, while enjoyment and side switches barely change.

Magnitude of each tactile-vs-joystick difference, sorted. Rank-biserial r_rb is reported for Wilcoxon tests (total time, going to Table 2, physical demand, path length, predictability, pickup, temporal demand, errors, effort, trust, intuitiveness, side switches, enjoyment) and Cohen's d_z for paired t-tests (control, frustration, mental demand, performance, comfort, speed, distances); the two scales are not strictly comparable and signs are dropped. Only eight differences survive Holm correction (p<.05): total time, going to Table 2, pickup, path length, mental demand, physical demand, frustration and control/influence. Enjoyment and side switches show essentially no difference.

One result favors touch: commands anchored on the robot's body were remembered better.

Share of commands correctly recalled in the post-experiment check: mean 54% (joystick) vs 73% (tactile), median 50% vs 75%. Paired Wilcoxon W=37.5, p=0.049, r=0.67; this p-value is uncorrected, so treat it as a trend.

Note
A small expert benchmark (two experts, three trials per interface) gave medians of 52.37 s with the joystick and 61.00 s with touch. The gap is much smaller than for novices, which suggests that much of the penalty comes from procedural friction and state management rather than from touch itself.

Discussion

Design implications

Make the robot's state transparent: persistent visual or audio cues for the active mode.
Treat patch placement as a design variable: larger or duplicated key patches, and tolerant zones that map a near-miss to the intended command.
Integrate interaction primitives: separate continuous guidance from discrete commands more clearly, and support combined motions such as translation with rotation.

Warning
This is a pilot study: 12 technically experienced participants, no training phase (which favors the familiar joystick), a single task geometry, and an expert benchmark of only two people. The results are trends for first-exposure use, not a verdict on tactile interfaces.

Publications

PaperTactile Guidance for Complex Mobile Manipulation: Performance and Design ImplicationsAdnan Saood, Gordon ChengUnder review · 2026

References

Adnan Saood, Gordon Cheng. Tactile Guidance for Complex Mobile Manipulation: Performance and Design Implications. Under review, 2026.

Overview

System

Gesture vocabulary

Joystick baseline

Method

Results

Discussion

Design implications

Publications

Related

References