Gestures

Gestures are input events based on human hands. There are two types of devices that raise gesture input events in MRTK:

• Windows Mixed Reality devices such as HoloLens. This describes pinching motions ("Air Tap") and tap-and-hold gestures.

  For more information on HoloLens gestures see the Windows Mixed Reality Gestures documentation.

  WindowsMixedRealityDeviceManager wraps the Unity XR.WSA.Input.GestureRecognizer to consume Unity's gesture events from HoloLens devices.

• Touch screen devices.

  UnityTouchController wraps the Unity Touch class that supports physical touch screens.

Both of these input sources use the Gesture Settings profile to translate Unity's Touch and Gesture events respectively into MRTK's Input Actions. This profile can be found under the Input System Settings profile.

Gesture Events

Gesture events are received by implementing one of the gesture handler interfaces: IMixedRealityGestureHandler or IMixedRealityGestureHandler<TYPE> (see table of event handlers).

See Example Scene for an example implementation of a gesture event handler.

When implementing the generic version, the OnGestureCompleted and OnGestureUpdated events can receive typed data of the following types:

• Vector2 - 2D position gesture. Produced by touch screens to inform of their deltaPosition.
• Vector3 - 3D position gesture. Produced by HoloLens to inform of:
  • cumulativeDelta of a manipulation event
  • normalizedOffset of a navigation event
• Quaternion - 3D rotation gesture. Available to custom input sources but not currently produced by any of the existing ones.
• MixedRealityPose - Combined 3D position/rotation gesture. Available to custom input sources but not currently produced by any of the existing ones.

Order of events

There are two principal chains of events, depending on user input:

• "Hold":

  1. Hold tap:
     • start Manipulation
  2. Hold tap beyond HoldStartDuration:
     • start Hold
  3. Release tap:
     • complete Hold
     • complete Manipulation

• "Move":

  1. Hold tap:
     • start Manipulation
  2. Hold tap beyond HoldStartDuration:
     • start Hold
  3. Move hand beyond NavigationStartThreshold:
     • cancel Hold
     • start Navigation
  4. Release tap:
     • complete Manipulation
     • complete Navigation

Example Scene

The HandInteractionGestureEventsExample scene in MixedRealityToolkit.Examples\Demos\HandTracking\Scenes shows how to use the pointer Result to spawn an object at the hit location.

Gesture Tester script is an example implementation to visualize gesture events via GameObjects. The handler functions change the color of indicator objects and display the last recorded event in text objects in the scene.