Getting started with Eye Tracking in MRTK
This page covers how to set up your Unity MRTK scene to use Eye Tracking in your app. The following assumes you are starting out with a fresh new scene. Alternatively, you can check out our already configured MRTK Eye Tracking Examples.
Eye tracking requirements
For Eye Tracking to work correctly, the following requirements must be met:
- An 'Eye Gaze Data Provider' must be added to the input system. This provides eye tracking data from the platform.
- The GazeProvider must have its "Use Eye Tracking" property set to true. Note that true is the default value (so no special action is required unless you have actively unchecked this property.)
- The 'Gaze Input' capability must be enabled in the application manifest. Currently this is only available in Visual Studio.
- The HoloLens must be calibrated for the current user under system settings.
IMPORTANT: If any of the above requirements are not met, the application will automatically fall back to head-based gaze tracking.
Below are the steps required to enable eye tracking in an application.
Setting up the scene
Set up the MixedRealityToolkit by simply clicking 'Mixed Reality Toolkit -> Configure…' in the menu bar.
Setting up the MRTK profiles required for Eye Tracking
After setting up your MRTK scene, you will be asked to choose a profile for MRTK. You can simply select DefaultMixedRealityToolkitConfigurationProfile and then select the 'Copy & Customize' option.
Create an "Eye Gaze Data Provider"
Navigate to the 'Input System Profile' in your MRTK main profile.
To edit the default one ( 'DefaultMixedRealityInputSystemProfile' ), click the 'Clone' button next to it.
Double click on your new input profile and select '+ Add Data Provider'.
Create a new data provider:
Under Type select 'Microsoft.MixedReality.Toolkit.WindowsMixedReality.Input' -> 'WindowsMixedRealityEyeGazeDataProvider'
For Platform(s) select 'Windows Universal'.
Enabling Eye Tracking in the GazeProvider
In HoloLens v1, head gaze was used as primary pointing technique. While head gaze is still available via the GazeProvider in MRTK which is attached to your Camera, you can check to use eye gaze instead by ticking the 'Use Eye Tracking' checkbox as shown in the screenshot below.
NOTE: Developers can toggle between eye tracking and head tracking in code by changing the 'UseEyeTracking' property of 'GazeProvider'.
Simulating Eye Tracking in the Unity Editor
You can simulate Eye Tracking input in the Unity Editor to ensure that events are correctly triggered before deploying the app to your HoloLens 2. The eye gaze signal is simulated by simply using the camera's location as eye gaze origin and the camera's forward vector as eye gaze direction. While this is great for initial testing, please note that it is not a good imitation for rapid eye movements. For this, it is better to ensure frequent tests of your eye-based interactions on the HoloLens 2.
- Enable simulated Eye Tracking:
- Navigate to your main 'MRTK Configuration Profile' -> 'Input System Profile' -> 'Data Providers' -> 'Input Simulation Service'.
- Check the 'Simulate Eye Position' checkbox.
- Disable default head gaze cursor:
In general, it is recommended to avoid showing an eye gaze cursor or if absolutely required to make it very subtle.
Check out our eye gaze cursor tutorial for more information on how to best handle it.
We do recommend to hide the default head gaze cursor that is attached to the MRTK gaze pointer profile by default.
- Navigate to your main 'MRTK Configuration Profile' -> 'Input System Profile' -> 'PointerSettings.PointerProfile'
- At the bottom of the 'PointerProfile', you should assign an invisible cursor prefab to the 'GazeCursor'. If you downloaded the MRTK Examples folder, you can simply reference the included -'EyeGazeCursor'_ prefab.
Accessing eye gaze data
Now that your scene is set up to use Eye Tracking, let's take a look at how to access it in your scripts: Accessing Eye Tracking Data in your Unity Script.
Testing your Unity app on a HoloLens 2
Building your app with Eye Tracking should be similar to how you would compile other HoloLens 2 MRTK apps. The only difference is that the 'Gaze Input' capability is unfortunately not yet supported by Unity under 'Player Settings -> Publishing Settings -> Capabilities'. To use Eye Tracking on your HoloLens 2 device, you need to manually edit the package manifest that is part of your build Visual Studio project. Follow these steps:
- Build your Unity project as you would normally do for HoloLens 2.
- Open your compiled Visual Studio project and then open the 'Package.appxmanifest' in your solution.
- Make sure to tick the 'Gaze Input' checkbox under Capabilities.
IMPORTANT: Don't forget to calibrate HoloLens for the current user. Without calibration, eye tracking won't work. Calibration can be found under system settings.
If everything is set up correctly, a prompt should pop up asking the user for permission to use Eye Tracking when you start your Unity app on a HoloLens 2 device for the first time.