mrtk_developmentreleases/2.0.0releases/2.1.0releases/2.2.0releases/2.3.0releases/2.4.0releases/2.5.0releases/2.5.1releases/2.5.2releases/2.5.3
  • Features and Architecture
  • API Documentation

We've moved!

Starting from MRTK 2.6, we are publishing both conceptual docs and API references on docs.microsoft.com. For conceptual docs, please visit our new landing page. For API references, please visit the MRTK-Unity section of the dot net API explorer. Existing content will remain here but will not be updated further.

  • Features and Architecture
  • Feature Overviews
  • Input System
  • Eyes
  • Getting Started
Search Results for

    Show / Hide Table of Contents
    • Welcome to MRTK
      • Installation Guide
      • Configuration
        • Using the Unity Package Manager
        • MRTK configuration dialog
        • Getting started with MRTK and XR SDK
      • Updates and Deployment
        • Updating from earlier versions
        • Upgrading from HTK
        • Building and Deploying MRTK
      • Packages and Release Notes
        • Release Notes
        • MRTK Packages
      • Performance and Best Practices
        • Performance
        • Hologram Stabilization
        • Using MRTK in large projects
    • Architecture
      • Overview
      • Framework and Runtime
      • Input System
        • Terminology
        • Core System
        • Controllers, Pointers, and Focus
      • Systems, Extension Services and Data Providers
    • Feature Overviews
      • Boundary System
        • Boundary System Overview
        • Configuring the Boundary Visualization
      • Camera System
        • Camera System Overview
        • Camera Settings Providers
          • Windows Mixed Reality Camera Settings
          • Unity AR Camera Settings [Experimental]
          • Creating a camera settings provider
      • Cross Platform Support
        • Configure MRTK for iOS and Android
        • Configure MRTK for Leap Motion Hand Tracking
        • Configure MRTK for Oculus Quest
      • Detecting Platform Capabilities
      • Diagnostics System
        • Diagnostics System Overview
        • Configuring the Diagnostics System
        • Using the Visual Profiler
      • Extension Services
        • Extension Service Creation Wizard
        • Scene Transition Service Overview
        • Hand Physics Service Overview
      • Input System
        • Input Overview
        • Input Actions
        • Input Events
        • Input Providers
          • Input Providers Overview
          • Creating an input data provider
        • Controllers
        • Eyes
          • Overview
          • Getting Started
          • Access Data via Code
          • Validate Tracking Calibration
        • Gaze
        • Gestures
        • Hands
        • How to Add Near Interaction
        • In-Editor Input Simulation
        • Pointers
        • Voice Input
          • Dictation
          • Speech (Command and Control)
      • Multi Scene System
        • Multi Scene System Overview
        • Scene Types
        • Content Scene Loading
        • Monitoring Content Loading
        • Lighting Scene Operations
      • Packaging
        • MRTK Packages
        • MRTK Modularization
      • Profiles
        • Profiles Overview
        • Configuration Guide
      • Rendering
        • MRTK Standard Shader
        • Material Instance Overview
        • Hover Light Overview
        • Proximity Light Overview
        • Clipping Primitive Overview
      • Services
        • What makes a mixed reality feature
        • What are the MixedRealityServiceRegistry and IMixedRealityServiceRegistrar
        • Extension services
      • Spatial Awareness System
        • Spatial Awareness Overview
        • Spatial Observers
          • Configuring Observers for Device
          • Configuring Observers for Editor
          • Controlling Observers via Code
          • Creating a custom Observer
      • Teleport System Overview
      • Tools
        • Dependency Window
        • Extension Service Creation Wizard
        • Holographic Remoting
        • Input Animation Recording
          • Input Animation File Format Specification
        • Migration Window
        • Optimize Window
        • Runtime tools
          • Controller Mapping tool
          • InputFeatureUsage tool
      • UX Building Blocks
        • Toolbox Window
        • Button
        • Bounds Control
        • Object Manipulator
        • Constraint Manager
        • Slate
        • System Keyboard
        • Interactable
        • Solvers
          • Tap to Place
        • Object Collection
        • Scrolling Object Collection
        • Tooltips
        • Slider
        • Hand Menu
        • Near Menu
        • App Bar
        • Fingertip Visualization
        • Progress Indicator
        • Dialog [Experimental]
        • Hand Coach [Experimental]
        • Pulse Shader [Experimental]
        • Dock Control [Experimental]
        • HoloLens Keyboard Helpers [Experimental]
        • Rigged Hand Visualizer [Experimental]
        • Elastic System [Experimental]
        • Bounding Box [Obsolete]
        • Manipulation Handler [Obsolete]
      • Example Scenes
        • Examples Hub
        • Hand Interaction Example
        • Eye Tracking Interaction Example
    • Contributing
      • Contributing Overview
      • Coding Guidelines
      • Writing and Running Tests
      • Writing Documentation
      • Pull Requests
      • Experimental Features
      • Breaking Changes
      • How to use DocFX
    • Planning
      • Roadmap
    • Notice
    • Authors

    Getting started with eye tracking in MRTK

    This page covers how to set up your Unity MRTK scene to use eye tracking in your app. The following assumes you are starting out with a fresh new scene. Alternatively, you can check out our already configured MRTK eye tracking examples with tons of great examples that you can directly build on.

    Eye tracking requirements checklist

    For eye tracking to work correctly, the following requirements must be met. If you are new to eye tracking on HoloLens 2 and to how eye tracking is set up in MRTK, don't worry! We will go into detail on how to address each of them further below.

    1. An 'Eye Gaze Data Provider' must be added to the input system. This provides eye tracking data from the platform.
    2. The 'GazeInput' capability must be enabled in the application manifest. This capability can be set in Unity 2019, but in Unity 2018 and earlier this capability is only available in Visual Studio and through the MRTK build tool
    3. The HoloLens must be eye calibrated for the current user. Check out our sample for detecting whether a user is eye calibrated or not.

    A note on the GazeInput capability

    The MRTK-provided build tooling (i.e. Mixed Reality Toolkit -> Utilities -> Build Window) can automatically enable the GazeInput capability for you. In order to do this, you need to make sure that the 'Gaze Input Capability' is checked on the 'Appx Build Options' tab:

    MRTK Build Tools

    This tooling will find the AppX manifest after the Unity build is completed and manually add the GazeInput capability. Prior to Unity 2019, this tooling is NOT active when using Unity's built-in Build Window (i.e. File -> Build Settings).

    Prior to Unity 2019, when using Unity's build window, the capability will need to be manually added after the Unity build, as follows:

    1. Open your compiled Visual Studio project and then open the 'Package.appxmanifest' in your solution.
    2. Make sure to tick the 'GazeInput' checkbox under Capabilities. If you don't see a 'GazeInput' capability, check that your system meets the prerequisites for using MRTK (in particular the Windows SDK version).

    Please note: You only have to do this if you build into a new build folder. This means that if you had already built your Unity project and set up the appxmanifest before and now target the same folder again, you will not need to reapply your changes.

    Setting up eye tracking step-by-step

    Setting up the scene

    Set up the MixedRealityToolkit by simply clicking 'Mixed Reality Toolkit -> Configure…' in the menu bar.

    MRTK

    Setting up the MRTK profiles required for eye tracking

    After setting up your MRTK scene, you will be asked to choose a profile for MRTK. You can simply select DefaultMixedRealityToolkitConfigurationProfile and then select the 'Copy & Customize' option.

    MRTK

    Create an "eye gaze data provider"

    • Click on the 'Input' tab in your MRTK profile.
    • To edit the default one ( 'DefaultMixedRealityInputSystemProfile' ), click the 'Clone' button next to it. A 'Clone Profile' menu appears. Simply click on 'Clone' at the bottom of that menu.
    • Double click on your new input profile, expand 'Input Data Providers', and select '+ Add Data Provider'.
    • Create a new data provider:
      • Under Type select 'Microsoft.MixedReality.Toolkit.WindowsMixedReality.Input' -> 'WindowsMixedRealityEyeGazeDataProvider'
      • For Platform(s) select 'Windows Universal'.

    MRTK

    Simulating eye tracking in the Unity Editor

    You can simulate eye tracking input in the Unity Editor to ensure that events are correctly triggered before deploying the app to your HoloLens 2. The eye gaze signal is simulated by simply using the camera's location as eye gaze origin and the camera's forward vector as eye gaze direction. While this is great for initial testing, please note that it is not a good imitation for rapid eye movements. For this, it is better to ensure frequent tests of your eye-based interactions on the HoloLens 2.

    1. Enable simulated eye tracking:

      • Click on the 'Input' tab in your MRTK configuration profile.
      • From there, navigate to 'Input Data Providers' -> 'Input Simulation Service'.
      • Clone the 'DefaultMixedRealityInputSimpulationProfile' to make changes to it.
      • Check the 'Simulate Eye Position' checkbox.

      MRTK

    2. Disable default head gaze cursor: In general, it is recommended to avoid showing an eye gaze cursor or if absolutely required to make it very subtle. We do recommend to hide the default head gaze cursor that is attached to the MRTK gaze pointer profile by default.

      • Navigate to your MRTK configuration profile -> 'Input' -> 'Pointers'
      • Clone the 'DefaultMixedRealityInputPointerProfile' to make changes to it.
      • At the top of the 'Pointer Settings', you should assign an invisible cursor prefab to the 'GazeCursor'. You can do this by selecting the 'EyeGazeCursor' prefab from the MRTK Foundation.

    Enabling eye-based gaze in the gaze provider

    In HoloLens v1, head gaze was used as primary pointing technique. While head gaze is still available via the GazeProvider in MRTK which is attached to your Camera, you can check to use eye gaze instead by ticking the 'IsEyeTrackingEnabled' checkbox in the gaze settings of the input pointer profile.

    Note

    Developers can toggle between eye-based gaze and head-based gaze in code by changing the 'IsEyeTrackingEnabled' property of 'GazeProvider'.

    Important

    If any of the eye tracking requirements are not met, the application will automatically fall back to head-based gaze.

    Accessing eye gaze data

    Now that your scene is set up to use eye tracking, let's take a look at how to access it in your scripts: Accessing eye tracking data via EyeGazeProvider and eye-supported target selections.

    Testing your Unity app on a HoloLens 2

    Building your app with eye tracking should be similar to how you would compile other HoloLens 2 MRTK apps. Be sure that you have enabled the 'Gaze Input' capability as described above in the section A note on the GazeInput capability.

    Eye calibration

    Finally, please don't forget to run through the eye calibration on your HoloLens 2. The eye tracking system will not return any input if the user is not calibrated. Easiest way to get to the calibration is by flipping up the visor and back down. A system notification should appear welcoming you as a new user and asking you to go through the eye calibration. Alternatively you can find the eye calibration in the system settings: Settings > System > Calibration > Run eye calibration.

    Eye tracking permission

    When starting the app on your HoloLens 2 for the first time, a prompt should pop up asking the user for permission to use eye tracking. If it is not showing up, then that is usually an indication that the 'GazeInput' capability was not set.

    After the permission prompt showed up once, it will not show up automatically again. If you "denied eye tracking permission", you can reset this in Settings -> Privacy -> Apps.


    This should get you started with using eye tracking in your MRTK Unity app. Don't forget to check out our MRTK eye tracking tutorials and samples demonstrating how to use eye tracking input and conveniently providing scripts that you can reuse in your projects.


    Back to "Eye tracking in the MixedRealityToolkit"

    • Improve this Doc
    In This Article
    • Eye tracking requirements checklist
      • A note on the GazeInput capability
    • Setting up eye tracking step-by-step
      • Setting up the scene
      • Setting up the MRTK profiles required for eye tracking
      • Create an "eye gaze data provider"
      • Simulating eye tracking in the Unity Editor
      • Enabling eye-based gaze in the gaze provider
      • Accessing eye gaze data
      • Testing your Unity app on a HoloLens 2
        • Eye calibration
        • Eye tracking permission
    Back to top Generated by DocFX