Spatial awareness

Spatial Awareness

The Spatial Awareness system provides real-world environmental awareness in mixed reality applications. When introduced on Microsoft HoloLens, Spatial Awareness provided a collection of meshes, representing the geometry of the environment, which allowed for compelling interactions between holograms and the real-world.

Note

At this time, the Mixed Reality Toolkit does not ship with Scene Understanding algorithms as originally packaged in the HoloToolkit. Scene Understanding generally involves transforming Spatial Mesh data to create simplified and/or grouped Mesh data such as planes, walls, floors, ceilings, etc. Starting from version 2.6 Scene Understanding is available as a experimental spatial observer called WindowsSceneUnderstandingObserver. However to use WindowsSceneUnderstandingObserver a separate package is needed. Please see the Scene Understanding page for more details.

Getting started

Adding support for Spatial Awareness requires two key components of the Mixed Reality Toolkit: the Spatial Awareness system and a supported platform provider.

  1. Enable the Spatial Awareness system
  2. Register and configure one or more spatial observers to provide mesh data
  3. Build and deploy to a platform that supports Spatial Awareness

Enable the spatial awareness system

The Spatial Awareness system is managed by the MixedRealityToolkit object (or another service registrar component). Follow the steps below to enable or disable the Spatial Awareness system in the MixedRealityToolkit profile.

The Mixed Reality Toolkit ships with a few default pre-configured profiles. Some of these have the Spatial Awareness system enabled OR disabled by default. The intent of this pre-configuration, particularly for when disabled, is to avoid the visual overhead of calculating and rendering the meshes.

Profile System Enabled by Default
DefaultHoloLens1ConfigurationProfile (Assets/MRTK/SDK/Profiles/HoloLens1) False
DefaultHoloLens2ConfigurationProfile (Assets/MRTK/SDK/Profiles/HoloLens2) False
DefaultMixedRealityToolkitConfigurationProfile (Assets/MRTK/SDK/Profiles) True

  1. Select the MixedRealityToolkit object in the scene hierarchy to open in the Inspector Panel.

    MRTK Configured Scene Hierarchy

  2. Navigate to the Spatial Awareness System section and check Enable Spatial Awareness System

    Enable Spatial Awareness

  3. Select the desired Spatial Awareness system implementation type. The MixedRealitySpatialAwarenessSystem is the default provided.

    Select the Spatial Awareness System Implementation

Register observers

Services in the Mixed Reality Toolkit can have Data Provider services that supplement the main service with platform specific data and implementation controls. An example of this is the Mixed Reality Input System which has multiple data providers to get controller and other related input information from various platform-specific APIs.

The Spatial Awareness system is similar in that data providers supply the system with mesh data about the real-world. The Spatial Awareness profile must have at least one Spatial Observer registered. Spatial Observers are generally platform specific components that act as the provider for surfacing various types of mesh data from a platform specific endpoint (i.e HoloLens).

  1. Open or expand the Spatial Awareness System profile

    Spatial Awareness System Profile

  2. Click the "Add Spatial Observer" button

  3. Select the desired Spatial Observer implementation type

    Select the Spatial Observer Implementation

  4. Modify configuration properties on the observer as necessary

Note

Users of the DefaultMixedRealityToolkitConfigurationProfile (Assets/MRTK/SDK/Profiles) will have the Spatial Awareness system pre-configured for the Windows Mixed Reality platform which uses the WindowsMixedRealitySpatialMeshObserver class.

Build and deploy

Once the Spatial Awareness system is configured with the desired observer(s), the project can be built and deployed to the target platform.

Important

If targeting the Windows Mixed Reality platform (ex: HoloLens), it is important to ensure the Spatial Perception capability is enabled in order to use the Spatial Awareness system on device.

Warning

Some platforms, including Microsoft HoloLens, provide support for remote execution from within Unity. This feature enables rapid development and testing without requiring the build and deploy step. Be sure to do final acceptance testing using a built and deployed version of the application, running on the target hardware and platform.

Next steps

After following the procedures above to enable the Spatial Awareness system, the system can be configured and controlled in more detail.

Information for configuring observers in inspector:

Information for controlling and extending observers via code:

See also