Creating a spatial awareness system data provider
The Spatial Awareness system is an extensible system for providing applications with data about real world environments. To add support for a new hardware platform or a new form of Spatial Awareness data, a custom data provider may be required.
This article describes how to create custom data providers, also called Spatial Observers, for the Spatial Awareness system. The example code shown here is from the SpatialObjectMeshObserver
class implementation which is useful for loading 3D mesh data in-editor.
Note
The complete source code used in this example can be found in the Assets/MRTK/Providers/ObjectMeshObserver
folder.
Namespace and folder structure
Data providers can be distributed in one of two ways:
- Third party add-ons
- Part of the Microsoft Mixed Reality Toolkit
The approval process for submissions of new data providers to the MRTK will vary on a case-by-case basis and will be communicated at the time of the initial proposal. Proposals can be submitted by creating a new Feature Request type issue.
Third party add-on
Namespace
Data providers are required to have a namespace to mitigate potential name collisions. It is recommended that the namespace includes the following components.
- Company name producing the add-on
- Feature area
For example, a Spatial Awareness data provider created and shipped by the Contoso company may be "Contoso.MixedReality.Toolkit.SpatialAwareness".
Folder structure
It is recommended that the source code for data providers be layed out in a folder hierarchy as shown in the following image.
Where the ContosoSpatialAwareness folder contains the implementation of the data provider, the Editor folder contains the inspector (and any other Unity editor specific code), and the Profiles folder contains one or more pre-made profile scriptable objects.
MRTK submission
Namespace
If a spatial awareness system data provider is being submitted to the Mixed Reality Toolkit repository, the namespace must begin with Microsoft.MixedReality.Toolkit (ex: Microsoft.MixedReality.Toolkit.SpatialObjectMeshObserver)
and the code should be located in a folder beneath MRTK/Providers (ex: MRTK/Providers/ObjectMeshObserver).
Folder structure
All code should be located in a folder beneath MRTK/Providers (ex: MRTK/Providers/ObjectMeshObserver).
Define the spatial data object
The first step in creating a Spatial Awareness data provider is determining the type of data (ex: meshes or planes) it will provide to applications.
All spatial data objects must implement the IMixedRealitySpatialAwarenessObject
interface.
The Mixed Reality Toolkit foundation provides the following spatial objects that can be used or extended in new data providers.
Implement the data provider
Specify interface and/or base class inheritance
All Spatial Awareness data providers must implement the IMixedRealitySpatialAwarenessObserver
interface, which specifies the minimum functionality required by the Spatial Awareness system. The MRTK foundation includes the BaseSpatialObserver
class which provides a default implementation of this required functionality.
public class SpatialObjectMeshObserver :
BaseSpatialObserver,
IMixedRealitySpatialAwarenessMeshObserver,
IMixedRealityCapabilityCheck
{ }
Note
The IMixedRealityCapabilityCheck
interface is used by the SpatialObjectMeshObserver
class to indicate that it provides support for the SpatialAwarenessMesh capability.
Apply the MixedRealityDataProvider attribute
A key step in creating a Spatial Awareness data provider is to apply the MixedRealityDataProvider
attribute to the class. This step enables setting the default profile and platform(s) for the data provider, when selected in the Spatial Awareness profile as well as Name, folder path, and more.
[MixedRealityDataProvider(
typeof(IMixedRealitySpatialAwarenessSystem),
SupportedPlatforms.WindowsEditor | SupportedPlatforms.MacEditor | SupportedPlatforms.LinuxEditor,
"Spatial Object Mesh Observer",
"ObjectMeshObserver/Profiles/DefaultObjectMeshObserverProfile.asset",
"MixedRealityToolkit.Providers")]
public class SpatialObjectMeshObserver :
BaseSpatialObserver,
IMixedRealitySpatialAwarenessMeshObserver,
IMixedRealityCapabilityCheck
{ }
Implement the IMixedRealityDataProvider methods
Once the class has been defined, the next step is to provide the implementation of the IMixedRealityDataProvider
interface.
Note
The BaseSpatialObserver
class, via the BaseService
class, provides only an empty implementations for IMixedRealityDataProvider
methods. The details of these methods are generally data provider specific.
The methods that should be implemented by the data provider are:
Destroy()
Disable()
Enable()
Initialize()
Reset()
Update()
Implement the data provider logic
The next step is to add the logic of the data provider by implementing the specific data provider interface, for
example IMixedRealitySpatialAwarenessMeshObserver
. This portion of the data provider will typically be platform specific.
Observation change notifications
To allow applications to respond to changes in the device's understanding of the environment, the data provider raises notification events as defined in the IMixedRealitySpatialAwarenessObservationtHandler<T>
interface.
OnObservationAdded()
OnObservationRemoved()
OnObservationUpdated()
The following code from the SpatialObjectMeshObserver
examples demonstrates raising and event when mesh data is added.
// The data to be sent when mesh observation events occur.
// This member variable is initialized as part of the Initialize() method.
private MixedRealitySpatialAwarenessEventData<SpatialAwarenessMeshObject> meshEventData = null;
/// <summary>
/// Sends the observations using the mesh data contained within the configured 3D model.
/// </summary>
private void SendMeshObjects()
{
if (!sendObservations) { return; }
if (spatialMeshObject != null)
{
MeshFilter[] meshFilters = spatialMeshObject.GetComponentsInChildren<MeshFilter>();
for (int i = 0; i < meshFilters.Length; i++)
{
SpatialAwarenessMeshObject meshObject = SpatialAwarenessMeshObject.Create(
meshFilters[i].sharedMesh,
MeshPhysicsLayer,
$"Spatial Object Mesh {currentMeshId}",
currentMeshId,
ObservedObjectParent);
meshObject.GameObject.transform.localPosition = meshFilters[i].transform.position;
meshObject.GameObject.transform.localRotation = meshFilters[i].transform.rotation;
ApplyMeshMaterial(meshObject);
meshes.Add(currentMeshId, meshObject);
// Initialize the meshEventData variable with data for the added event.
meshEventData.Initialize(this, currentMeshId, meshObject);
// Raise the event via the spatial awareness system.
SpatialAwarenessSystem?.HandleEvent(meshEventData, OnMeshAdded);
currentMeshId++;
}
}
sendObservations = false;
}
Note
The SpatialObjectMeshObserver
class does not raise OnObservationUpdated
events since the 3D model is only loaded once. The implementation in the WindowsMixedRealitySpatialMeshObserver
class provides an example of raising an OnObservationUpdated
event for an observed mesh.
Add Unity Profiler instrumentation
Performance is critical in mixed reality applications. Every component adds some amount of overhead for which applications must account. To this end, it is important that all spatial awareness data providers contain Unity Profiler instrumentation in inner loop and frequently utilized code paths.
It is recommended to implement the pattern utilized by the MRTK when instrumenting custom providers.
private static readonly ProfilerMarker UpdateObserverPerfMarker = new ProfilerMarker("[MRTK] WindowsMixedRealitySpatialMeshObserver.UpdateObserver");
/// <summary>
/// Requests updates from the surface observer.
/// </summary>
private void UpdateObserver()
{
using (UpdateObserverPerfMarker.Auto())
{
// Code to be measured.
}
}
Note
The name used to identify the profiler marker is arbitrary. The MRTK uses the following pattern.
"[product] className.methodName - optional note"
It is recommended that custom data providers follow a similar pattern to help simplify identification of specific components and methods when analyzing traces.
Create the profile and inspector
In the Mixed Reality Toolkit, data providers are configured using profiles.
Define the profile
Profile contents should mirror the accessible properties of the data provider (ex: update interval). All of the user configurable properties defined in each interface should be contained with the profile.
Base classes are encouraged if a new data provider extends an existing provider. For example, the SpatialObjectMeshObserverProfile
extends the MixedRealitySpatialAwarenessMeshObserverProfile
to enable customers to provide a 3D model to be used as the environment data.
[CreateAssetMenu(
menuName = "Mixed Reality Toolkit/Profiles/Spatial Object Mesh Observer Profile",
fileName = "SpatialObjectMeshObserverProfile",
order = 100)]
public class SpatialObjectMeshObserverProfile : MixedRealitySpatialAwarenessMeshObserverProfile
{
[SerializeField]
[Tooltip("The model containing the desired mesh data.")]
private GameObject spatialMeshObject = null;
/// <summary>
/// The model containing the desired mesh data.
/// </summary>
public GameObject SpatialMeshObject => spatialMeshObject;
}
The CreateAssetMenu
attribute can be applied to the profile class to enable customers to create a profile instance using the Create > Assets > Mixed Reality Toolkit > Profiles menu.
Implement the inspector
Profile inspectors are the user interface for configuring and viewing profile contents. Each profile inspector should extend the BaseMixedRealityToolkitConfigurationProfileInspector
class.
The CustomEditor
attribute informs Unity the type of asset to which the inspector applies.
[CustomEditor(typeof(SpatialObjectMeshObserverProfile))]
public class SpatialObjectMeshObserverProfileInspector : BaseMixedRealityToolkitConfigurationProfileInspector
{ }
Create assembly definition(s)
The Mixed Reality Toolkit uses assembly definition (.asmdef) files to specify dependencies between components as well as to assist Unity in reducing compilation time.
It is recommended that assembly definition files are created for all data providers and their editor components.
Using the folder structure in the earlier example, there would be two .asmdef files for the ContosoSpatialAwareness data provider.
The first assembly definition is for the data provider. For this example, it will be called ContosoSpatialAwareness and will be located in the example's ContosoSpatialAwareness folder. This assembly definition must specify a dependency on Microsoft.MixedReality.Toolkit and any other assemblies upon which it depends.
The ContosoInputEditor assembly definition will specify the profile inspector and any editor specific code. This file must be located in the root folder of the editor code. In this example, the file will be located in the ContosoSpatialAwareness\Editor folder. This assembly definition will contain a reference to the ContosoSpatialAwareness assembly as well as:
- Microsoft.MixedReality.Toolkit
- Microsoft.MixedReality.Toolkit.Editor.Inspectors
- Microsoft.MixedReality.Toolkit.Editor.Utilities
Register the data provider
Once created, the data provider can be registered with the Spatial Awareness system to be used in the application.
Packaging and distribution
Data providers that are distributed as third party components have the specific details of packaging and distribution left to the preference of the developer. Likely, the most common solution will be to generate a .unitypackage and distribute via the Unity Asset Store.
If a data provider is submitted and accepted as a part of the Microsoft Mixed Reality Toolkit package, the Microsoft MRTK team will package and distribute it as part of the MRTK offerings.
See also
- Spatial awareness system
IMixedRealitySpatialAwarenessObject
interfaceBaseSpatialAwarenessObject
classSpatialAwarenessMeshObject
classSpatialAwarenessPlanarObject
classIMixedRealitySpatialAwarenessObserver
interfaceBaseSpatialObserver
classIMixedRealitySpatialAwarenessMeshObserver
interfaceIMixedRealityDataProvider
interfaceIMixedRealityCapabilityCheck
interface