In this joint development effort, Microsoft teamed up with Plas.md, a Philadelphia startup, to help prototype an IoT solution for their exercise and physical therapy application, BioNautica. The key technologies used in this project are Universal Windows Platform (UWP), Unity 3D, Azure IoT Hub, Android Wear 2.0, and Microsoft Band 2.

Project team:

  • Vance Souders (@plasmdinc) – CEO, Plas.md
  • Steven Gussman – Engineer, Plas.md
  • Amanda Lange (@second_truth) – Technical Evangelist, Microsoft

Company profile

Plas.md is a startup specializing in health solutions. Plas.md leverages next-generation technologies such as virtual reality (VR), augmented reality (AR), wearable devices, and artificial intelligence to solve some of the hardest problems in healthcare and education. The company’s goal is to create targeted applications and experiences that enhance wellness, improve performance, and enable individuals across the world to attain the highest quality of life.

Business case

Plas.md is working with Magee Rehabilitation Hospital on this phase of the project. Magee is a medical rehabilitation hospital in Philadelphia specializing in physical therapy. The goal of BioNautica is to create a platform that motivates physical therapy patients to do physical exercise on a treadmill as part of their rehab.

As stated on the BioNautica project website (BioNautica.com), churn rate for physical therapy programs can be as high as 40 percent, with many patients quitting therapy before the completion of their prescribed treatments. Although patients may choose to prematurely exit treatment for several reasons, a mismatch in expectations regarding the duration of treatment and the speed at which improvement is likely to occur is often cited as a leading cause of dropout. This is especially important for patients recovering from myocardial infarction, where participation in cardiac rehabilitation programs is associated with a significant reduction in readmission and death.

Plas.md hypothesized that patients who have the enhanced feedback and motivation from the BioNautica application will be more willing to do their required walking therapies. This should show better results than control patients who do not do the BioNautica application therapy. Data stored in Azure Blob will enable Magee Rehabilitation to track the vital signs of patients. Historical patient fitness data can be visualized and compared to current data, enabling better future results.

Solution and steps

Amanda Lange from Microsoft began initial code review and early prototypes at the Plas.md startup working space in Camden, New Jersey. She drafted an interface that enabled this code to connect with Azure IoT Hub by using the Bluetooth functionality of the Microsoft Band. Plas.md was then able to deploy a similar solution using a native Java app on Android Wear 2.0.

Plas.md has now moved to WeWork Office in Philadelphia where they were able to demo the solution for Magee Rehabilitation. They also showcased the application in a public demonstration at the Franklin Institute in Philadelphia. Most of the coding on this project was done in Microsoft Visual Studio 2015.

Demo at Franklin Institute

Demo at Franklin Institute

Demo at Franklin Institute

Demo for Magee at WeWork Philly

Demo at WeWork in Philadelphia

BioNautica is an application designed in Unity 3D for Windows 10 UWP, which communicates with a wearable device using Bluetooth and Azure IoT Hub. The wearable devices used for the project are Microsoft Band 2 and Android Wear 2.0. This project was originally prototyped before the Microsoft Band SDK was discontinued, which is why the Microsoft Band was the first wearable used.

The project aims to gamify exercise and physical therapy with the aid of a 3D visualization. The solution involved an application to communicate with the Microsoft Band or Android Wear device, along with a Unity 3D application interface to present the data to the user and provide feedback, and IoT Hub to track and store the data for future processing using Azure Blob storage.

Android Wear 2.0 powers a wearable device and allows native Android apps to run in the interface, allowing direct communication to Azure.

Microsoft Band 2 is the fitness wearable for which the project was initially prototyped, using Bluetooth to communicate with Azure and Unity 3D to track heart rate and patient motion data.

Azure IoT Hub provides all the back-end and business-intelligence functions, including device registration, security, data ingestion, real-time analytics, storage, and display.

Unity 3D provides the interface for the end user of the application and does 3D visualization through Windows 10 UWP.

Architecture for Android Wear version

Architecture for Android Wear version

Architecture for Microsoft Band version

Architecture for Microsoft Band version

Technical delivery

Microsoft Band vs. Android Wear

Initially, the Microsoft Band was chosen for this project because of its wide array of sensors and because of the ease of support from Microsoft. Microsoft Band 2 was used in the first-round prototype. However, the project is now pivoting to Android Wear 2.0, which also has a wide range of sensors. Android Wear 2.0 allows for a native app to run in Java directly on the Android Wear device, which simplifies the architecture. Architecture for both the Microsoft Band and Android Wear versions of the prototype are provided.

Android Wear 2.0

A native Java app on an Android Wear 2.0 device collects data about the user heart rate. This is a push to pivot the full solution to Android Wear in time for more rigorous user testing.

Code excerpt: connection for Android with SAS token

public class MainActivity extends WearableActivity implements SensorEventListener, GoogleApiClient.ConnectionCallbacks, GoogleApiClient.OnConnectionFailedListener {
	
	String connString = "HostName=Bionautica.azure-devices.net;Deviceld=Note4;SharedAccessKey="...";
	public static final int MY_PERMISSIONS_REQUEST_BODY_SENSORS = 42;
	public static int messageCount = 0;
	
	private static final SimpleDateFormat AMBIENT_DATE_FORMAT = 
		new SimpleDateFormat("HH:mm", Locale.US);
	
	private BoxInsetLayout mContainerView; 
	private TextView mTextViewBPM;
	private TextView mClockView;
	private SensorManager mSensorManager; 
	private Sensor mHeartRateSensor; 
	private SensorEventListener sensorEventListener; 
	private DeviceC1ient client;

	private GoogleApiClient mApiClient; 

	@Override 
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_main);
		setAmbientEnabled();

		//Request permission to use biometric sensors and internet 
		requestPermission(new String[]{Manifest.permission.BODY_SENSORS, Manifest.permission.INTERNET},
				  MY_PERMISSIONS_REQUEST_BODY_SENSORS);

		//Connect to the Azure IoT hub and send test messages 
		try {
			InitAzureIoT();
		}
		catch(IOException e1)
		{
			System.out.println("Exception while opening IoTHub connection: " + e1.toString());
		}
		catch(Exception e2)
		{
			System.out.println("Exception while opening IoTHub connection: " + e2.toString());
		}
	}

Microsoft Band 2

For the prototype, Microsoft Band 2 was used. Microsoft Band pushes the following data to a UWP app using Bluetooth: heart rate, timestamp, and user motion (run, walk, still). Additionally, the light sensor collects ambient-light data for processing feedback about UV in the area, which is being tested as a possible secondary input for the gamification portion of the application.

Stream Analytics

Whether the user is wearing the Microsoft Band 2 or an Android Wear device, the collected data is pushed to Azure IoT Hub through Azure Stream Analytics.

Code excerpt: basic query in Stream Analytics

SELECT deviceid,heartrate,time

INTO blobOutput

FROM banddata
Timestamp by EventEnqueuedUtcTime

Azure Blob storage

Azure Blob storage enables holding of the data (heart rate, timestamp, and user motion) for processing and further review. The data is stored in CSV format.

Unity visualization

A component was written to connect Unity 3D and Azure storage for real-time ingestion of data into the Unity dashboard, which provides up-to-date feedback for the patient who is using the workout application. The Unity 3D world is a custom-developed environment that provides a virtual racetrack for the user. The wearable device tracks the user motion (walking, running, standing still) and heart rate. The path contains markers for feedback that change based on user heart rate. This gives the user real-time feedback about workout progress.

The UWP application runs on a Windows 10 PC or tablet device.

Code excerpt: UWP from Unity

void Awake(){ 
	DontDestroyOnLoad(transform.gameObject);
}

// Use this for initialization
void Start () {

	UnityEngine.WSA.Application.InvokeOnUIThread(() => { p.InitializeBand(); }, true);

}

// Update is called once per frame 
void Update () {

	//Get the current HR 
	UnityEngine.WSA.Application.InvokeOnUIThread(() => { hr = p.GetHR(); }, true);

}

Security

Security is a big concern for applications that use customer medical data. Azure is trusted to provide compliance for data storage. It is critical that any data solution adheres to the Health Insurance Portability and Accountability Act (HIPAA). We built an implementation that uses Azure services that are in scope for HIPAA and the HITECH Act through the Microsoft Business Associate Agreement. For more information, read HIPAA and the HITECH Act in the Microsoft Trust Center.

Conclusion

“Bionautica Trails would not have been possible without the support of Microsoft technologies and platforms. They have enabled Plas.md to innovate faster and with a smaller team than would have been possible otherwise.” —Vance Souders, Plas.md

Azure IoT Hub enables data to flow to the visualization while tracking it in storage for future processing. Plas.md is interested in the possibilities of putting the data gathered in Azure IoT Hub through Azure Machine Learning for greater processing. The trial on a first round of Magee patients is beginning January 2017. The data will also be exposed in the future to an EMR (emergency medical records) system, using a HIPAA-compliant implementation.

After the initial patient trials, Plas.md plans to add more simulated levels (maps) to the experience to provide new worlds for the runners. The Android Watch version of the BioNautica experience is also not production ready and will require additional engineering this year prior to a final rollout. Plas.md is adding an additional engineer to this aspect of the project and will be rolling out the Android components to production as the next priority for patients. When these components are finalized and the security audits are complete, the BioNautica experience will be production ready.

Additional resources

Azure portal showing resources

Azure portal showing resources

Deploying from Visual Studio

Deploying from Visual Studio