Microsoft worked with Zerotech, the No. 1 pocket drone company in China, to develop a smart pocket drone that can collect real-time information, monitor its working status, and perform facial recognition.

Customer profile

Zerotech is a top provider of intelligent drones and smart drone turnkey solutions. Headquartered in China, it is the first partner to build a drone development platform on a Qualcomm SOC. Zerotech produces Dobby, the first and most popular pocket drone in the Chinese market. It also is a solution provider—Dobby is not only a product but an original design manufacturer (ODM) solution platform that is adopted by other companies.

Problem statement

Zerotech invests a lot of money in research and development (R&D), but it still lacks the capability to further empower its drone and compete with other players in the industry. Zerotech has a first-rate flight-control system and power-management solution, but it also has big gaps in IoT device data processing, virtualization, predictive maintenance, and further data analysis.

Solution

This solution will focus on making the Zerotech Dobby pocket drone smart and fun based on Azure back-end services. The proposed solution includes:

  • Integrating Microsoft Cognitive Services into a pocket drone control app on Android/iOS to realize selfie age/emotion recognition.
  • Collecting drone flight status and sensor data through Azure IoT Hub and processing it through Azure Stream Analytics into Azure Storage/SQL Database and Microsoft Power BI.
  • Monitoring drone flight status and presenting real data within a portal based on Power BI.

In this smart drone solution, Zerotech will use the following Microsoft technologies:

  • Azure IoT Hub
  • Azure Stream Analytics
  • Azure Storage
  • Azure SQL Database
  • Power BI
  • Cognitive Services

Architecture

The architecture for the Zerotech smart pocket drone solution can be represented as follows:

  • The smart drone will send all sensors’ data to the smart phone as controller by Wi-Fi connection.
  • The smart phone drone-control app will send sensor data to Azure IoT Hub every 5 seconds.
  • Stream Analytics will transfer this data into Azure SQL Database, then Power BI will synch these table updates into the Zerotech drone flight-control dashboard.
  • Zerotech integrates Azure Cognitive Services into the drone-control app; when end users take a selfie, they can click the picture to show age/emotion and do fortune-telling.

Figure 1. Zerotech smart pocket drone solution architecture

Zerotech smart pocket drone solution Architecture

Device used and code artifacts

The Zerotech Dobby drone spec is the following:

  • Chipset: Qualcomm Snapdragon 801
  • Memory: 2 GB
  • Camera: 12 MB
  • OS: Linaro Android

Microsoft’s China DX technical evangelist team and the Zerotech developer team split the engagement into four segments:

  • Drone app IoT SDK integration
  • Azure IoT Hub, Stream Analytics, and Azure SQL Database deployment
  • Data visualization in Power BI
  • Azure Cognitive Services integration

Drone app IoT SDK integration

Zerotech provides Android and iOS apps for controlling the drone. In this integration, Microsoft’s China DX technical evangelist team worked with the Zerotech dev team on the Android platform as a pilot. We coded on the IoT Hub Android SDK integration, collected data from the drone device, and sent it to an IoT hub.

The following is an Android app code sample:

package com.zerotech.cameratime.flight;

import android.app.Dialog;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.TextView;

import com.afollestad.materialdialogs.MaterialDialog;
import com.microsoft.azure.iothub.DeviceClient;
import com.microsoft.azure.iothub.IotHubClientProtocol;
import com.microsoft.azure.iothub.IotHubEventCallback;
import com.microsoft.azure.iothub.IotHubStatusCode;
import com.microsoft.azure.iothub.Message;
import com.zerotech.cameratime.MyApplication;
import com.zerotech.cameratime.R;
import com.zerotech.cameratime.activity.CameraActivity;
import com.zerotech.cameratime.bean.EventCenter;
import com.zerotech.cameratime.bean.FlyControllerEntity;
import com.zerotech.cameratime.bean.ZOWarningMdel;
import com.zerotech.cameratime.constants.uav.FlyCmdConstant;
import com.zerotech.cameratime.constants.uav.ServerLinks;
import com.zerotech.cameratime.constants.uav.UavConstants;
import com.zerotech.cameratime.constants.uav.UserDataConstans;
import com.zerotech.cameratime.dialog.MaterialDialogBuilderL;
import com.zerotech.cameratime.dialog.TipsPop;
import com.zerotech.cameratime.manager.UserAndFlyData;
import com.zerotech.cameratime.net.MyCallback;
import com.zerotech.cameratime.net.MyOKHttpUtils;
import com.zerotech.cameratime.utils.CommonUtils;
import com.zerotech.cameratime.utils.LocationUtil;
import com.zerotech.cameratime.utils.LogUtil;
import com.zerotech.cameratime.utils.UavDataPaserUtil;
import com.zerotech.cameratime.utils.UiUtil;
import com.zerotech.cameratime.utils.UserUtils;
import com.zerotech.cameratime.utils.VibratorUtil;
import com.zerotech.cameratime.utils.WarningUtil;
import com.zerotech.cameratime.widgets.TakeoffLandDialog;

import org.greenrobot.eventbus.EventBus;
import org.greenrobot.eventbus.Subscribe;
import org.greenrobot.eventbus.ThreadMode;
import org.json.JSONException;
import org.json.JSONObject;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import okhttp3.Call;

/**
 * Created by chengbin on 2016/4/13.
 * 特殊飞行和起飞
 * modify:MrLiKH
 */
public class TakeOffAndFly {
  private FlyControllerEntity mFlyInfo;
  private int gpsNum;
  private float uavLng;
  private float uavLat;
  private int flyState;
  private int motorState;//电机状态
  private int flyTimeLong;            //飞行总时间 不关机状态下
  private int mStartFlyTime;          //飞行架次开始时间
  private int mFlyTimeHold;           //本架次飞行时间
  private int backState;//返航状态

  private MyApplication mMyApplication;
  private CameraActivity mCameraActivity;
  private Button mOneKeyTakeOffBtn;
  private ImageView mLandImageView;
  private ImageView mEmergencyHoverImageView;
  private int locationMode;
  private int batteryPercentage;//飞机电量百分比0--100%
  private int isEnableFly;
  private String appVersion = "";
  private String UserId = "";
  private int isOnGroud;

  private String mAzureIoTConnString = "{connection_string}";
  private DeviceClient mDeviceClient;

  

  public TakeOffAndFly(MyApplication myApplication, CameraActivity cameraActivity,
                       Button oneKeyTakeOff, ImageView land, ImageView emergencyHover) {
    EventBus.getDefault().register(this);
    appVersion = CommonUtils.getVersionName(cameraActivity);
    UserId = UserUtils.getUserID();
    mMyApplication = myApplication;
    mCameraActivity = cameraActivity;
    mOneKeyTakeOffBtn = oneKeyTakeOff;
    mLandImageView = land;
    mEmergencyHoverImageView = emergencyHover;
  }

  public void destroy() {
    EventBus.getDefault().unregister(this);
    MyOKHttpUtils.getIntance().cancelTag(this);
  }

  

  private void initAZureIotDeviceClient() {
    IotHubClientProtocol protocol = IotHubClientProtocol.HTTPS;
    try {
      mDeviceClient = new DeviceClient(mAzureIoTConnString, protocol);
      mDeviceClient.open();
    } catch(IOException e1) {
      LogUtil.e("Exception while opening IoTHub connection: " + e1.toString());
    } catch(Exception e2) {
      LogUtil.e("Exception while opening IoTHub connection: " + e2.toString());
    }
  }


  private void uploadFlyInfoWhenConnUAV() {
    Map<String, String> params = new HashMap<>();
    String uploadData = uploadFlyInfoToWeb();
    params.put("data", uploadData);
    MyOKHttpUtils.getIntance().oKHttpPost(ServerLinks.CONN_UAV_4G_UPLOAD, this, params, new MyCallback() {
      @Override
      public void onError(Call call, Exception e, int id) {
      }

      @Override
      public void onResponse(String response, int id) {

      }
    });

    try
    {
      String uploadFlyInfo2Iot = uploadFlyInfoToIot();
      if(uploadFlyInfo2Iot != null) {
        Message msg = new Message(uploadFlyInfo2Iot);
        msg.setProperty("DobbyFlyState", Integer.toString(flyState));

        LogUtil.d("sending message to IoTHub : " + uploadFlyInfo2Iot);

        EventCallback eventCallback = new EventCallback();
        mDeviceClient.sendEventAsync(msg, eventCallback, flyState);
      }
    } catch (Exception e) {
      LogUtil.e("Exception while sending message to IoTHub : " + e.toString());
    }
  }

  protected static class EventCallback implements IotHubEventCallback {
    public void execute(IotHubStatusCode status, Object context){
      Integer flyState = (Integer) context;
      System.out.println("IoT Hub responded to message "+ flyState.toString()
        + " with status " + status.name());
    }
  }

  

  private String uploadFlyInfoToIot() {
    if(mFlyInfo == null)
      return null;

    JSONObject jsonobj = new JSONObject();
    try {
      jsonobj.put("year", mFlyInfo.year); // 年
      jsonobj.put("month", mFlyInfo.month); // 月
      jsonobj.put("day", mFlyInfo.day); // 日
      jsonobj.put("hour", mFlyInfo.hour); // 时
      jsonobj.put("minute", mFlyInfo.minute); // 分
      jsonobj.put("second", mFlyInfo.second); // 秒
      jsonobj.put("curlongitude", mFlyInfo.uavLng); // 当前经度
      jsonobj.put("curlatitude", mFlyInfo.uavLat); //当前纬度
      jsonobj.put("dstlongitude", mFlyInfo.dstLng); //目标经度
      jsonobj.put("dstlatitude", mFlyInfo); // 目标纬度
      jsonobj.put("gpsnumber", mFlyInfo.gpsNumber); // GPS星数
      jsonobj.put("speedgpsx", mFlyInfo.speedGpsX); // gps velx
      jsonobj.put("speedgpsy", mFlyInfo.speedGpsY); // gps vely
      jsonobj.put("height", mFlyInfo.height); // 无人机离地高度
      jsonobj.put("flyControlVoltage", mFlyInfo.flyControlVoltage); // 飞控电压
      jsonobj.put("batteryPercentage", mFlyInfo.batteryPercentage); // 电量百分比
      jsonobj.put("temperature", mFlyInfo.temperature); // 温度
      jsonobj.put("shakingCoefficient", mFlyInfo.shakingCoefficient); // 晃动系数
      jsonobj.put("shockCoefficient", mFlyInfo.shockCoefficient); // 震动系数
      jsonobj.put("flyState", flyState); // 飞行状态:/ 0:飞行中, 1:地面, 2:起飞中, 3:降落中, 4:悬停中;
    } catch (JSONException e) {
      e.printStackTrace();
    } catch (Exception e) {
      e.printStackTrace();
    }

    return jsonobj.toString();
  }

}

Azure IoT Hub, Stream Analytics, and Azure Storage/SQL Database deployment

The Zerotech team deployed Azure IoT Hub, Stream Analytics, and Azure Storage on their own Azure subscription. Microsoft’s China DX technical evangelist team held a technical workshop with Zerotech to go through the main process, and then Zerotech created a free IoT Hub account for a POC. When the IoT Hub connection worked well, they deployed Stream Analytics and created Azure Storage/SQL Database to process all data collected from the drone device.

Figure 2. Azure IoT Hub configuration

Azure IoT Hub Configuration

Figure 3. Stream Analytics configuration

Stream Analytics configuration in Azure

Figure 4. Azure Storage information

Azure Storage information

Data visualization in Power BI

The Zerotech team attended a Microsoft Power BI workshop conducted by Microsoft’s China DX technical evangelist team. At this workshop they created a main dashboard to monitor all flying drones and analyze issues. An administrator can drill down into every drone’s status to look at details in a separate report. In the future, Zerotech will build this drone status report within Power BI Embedded into the drone-control app and allow end users to use it.

Figure 5. Smart drone dashboard in Power BI

smart drone dashboard in Power BI

Figure 6. Smart drone status in Power BI

smart drone status in Power BI

Azure Cognitive Services integration

To add more features into the smart drone, Zerotech integrated Azure Cognitive Services into the drone-control app. When end users take a selfie/photo by drone, they can click a button on the drone-control app that gives a result similar to how-old.net. Users get a view that guesses at age/emotion and even tells their fortune, which make the pocket drone more fun and interactive. Already, more than 10,000 drone users have uploaded about 100,000 photos calculated by Azure Cognitive Services.

Figure 7. Azure Cognitive Services integration

Azure Cognitive services integration

#import "NetManager.h"
#import "HttpTool.h"

@implementation NetManager

+(NSString *)FaceSubscriptionKey
{
    return @"4377a4ac43b1490c98ac809568c425c9";
}

+(NSString *)EmotionSubscriptionKey
{
    return @"3393e0594b0d42a89cb5e318d6e160d9";
}


+(NSString *)FACEBASEURL
{
    return @"https://api.cognitive.azure.cn/face/v1.0/";
}

+(NSString *)EMOTIONBASEURL
{
    return @"https://api.cognitive.azure.cn/emotion/v1.0/";
}

+(void)addKey:(NSString *)key value:(NSString *)value params:(NSMutableDictionary *)params
{
    if(value != nil || [value isEqualToString:@""])
    {
        [params setObject:value forKey:key];
    }
}


+(void)detectWithImageData:(NSData *)imageData
              returnFaceId:(NSString *)returnFaceId
       returnFaceLandmarks:(NSString *)returnFaceLandmarks
      returnFaceAttributes:(NSString *)returnFaceAttributes
                   completionBlock:(void (^)(id responseObject, NSError *error))completion
{
    NSString *url = [NSString stringWithFormat:@"%@detect",[NetManager FACEBASEURL]];
    
    NSMutableDictionary *urlParams = [[NSMutableDictionary alloc] init];
    [self addKey:@"returnFaceId" value:returnFaceId params:urlParams];
    [self addKey:@"returnFaceLandmarks" value:returnFaceLandmarks params:urlParams];
    [self addKey:@"returnFaceAttributes" value:returnFaceAttributes params:urlParams];
    
    NSMutableDictionary *headerParams = [[NSMutableDictionary alloc] init];
    [self addKey:@"Ocp-Apim-Subscription-Key" value:[NetManager FaceSubscriptionKey] params:headerParams];
    [self addKey:@"Content-Type" value:@"application/octet-stream" params:headerParams];
    
    
    [HttpTool post:url urlParams:urlParams params:nil headerParams:headerParams data:imageData completionBlock:^(id responseObject, NSError *error) {
        completion(responseObject,error);
    }];
    
}


+(void)detectWithImageURL:(NSString *)imageURL
             returnFaceId:(NSString *)returnFaceId
      returnFaceLandmarks:(NSString *)returnFaceLandmarks
     returnFaceAttributes:(NSString *)returnFaceAttributes
          completionBlock:(void (^)(id responseObject, NSError *error))completion
{
    NSString *url = [NSString stringWithFormat:@"%@detect",[NetManager FACEBASEURL]];
    
    NSMutableDictionary *urlParams = [[NSMutableDictionary alloc] init];

    [self addKey:@"returnFaceId" value:returnFaceId params:urlParams];
    [self addKey:@"returnFaceLandmarks" value:returnFaceLandmarks params:urlParams];
    [self addKey:@"returnFaceAttributes" value:returnFaceAttributes params:urlParams];
    
    NSMutableDictionary *params = [[NSMutableDictionary alloc] init];
    [self addKey:@"url" value:imageURL params:params];

    
    NSMutableDictionary *headerParams = [[NSMutableDictionary alloc] init];
    [self addKey:@"Ocp-Apim-Subscription-Key" value:[NetManager FaceSubscriptionKey] params:headerParams];
    [self addKey:@"Content-Type" value:@"application/json" params:headerParams];
    
    [HttpTool post:url urlParams:urlParams params:params headerParams:headerParams completionBlock:^(id responseObject, NSError *error) {
        completion(responseObject,error);
    }];

}


+(void)verifyWithFirstFaceId:(NSString *)faceId1
                     faceId2:(NSString *)faceId2
             completionBlock:(void (^)(id responseObject, NSError *error))completion
{
    NSString *url = [NSString stringWithFormat:@"%@verify",[NetManager FACEBASEURL]];
    
    NSMutableDictionary *params = [[NSMutableDictionary alloc] init];
    
    [self addKey:@"faceId1" value:faceId1 params:params];
    [self addKey:@"faceId2" value:faceId2 params:params];
    
    NSMutableDictionary *headerParams = [[NSMutableDictionary alloc] init];
    [self addKey:@"Ocp-Apim-Subscription-Key" value:[NetManager FaceSubscriptionKey] params:headerParams];
    [self addKey:@"Content-Type" value:@"application/json" params:headerParams];
    
    [HttpTool post:url urlParams:nil params:params headerParams:headerParams completionBlock:^(id responseObject, NSError *error) {
        completion(responseObject,error);

    }];
   
    
 
}

//Emotion API
+(void)recognizeWithImageData:(NSData *)imageData
               faceRectangles:(NSString *)faceRectangles
              completionBlock:(void (^)(id responseObject, NSError *error))completion
{
    NSString *url = [NSString stringWithFormat:@"%@recognize",[NetManager EMOTIONBASEURL]];
    
    NSMutableDictionary *urlParams = [[NSMutableDictionary alloc] init];
    [self addKey:@"faceRectangles" value:faceRectangles params:urlParams];

    NSMutableDictionary *headerParams = [[NSMutableDictionary alloc] init];
    [self addKey:@"Ocp-Apim-Subscription-Key" value:[NetManager EmotionSubscriptionKey] params:headerParams];
    [self addKey:@"Content-Type" value:@"application/octet-stream" params:headerParams];
    
    
    [HttpTool post:url urlParams:urlParams params:nil headerParams:headerParams data:imageData completionBlock:^(id responseObject, NSError *error) {
        completion(responseObject,error);
    }];
}

+(void)MediaListByType:(int)type startPos:(int)startPos
               success:(void (^)(id json))success failure:(void (^)(NSError *error))failure
{
    NSString *url = @"";
    if(type == 0)
    {
        url = [NSString stringWithFormat:@"%@/uav.cgi?op=select&type=pic&startPos=%i",kHTTPS,startPos];
    }
    else if(type == 1)
    {
        url = [NSString stringWithFormat:@"%@/uav.cgi?op=select&type=video&startPos=%i",kHTTPS,startPos];
    }
    
    NSMutableDictionary *dic = [[NSMutableDictionary alloc] init];
    
    // 发送请求
    [HttpTool get:url params:dic success:^(id json) {
        success(json);
    } failure:^(NSError *error) {
        failure(error);
    }];
}


+(void)DeleteMediaByTitle:(NSString *)title
                  success:(void (^)(id json))success failure:(void (^)(NSError *error))failure
{
    NSString *url = @"";
    if([title containsString:@".jpg"])
    {
        url = [NSString stringWithFormat:@"%@/uav.cgi?op=delete&type=pic&name=%@",kHTTPS,title];
    }
    else if([title containsString:@".mp4"])
    {
        url = [NSString stringWithFormat:@"%@/uav.cgi?op=delete&type=video&name=%@",kHTTPS,title];
    }
    
    NSMutableDictionary *dic = [[NSMutableDictionary alloc] init];
    
    // 发送请求
    [HttpTool get:url params:dic success:^(id json) {
        success(json);
    } failure:^(NSError *error) {
        failure(error);
    }];
}




+(void)DeleteMediaByTitleOld:(NSString *)title
                  success:(void (^)(id json))success failure:(void (^)(NSError *error))failure
{
    NSString *urlString = @"";
    if([title containsString:@".jpg"])
    {
        urlString = [NSString stringWithFormat:@"%@/uav.cgi?op=delete&type=pic&name=%@",kHTTPS,title];
    }
    else if([title containsString:@".mp4"])
    {
        urlString = [NSString stringWithFormat:@"%@/uav.cgi?op=delete&type=video&name=%@",kHTTPS,title];
    }
    

    NSURL *url = [NSURL URLWithString:urlString];
    
    NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url];
    
    [request setHTTPMethod:@"GET"];

    NSURLResponse *response = nil;
    NSError *error = nil;

    NSData *data = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
    if(!error)
    {
        NSString *str = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
        NSLog(@"%@",str);
    }
    else
    {
        NSLog(@"%@",error);
    }
    
}

Conclusion

“Azure IoT platform gave us powerful tools to build the end-to-end smart-drone solution in a short time. The services are very easy to use and very scalable. By using IoT Hub, we can collect all drones’ real-time data and monitor their flying status. That gives us insight into data analysis and predictive maintenance.

“Power BI makes data visualization easier—we don’t need to invest in report design and development any more. The Dobby fortune-telling feature that’s realized through Cognitive Services integration adds more fun to the current drone and attracts more customers.”

—Xiaolei Chen, Zerotech lab director

As a result of this technical engagement, Zerotech decided to migrate all back-end services into Azure. They will deploy Azure IoT integration in batches. There will be about 100,000 smart drones connected to Azure IoT Hub and about 150,000 drones using Azure Cognitive Services by July 2017.

Zerotech also plans to add Cortana support to the drone-control app in the coming months.

Figure 8. Zerotech hackfest team

Zerotech hackfest team

Figure 9. Zerotech smart drone deployment at Microsoft Exhibition Center

Zerotech smart drone deployment in Microsoft Exhibition Center