On Dec 6–8, 2016, the Microsoft DX Audience Evangelism team, along with the Microsoft China DX Technical Evangelism team, delivered a Microsoft Azure App Service hackfest in the Microsoft Beijing office. Microsoft teamed up with Quantum Technologies, an Astro Reality company focused on teaching astronomy, to develop a backend server to host their services and create a management portal for operators to configure systems and update content.

Key technologies used

Microsoft Azure App Service, Azure File storage, Azure SQL Database, .NET Core

Core team

Special thanks to the Quantum Technologies team, the Microsoft DX Audience Evangelism team, and the Microsoft China DX Technical Evangelism team, which included the following participants:

  • Li Ming – CTO, Quantum Technologies
  • Zhang Chen – Backend Engineer, Quantum Technologies
  • Xing Xing – Mobile Engineer, Quantum Technologies
  • Rita Zhang – Principal Technical Evangelist, Microsoft US
  • Bhargav Nookala – Technical Evangelist, Microsoft US
  • Haishi Bai – Technical Assistant, Microsoft US
  • Tory Xu – Senior Technical Evangelist, Microsoft China
  • Yan Zhang – Audience Evangelism Manager, Microsoft China
  • Shijun Liu – Technical Evangelist, Microsoft China
  • Yuheng Ding – Technical Evangelist, Microsoft China


hackthon

Customer profile

Quantum Technologies is a startup focusing on providing the leading Astro Reality service across devices and products, as well as cloud-based Mixed Reality solutions for smart home, and interactive control operation solutions for a variety of devices. Their vision for the Astro Reality service is to help everyone learn more about astronomy. The CTO, as one of the founders, showed the Astro Reality prototype to Satya Nadella, and got Satya’s recommendation. Quantum’s vision is to “let everyone have their own planetary museum.”

To learn more about Quantum Technologies, visit the Quantum Technologies Home page.


Quantum Technologies website

Problem statement

The client of the Astro Reality system is a mobile app on iOS or Android that can add information on a phone screen by rendering a virtual planet on a physical ball. To support the expected user growth, Quantum Technologies needs a powerful and scalable backend to provide API service and content management. This backend:

  • Should be automatically scalable when large numbers of users use the product concurrently.
  • Should have the ability to be debugged remotely if it breaks down to ensure the stability of the system.
  • Could be quickly swapped between test and production environments to rapidly deploy and upgrade it.

Solution, steps, and delivery

To achieve these goals, we proposed to use the following:

  • .NET Core to develop all required services
  • Azure App Service to deploy the API server and management portal on the Mooncake (China Azure) Web App
  • Azure App Service to build the architecture of the backend server to make it more scalable
  • Azure File storage and Azure SQL Database to store data on a data layer


App Service Architecture Draft

Architecture

The Quantum Technologies backend architecture can be represented as follows:

  • A management portal and mobile client that use the gateway to call the different API servers
  • An OpenID that verifies the identification of the user by querying the data in SQL Database
  • Two applications that provide APIs: the Web API for Apps and the Management API
  • A static file server that provides users with the files and data in Azure File storage and SQL Database
  • A number of storage locations, including Azure File storage accounts and SQL Database


App Service Architecture Diagram

Code artifacts

The Microsoft China DX Technical Evangelism team and the Quantum Technologies developer team split the engagement into the following segments:

  1. Build API service to host management portal, Management API, and Web API
  2. Build File storage and SQL database

Build API service to host management portal, Management API, and Web API

Because our Web Apps support “built-in auto-scale and load balancing,” it makes our developing job easier and more focused.


App services


  1. Get the publish profile from Mooncake.


    App services


  2. Import it to Visual Studio project.


    App services


  3. Publish it to Mooncake.


    App services


Build File storage


File storage



Use this code:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Threading.Tasks;
    using Matrix.Api;
    using System.IO;
    using Microsoft.Extensions.Options;
    using Microsoft.WindowsAzure.Storage;
    using Microsoft.WindowsAzure.Storage.Blob;
    using Microsoft.AspNetCore.Http;
    using Microsoft.EntityFrameworkCore;

    namespace Matrix.Middleware.Upload
    {
        public class FileUploadService<T> : IFileUploadService<T> where T:class, IUploadFile
        {
            protected IUploadFileDbContext<T> database;
            protected readonly UploadOptions _options;
            protected IHttpContextAccessor contextAccessor;
            protected CloudStorageAccount storageAccount ;
            protected CloudBlobClient blobClient;
            protected CloudBlobContainer container;

            protected string Host;

            public bool UseExternalUrl
            {
                get
                {
                    return !_options.UseProxy;
                }
            }

            public FileUploadService(IUploadFileDbContext<T> database,UploadOptions options, IHttpContextAccessor contextAccessor) {
                this.database = database;
                _options = options;
                this.contextAccessor = contextAccessor;
                var containerName = _options.AzureContainer.ToLower();
                var connection = _options.AzureStorageConnection;
                storageAccount = CloudStorageAccount.Parse(connection);
                blobClient = storageAccount.CreateCloudBlobClient();
                container = blobClient.GetContainerReference(containerName);

                Host = contextAccessor.HttpContext.Request.Scheme + "://" + contextAccessor.HttpContext.Request.Host;
            }


            public Task<PagedResponse<T>> List(UploadFileSearchOption option)
            {
                int start = option.Start,
                    limit = option.Limit;
                string clientId = option.ClientId,
                      query = option.Query;

                if (start < 0)
                {
                    start = 0;
                }

                if (limit < 0)
                {
                    limit = 0;
                }

                IQueryable<T> set = database.UploadFiles
                    .Where(f => f.ClientId == clientId);

                if (string.IsNullOrEmpty(query) == false)
                {
                    query = query.ToLower();
                    set = set.Where(f => f.RawName.ToLower().Contains(query) || f.Path.Contains(query));
                }


                set = set.OrderByDescending(f => f.CreateTime);

                var response = set.CreatePagedResponse(option);

                foreach (var item in response.Items)
                {
                    SetUrl(item, true);
                }

                return Task<ItemResponse<T>>.FromResult(response);
            }

            public async Task<ApiResponse> Remove(string id,string clientId)
            {
                var response = new ApiResponse();
                var it = await database.UploadFiles.SingleOrDefaultAsync(i => i.Id == id);
                if (it == null)
                {
                    return response;
                }

                if (it.ClientId != clientId)
                {
                    response.Status = Status.NoPermission;
                    return response;
                }

                var fileRemoved = await RemoveFile(it);

                if (!fileRemoved)
                {
                    response.Status = Status.UnknownError;
                    response.Message = $"Failed to remove file ({it.RawName}) from azure";
                    return response;
                }

                var track = database.UploadFiles.Remove(it);
                try
                {
                    var effect = await track.Context.SaveChangesAsync();
                    if (effect == 1)
                    {
                        return response;
                    }
                    else
                    {
                        response.Status = Status.DatabaseCMDError;
                        response.Message = "SQL command success, but no row effect.";
                    }
                }
                catch (Exception ex)
                {
                    response.Status = Status.DatabaseCMDError;
                    response.Message = ex.Message;
                }
                
                return response;
            }

            public async Task<ItemResponse<T>> SaveStream(T file,Stream stream)
            {
                file.Id = Guid.NewGuid().ToString("N");
                await DoSaveStream(file, stream);
                SetUrl(file, true);
                var it = database.UploadFiles.Add(file);

                try
                {
                    await it.Context.SaveChangesAsync();
                    return new ItemResponse<T>()
                    {
                        Item = file
                    };
                }
                catch (Exception ex)
                {
                    return new ItemResponse<T>()
                    {
                        Status = Status.DatabaseCMDError,
                        Message = ex.Message
                    };
                }
            }


            private async Task<bool> RemoveFile(T file)
            {
                var containerName = _options.AzureContainer.ToLower();
                var connection = _options.AzureStorageConnection;
                var blobPath = GetBlobPath(file);

                var created = await container.CreateIfNotExistsAsync(BlobContainerPublicAccessType.Blob, new BlobRequestOptions(), new OperationContext());

                // Retrieve reference to a blob named "myblob".
                CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobPath);

                var result = await blockBlob.DeleteIfExistsAsync();
                return result;
            }

            private async Task DoSaveStream(T file, Stream stream)
            {
                var blobPath = GetBlobPath(file);

                var created = await container.CreateIfNotExistsAsync(BlobContainerPublicAccessType.Blob, new BlobRequestOptions(), new OperationContext());

                // Retrieve reference to a blob named "myblob".
                CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobPath);

                await blockBlob.UploadFromStreamAsync(stream);


                // Create or overwrite the "myblob" blob with contents from a local file.
                file.Path = blobPath;
            }

            private string GetBlobPath(T file)
            {
                var fileName = file.RawName;
                var clientId = file.ClientId.ToLower();
                var ext = Path.GetExtension(fileName).ToLower();
                fileName = file.Id + ext;
                var folder = fileName.Substring(0, 2);
                var folder2 = fileName.Substring(2, 2);
                var blobPath = clientId + "/" + folder +"/" + folder2 + "/" + fileName;
                blobPath = blobPath.ToLower();
                return blobPath;
            }

            private void SetUrl(T file,bool useProxy = false)
            {
                if (_options.UseProxy || useProxy)
                {
                    file.Url = Host + _options.DownloadPathFormat.Replace("{id}", file.Id);
                }
                else
                {
                    CloudBlockBlob blockBlob = container.GetBlockBlobReference(file.Path);
                    file.Url = blockBlob.Uri.ToString();
                }
            }

            public async Task GetStream(T file,Stream output)
            {
                var blobPath = GetBlobPath(file);

                // Retrieve reference to a blob named "myblob".
                CloudBlockBlob blockBlob = container.GetBlockBlobReference(blobPath);
                
                await blockBlob.DownloadToStreamAsync(output);
            }

            public async Task<T> Get(string id)
            {
                var file = await database.UploadFiles.SingleOrDefaultAsync(f => f.Id == id);
                SetUrl(file);
                return file;
            }
        }
    }


Build SQL database


SQL storage


Use this code:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Threading.Tasks;
    using Microsoft.AspNetCore.Builder;
    using Microsoft.AspNetCore.Hosting;
    using Microsoft.AspNetCore.Http;
    using Microsoft.Extensions.DependencyInjection;
    using Microsoft.Extensions.Logging;
    using IdentityModel.AspNetCore.OAuth2Introspection;
    using Microsoft.Extensions.Configuration;
    using Microsoft.EntityFrameworkCore;
    using Matrix.Middleware.Upload;
    using Matrix.Media.Data;
    using Microsoft.Extensions.DependencyInjection.Extensions;
    using Microsoft.Extensions.Options;

    namespace Matrix.Media

     {

      public class Startup
      {
          public Startup(IHostingEnvironment env)
          {
              var builder = new ConfigurationBuilder()
                  .SetBasePath(env.ContentRootPath)
                  .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                  .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
                  .AddEnvironmentVariables();
              Configuration = builder.Build();
          }

          public void ConfigureServices(IServiceCollection services)
          {
              services.AddOptions();
              var connString = Configuration.GetConnectionString("DefaultConnection");
              services.AddDbContext<MediaDbContext>(options => options.UseSqlServer(connString));

              var uploadOptions = new UploadOptions();
              Configuration.GetSection("UploadOptions").Bind(uploadOptions);
              uploadOptions.AzureStorageConnection = Configuration.GetConnectionString("AzureStorageConnection");
              services.AddSingleton(uploadOptions);

              services.TryAddSingleton<IHttpContextAccessor, HttpContextAccessor>();
              services.AddTransient<IUploadFileDbContext<UploadFile>, MediaDbContext>();
              services.AddTransient<IFileUploadService<UploadFile>, FileUploadService<UploadFile>>();

              services.AddCors(options =>
              {
                  // this defines a CORS policy called "default"
                  options.AddPolicy("default", policy =>
                  {
                      policy.AllowAnyOrigin()
                          .AllowAnyHeader()
                          .AllowAnyMethod();
                  });
              });

              services.AddMvc();
          }

          public IConfigurationRoot Configuration { get; }

          // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
          public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
          {
              loggerFactory.AddConsole();
              loggerFactory.AddDebug();
              loggerFactory.AddAzureWebAppDiagnostics(new Microsoft.Extensions.Logging.AzureAppServices.AzureAppServicesDiagnosticsSettings { OutputTemplate = "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level}] [{SourceContext}] {Message}{NewLine}{Exception}" });

              if (env.IsDevelopment())
              {
                  app.UseDeveloperExceptionPage();
              }
              else
              {
                  app.UseExceptionHandler("/Home/Error");
              }

              // this uses the policy called "default"
              app.UseCors("default");

              var oauth2Options = new IdentityServerAuthenticationOptions();
              var queryTokenRetriever = TokenRetrieval.FromQueryString();
              var headerTokenRetriever = TokenRetrieval.FromAuthorizationHeader();
              oauth2Options.TokenRetriever = request =>
              {
                  var token = headerTokenRetriever(request) ?? queryTokenRetriever(request);
                  return token;
              };
              Configuration.GetSection("OpenIdOptions").Bind(oauth2Options);

              app.UseIdentityServerAuthentication(oauth2Options);

              app.UseMvc(routes=> {
                  routes.MapRoute(
                      name: "default",
                      template: "{controller=Home}/{action=Index}/{id?}");
              });
          }
      }
  }


Conclusion

After three days of the hackfest, Quantum Technologies used Azure App Service to deploy the API server and management portal on Mooncake Web Apps, which gives other Astro Reality companies more choices when they try to build their own system. Azure App Service can be a very good choice due to its ease in developing and deploying facts and its auto-scaling benefits.

Customer feedback

“With the help of Microsoft, we built up the architecture of the backend, using App Service to make it more scalable.” — Chen Zhang, Chief Engineer from Quantum Technologies

Opportunities going forward

During the hackfest, we happened to introduce the Microsoft Bot framework to our customers. As in the Astro Reality system, voice and conversation have more advantages in interaction, so Quantum Technologies may try adopting Microsoft Bot technology and Cognitive Services for a more natural interaction between the user and the system in the future.

Additional resources