Roomsy, a startup based in Vancouver, Canada, recently partnered with Microsoft in a hackfest to further understand how the Web Apps feature of Azure App Service and DevOps best practices better address their need to quickly make changes to both infrastructure and code when their clients need the support most. The proof of concept-oriented hackfest made use of the following services and practices:

  • Azure App Service
  • Azure Functions
  • Azure Storage
  • Infrastructure as code
  • Continuous integration

The hack took place over the course of 20 hours with the premise of migrating Roomsy’s offering to Azure, recreating their web portal, and enabling best practices for continuous integration. The hack team was composed of members from both Microsoft and Roomsy. It included:

  • Jaeyun Noh – Operations Lead and Product Owner, Roomsy
  • Sergii Baidachnyi – Senior Technical Evangelist, Microsoft
  • Jef King – Senior Technical Evangelist, Microsoft
  • Pierre Roman – Senior Technical Evangelist, Microsoft

Customer profile

Roomsy is a cloud-based property management system. It provides its customers with a platform to manage their rental properties and take reservations created through booking engines. Roomsy also provides visibility for its customers to online travel agencies such as Booking.com and Expedia.

Customer testimony

Architecture overview

Currently, Roomsy operates the http://pagodabox.io/ site in a cloud solution for hosting LAMP applications.

Roomsy’s solution contains three PHP applications and two MySQL databases that are located on the same server. There are also tasks that run once every 15 minutes. The task has to invoke PHP pages in order to invoke some sync mechanisms with external providers such as Booking.com or Expedia. The pages are not secured.

The list of the applications:

  • Channel Manager (cm.roomsy.com) plus two tasks (every 15 minutes)
  • Roomsy Property Management System/API (api.roomsy.com, pms.roomsy.com plus two tasks (hourly and daily)
  • Roomsy website CMS (pages.roomsy.com)

The applications use PHP 7.0, CodeIgniter 3.1 PHP framework, and ImageMagic extension with a MySQL back end.

MySQL connection strings (credentials) are stored in the PHP environment file.

MySQL databases are 800 MB only and they are growing slowly.

It’s possible to stop the applications for 10-30 minutes in order to update connection strings and finish migration tasks.

The Roomsy domain is configured outside pagodabox.io using a third-party domain registrant. Here are the configuration details:

Configuration details

The pagodabox.io cloud doesn’t support CNAME for custom domain names, so in order to configure the domain, you have to provide A records only.

Problem statement

The application enables management of different properties including hotels, motels, RV parking camps, and so on. Managers use the solution to order and book rooms, generate reports, and check guests in and out. As such, any delays are critical because the whole business shuts down. As a global solution, there isn’t a window of time where maintenance can be permitted to bring the system down.

We spent half the day establishing a value stream map (VSM) of the current delivery process, from conception to production. The VSM allowed Roomsy to realize that its current way of developing and deploying their solution leaves them vulnerable to significant issues, with little or no way to mitigate risks.

Creating the value stream map

While dev and staging environments did exist, they were not used in the current process.

The process can be described in two steps:

  1. Write code.
  2. Copy code to production servers.

While this seems to work today, this approach means there is no easy way to back out if a change breaks the production environment. There is no easy way to test new features without impacting the production environment and existing customers. And no way to ensure that availability is maintained because the entire solution currently runs on a single virtual machine.

Hackfest objectives

In order to create a prototype, we have to execute the following actions:

  • Web portal migration. Create three Azure web apps: configure PHP, deployment, slots and Virtual Network.
  • The database migration. Create a virtual machine based on Linux and set up RAID 0. The virtual machine should be deployed at the same Virtual Network.
  • Deploy and set up MySQL using the most recent snapshot of the database.
  • The tasks. Implement Azure Functions.
  • Implement a continuous integration model with the capacity to roll back any changes. It should be based on an infrastructure defined by templates and automation to ensure that it can be replicated easily.

Migration plan

Due to pagodabox.io limitations, we cannot make a smooth migration of all components at once to production. Instead, we have to divide the migration process into three steps:

  1. Create a working prototype in Azure. MySQL database should be reachable outside the local network.
  2. Make sure that MySQL in Azure contains up-to-date data and redirect pagodabox.io applications to it.
  3. Once Azure and pagodabox.io applications can work with MySQL in Azure, update DNS records.

Using this approach, we would affect users only during the MySQL migration phase, which should not take much time. We agreed that it’s possible to notify users in advance about maintenance work. In order to migrate the database, we would stop all services, create a hot copy of the database, and restore it in Azure. We could test this approach in advance to make sure we knew all possible issues and timing.

All components would be deployed using the new Azure portal in the same resource group. Potentially we could create a script for Azure Resource Manager, but in this case we wanted to educate the partner and show how to deploy all components step by step.

Continuous integration plan

The following processes were included:

  • As part of the Azure Migration proof of concept (POC), we would identify a continuous integration process that would automatically push the master branch of the code to a dev slot in the Azure Resource Group each time the branch is merged or updated.
  • The dev environment in Azure would be turned into a JSON template to facilitate the capabilities of automated deployment.
  • Implement an availability set of virtual machines deployed across fault domains and update domains. Availability sets will ensure that the Roomsy application is not affected by single points of failure, such as the network switch or the power unit of a rack of servers.

Solutions, steps, and delivery

Web portals migration

Creating a new portal is not challenging. Just make sure to use the same resource group for all components and service plans in the same region (and create a new group for the first portal). For example, it’s possible to use Central Canada as a region, but we decided to use West US due to customer locations:

Portal creation


For all portals, we could use just one App Service plan. Using the same plan allows us to share resources among all portals. In case we decide to use separate plans for each of the portals, we would be able to make changes through the portal.

In order to use all needed features, we have to select at least the Standard plan that supports by default things like SSL, deployment slots, and custom domains.

Once the portals are deployed, we have some actions to take prior to deployment.

One is to make sure that the right version of PHP is selected (7.0 in this case). This is done using the Application settings tab.


Application settings tab


Custom domain and SSL certificate features are available on the Custom domains and SSL tab. In this step we have to upload a certificate only. The domain name should be assigned right before making the solution publicly available. We don’t expect any problems there. We just have to make sure that the certificate and access to the domain settings are available for the migration.

Note: Uploading a .pfx certificate at this stage is required. In order to generate a .pfx based on .crt/.cer, it’s possible to use the openssl tool:

openssl pkcs12 -export -out domain.name.pfx -inkey domain.name.key -in domain.name.crt


At the same time, it’s important to provide a private key that was used for .crt/.cer generation. If the key is lost, the certificate would have to be reissued using a dashboard of the certificate provider.

Certificate dashboard

We will use additional deployment slots to deploy the portals from GitHub to Azure. We use this approach for testing purposes, not just for the migration but for future development as well. Once testing is done, we will be able to swap the slots. So, a new slot should be created for each of the portals. For more information about deployment slots, see Set up staging environments in Azure App Service.

Make sure that the slots are created using the same application plan as production so that the slots share resources with components in production. That’s why, if there are any load tests, the test slots should be moved to a separate application plan first.

Slot creation


If we were going to use FTP for any tasks, deployment credentials would have to be provided. But we don’t recommend doing this because the most important features are available through a KUDU panel (for example, https://roomsywebapp1.scm.azurewebsites.net/). Deployment source for the testing slot should be configured in order to use GitHub.

Note: GitHub integration should be activated for the testing slot only. The source should be deployed to the testing slot only. Once all testing activities are completed, it will be possible to switch the testing and production slots. We are not going to set up any deployment sources for production deployment.


Deployment sources


The applications can potentially have some settings, including a connection string. Using the portal, we can apply environment settings (Application settings), handler mappings, default documents, and a new root folder:

Application settings


All other PHP settings should be applied manually using standard PHP files. For more information about settings for PHP sites, see the following:

At the hackfest, we discovered that all settings would be provided using environment PHP variables. In this case, it’s possible to provide them using the App Settings tab, so it isn’t necessary to edit any files directly. Additionally, it’s possible to provide different settings for testing and production slots (just use the Slot Settings checkbox to freeze a record for any selected slot).

Virtual network

In order to establish connections between Azure Apps and virtual machines using local IP addresses, we will build a virtual network. It’s possible if we configure the network as Point to Site VPN, as shown here:

Point to Site VPN

To configure a virtual gateway and apply Point to Site settings automatically, it’s better to use the App Service interface to create a new one. That has to be done prior to creation of any virtual machines. Of course, it’s possible to reconfigure any existing virtual network but it requires PowerShell knowledge or access to the old portal. So, prior to creating any virtual machines, we will create at least one web application to host one of the web applications and we will be able to use the Networking tab to set up a new virtual network:

Networking tab


Using the interface that is embedded to App Service, we will create a network with a gateway. No additional configurations are needed. Just make sure to use this network for all app services and virtual machines. For more details about virtual networks, see Integrate your app with an Azure Virtual Network.

Network with gateway


The default address block will be created for all of the web applications we are going to create.

Note that it requires about 10-25 minutes to configure a virtual network. Don’t create any virtual machines before the virtual network is done.

Once the virtual network is created, it’s possible to create a virtual machine. In this step you can specify an already existing virtual network:

Creation of virtual machine


Once the virtual machine is created, it’s possible to connect it from Web Apps using a local address. This probably can’t be done by default because no traffic is allowed by default. Therefore, it’s necessary to modify network security group settings for the virtual machine:

Modify network security group settings


By adding new inbound rules, it’s possible to specify any custom port or select a service from the list. (MySQL was selected in our case.)

Once the port is open, it’s possible to connect the virtual machine from Web Apps using a local IP address.

A Kodu panel can be used to test connection status. In order to open the Kodu panel, we can open the Web Apps dashboard and Developer Tools->Advanced Tools menu item:


Test connection status


The Kodu tool allows us to run the console windows in the browser, but a standard ping command doesn’t work there. Instead, it’s possible to use tcpping.exe.

Using tcpping.exe

The database migration

Microsoft Azure already contains several ready-to-use images for different Linux operating systems. For example, we could use Ubuntu:

Using Ubuntu


Resource Manager will be used by default to deploy a virtual machine.

Using the wizard, we have to provide some basic parameters on the first step: Name, User Name, SSH public key, and Location. The location must be the same (Central Canada, for example) for all services. SSD allows us to get a better performance, which is critical for database solutions.

Provide parameters


In the second step, we have to select a service plan. In the case of database solutions, we have to select a service plan that supports at least two data disks. Using the disks, we are going to create a RAID 0 array to increase performance.

Once the plan is selected, we can provide network settings. Here we have to select a virtual network that we created earlier.

Additionally, we can provide two storage names. It’s better to assign more friendly names in this step.

Friendly storage name


Once the virtual machine is created, we can connect to it using Putty or any other SSH client. But before doing that, it’s better to create two data disks for our RAID array. Right after that we can connect the virtual machine and start implementing all configuration tasks.

In the next steps we have to create RAID and “copy” the existing database. In order to create the RAID array, we can refer to the following:

Additionally, we can use the mysqldump tool to get a current snapshot of the database, so that it doesn’t affect customers. For more information about the tool, see Migrating MySQL DB to Azure Linux CentOS VM using MySQLDump and SCP.

Before starting to create any databases, it’s a good idea to generalize the image. It will help us to recreate the virtual machine with all needed settings there. Before generalizing our virtual machine, we have to make sure that RAID 0 is configured, MySQL is installed, and MySQL uses RAID 0 disk to store all database files.

Note: MySQL stores all DB files on the system disk by default. Therefore, it’s important to change the configuration of MySQL. Follow the guidance here: Migrating MySQL database from OS disk to Data disk on Azure Linux VM

Finally, we can make a snapshot. The steps on capturing a snapshot can be found here: Step-by-Step: Capture a linux VM Image from a running VM

Once we have the snapshot, we can use the following procedure to redeploy our virtual machine based on the snapshot: Step-by-Step: Deploy a new Linux VM from a captured image

Note: We used only step 2 of this procedure because we already had all other network components.

For testing purposes, do not use the database in production. It’s better to create one more virtual machine. In this case, we can use the cheapest virtual machine without RAID and SSD—just for testing (not performance testing). Make sure the connection string for Testing slots refers to the testing DB.

Tasks

In order to execute the task, we can use Azure Functions, which can run in the same service plan. It’s still not clear how the PHP page is secured; however, we should not have any problems there. The script is not complex. For example, we can use something like this (C#):


using System;

public async static void Run(TimerInfo myTimer, TraceWriter log)
{
    log.Info($"C# Timer trigger function executed at: {DateTime.Now}");   
    HttpClient client=new HttpClient();
    await client.GetAsync("http://www.microsoft.com");
}

Backup tips

We used mysqldump to create the required backups. The script should create a local file. Once the file is created, it should be copied to Azure Blob storage. A cross-platform command-line tool can be used for this task. Azure Blob storage should be created (private blob).

Continuous integration

Roomsy uses a private GitHub repository where it stores the solution code. This provides Roomsy with an affordable way to ensure speed, data integrity, and support for distributed, non-linear workflows.

The master branch of the repo will be pushed to the Dev slot in the Roomsy Resource group. Please note that GitHub integration will be activated for the Dev slot only. Once all testing activities are completed, it will be swapped with the production slots. We are not going to set up any deployment sources for production deployment.

Deployment slots


GitHub deployment


Project and branch


Infrastructure as code

We split the infrastructure as code section into two sections:

  • Network configuration
  • Linux/MySQL back end

The network configuration was defined in a JSON template.

The first template is needed to create the following items in the Resource group:

  • Virtual Network.
  • Virtual Network interface card for the Linux host that will run MySQL.
  • Network Security Group that will hook incoming ports to SSH (TCP/22) and MySQL (TCP/3306).
  • A public IP address for the MySQL host with DNS name label to avoid hardcoding IP addresses.
  • A Point to Site VPN Gateway to allow the WeApp to connect to the back-end database.

First template


The second JSON template is to deploy a back-end Linux server with a proper RAID configuration to host the DB.

To perform that, we created the server manually, attached two empty 512 GB disks with no cache, configured the two-2 disk RAID 0 array, installed MySQL, configured MySQL to use the RAID array to store the data, and imported test data from Roomsy’s existing server.

Once the machine was complete and tested, we used the information included in the following posts to capture the machine and generate a JSON file that we modified for our needs:

The modifications done to the JSON files included changing the logon credentials from a password to a public certificate.

In the variable section of the JSON template, we replaced the following parts:

Replaced parts

We replaced them with:

New parts

And in the virtual machine resource definition, changed the “osprofile” to:

Osprofile change

General lessons

Some key points to consider:

  • MySQL migration requires some knowledge of operations rather than development. It’s important to understand how to increase performance using RAID and how to use data disks to move database files there.
  • Thanks to Azure Virtual Network and Point to Site settings (and a Virtual Gateway) there, it’s possible to use Web Apps and virtual machines in the same network to hide virtual machines disabling public IP addresses. It’s really useful in the case of databases.
  • It’s important to divide the migration process into several independent steps due to delays in domain records update. When migrating from third-party services, it’s not always possible to use CName. Therefore, it’s not always possible to use Traffic Manager and migration to production can be challenging.
  • It’s important to understand the sequence of steps during migration. For example, it’s better to create a Virtual Network using the Web Apps interface than the virtual machine interface. That way, it’s not necessary to reconfigure the network from Site to Site to Point to Site.
  • Automated testing needs to always be a top priority, from unit tests to integration and load tests. Being confident in the quality of the code should be a prerequisite in order to release it.
  • Automation of the deployment for the development, testing, and production environments provides certitude that standards are maintained and that each environment is true to design and will lower failure risks.

Conclusion

We could use both infrastructure as a service (IaaS) and platform as a service (PaaS) approaches to guarantee the best result. Migration to the Web Apps feature of Azure App Service is not challenging at all due to PHP 7 support and ability to use the staging environment that is integrated with GitHub. Creating the right virtual machine for MySQL is a little tricky but thanks to the generalization process, we could build an image with a proper setup. So, it’s really easy to redeploy the virtual machine.

The value stream mapping portion of the hack helped Roomsy see the big picture and understand where automation, proper processes, and DevOps practices can mitigate risks and allow for growth.

A lot of very interesting ideas on how to improve the process were discussed during this hackfest, some more doable than others. Most importantly, though, Roomsy realized the value of continuously improving and is committed and willing to put a lot of effort into this.