Logo of JYP Entertainment

Customer profile

JYP Entertainment, a leading entertainment company in South Korea, has broad experience in planning and creating content; managing celebrities; operating online, mobile, and international businesses; and finding new artists for more than 10 years. Founded by producer Jin-Young Park, JYP continues to discover and produce many talented artists, such as Wonder Girls, 2PM, 2AM, miss A, TWICE, and GOT7.

Problem areas to be addressed

JYP Entertainment has self-developed fansites for celebrities under its own management and is operating those sites using Microsoft Azure. The fansites have hundreds of thousands of registered users; JYP is working on more than 10 groups for artists they support.

JYP fansites

The JYP IT team consists of one project manager (PM) and two developers. To handle the demands of internal staff and fans, the IT team is committed to improve the features of operational websites and to develop new features if the need arises. Such demands are made more than two or three times per month.

Because JYP fansites are configured across more than 10 front-end web servers using Azure Virtual Machines, the time needed for testing and deploying developed products to those servers is difficult to estimate. Because of the characteristics of the entertainment business, many events can cause traffic to spike, flooding the sites unexpectedly. When service traffic is flooded, building additional servers and configuring servers needs to be done quickly. These tasks have either been performed by developers manually or resolved by preparing a number of additional servers before an event begins.

Version control for the source code of services was managed by the developers using Visual Studio Team Services. However, they had not configured continuous delivery and release management for applying correct configurations to both test and production servers, nor change-management tasks to be processed after any DevOps steps.

For these reason, JYP IT teamed with Microsoft Korea for a three-day hackfest. During that time, overall DevOps processes in JYP (including development, deployment, and monitoring) were analyzed and examined by using value stream mapping (VSM). The timing of the hackfest was as follows:

  • 1.5 days: DevOps basic hackFest
    • VSM
    • Technical contents and messages delivery
  • 0.5 day: Q&A, infra/code analytic
  • 1 day: integrate results to production

As a result, unnecessary and time-consuming steps caused by manual operations were found and classified as potential improvements. Processes in DevOps scenarios that JYP wanted to resolve during the hackfest were as follows:

  • Configuring environments for test and development equal to the production environment
  • Deploying developed product automatically to both the test environment and production environment and assuring their integrity
  • Implementing scale-out in the form of templates and configuring target environments that match operating environments when unexpected traffic floods occur

JYP fansite architecture

Participants

  • Chan Park – CIO (PM), JYP
  • Min Jae Park – Developer, JYP
  • Chan Moo Choi – Developer, JYP

  • Ye Jin Koo – Audience Evangelism Manager, Microsoft
  • Eun Ji Kim – Technical Evangelist, Microsoft
  • Seung Joo Baek – Technical Evangelist, Microsoft
  • Jung Hyun Choi – Azure MVP

Solution, steps, and delivery

In terms of development, organizing, and operating the fansites of JYP, the overall state examined through VSM looked like this:

JYP fansite VSM

The steps of implementing features, merging with the production environment, and building products were well-organized through Visual Studio and Team Services. However, to test products, developers needed to configure virtual machines (VM) manually to match the production environment. Thus, we automated the process by using an Azure Resource Manager template to automate generating VMs and configuring detail settings.

Previously, configuring a test environment took 2–3 hours per server; after applying the Resource Manager template, configuration time is reduced to about 10 minutes. In addition, the probability of reconstitution that is caused by discovering missing components during applying and testing products is reduced from 20 percent to zero.

The following Resource Manager template creates a front-end web server VM from a captured image.

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string"
        },
        "virtualMachineName": {
            "type": "string"
        },
        "virtualMachineSize": {
            "type": "string"
        },
        "adminUsername": {
            "type": "string"
        },
        "storageAccountName": {
            "type": "string"
        },
        "virtualNetworkName": {
            "type": "string"
        },
        "networkInterfaceName": {
            "type": "string"
        },
        "adminPassword": {
            "type": "securestring"
        },
        "availabilitySetName": {
            "type": "string"
        },
        "diagnosticsStorageAccountName": {
            "type": "string"
        },
        "diagnosticsStorageAccountId": {
            "type": "string"
        },
        "subnetName": {
            "type": "string"
        }
    },
    "variables": {
        "vnetId": "[resourceId('JYP-FANS-RG','Microsoft.Network/virtualNetworks', parameters('virtualNetworkName'))]",
        "subnetRef": "[concat(variables('vnetId'), '/subnets/', parameters('subnetName'))]"
    },
    "resources": [
        {
            "name": "[parameters('virtualMachineName')]",
            "type": "Microsoft.Compute/virtualMachines",
            "apiVersion": "2015-06-15",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[concat('Microsoft.Network/networkInterfaces/', parameters('networkInterfaceName'))]"
            ],
            "properties": {
                "osProfile": {
                    "computerName": "[parameters('virtualMachineName')]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]",
                    "windowsConfiguration": {
                        "provisionVmAgent": "true"
                    }
                },
                "hardwareProfile": {
                    "vmSize": "[parameters('virtualMachineSize')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "MicrosoftWindowsServer",
                        "offer": "WindowsServer",
                        "sku": "2016-Datacenter",
                        "version": "latest"
                    },
                    "osDisk": {
                        "name": "[parameters('virtualMachineName')]",
                        "vhd": {
                            "uri": "[concat(concat(reference(resourceId('jyp-fans-rg', 'Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2015-06-15').primaryEndpoints['blob'], 'vhds/'), parameters('virtualMachineName'), '-osdisk.vhd')]"
                        },
                        "createOption": "fromImage"
                    },
                    "dataDisks": []
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName'))]"
                        }
                    ]
                },
                "diagnosticsProfile": {
                    "bootDiagnostics": {
                        "enabled": true,
                        "storageUri": "[reference(resourceId('jyp-fans-rg', 'Microsoft.Storage/storageAccounts', parameters('diagnosticsStorageAccountName')), '2015-06-15').primaryEndpoints['blob']]"
                    }
                },
                "availabilitySet": {
                    "id": "[resourceId('Microsoft.Compute/availabilitySets', parameters('availabilitySetName'))]"
                }
            }
        },
        {
            "name": "[parameters('networkInterfaceName')]",
            "type": "Microsoft.Network/networkInterfaces",
            "apiVersion": "2016-09-01",
            "location": "[parameters('location')]",
            "dependsOn": [],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "ipconfig1",
                        "properties": {
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            },
                            "privateIPAllocationMethod": "Dynamic"
                        }
                    }
                ]
            }
        }
    ],
    "outputs": {
        "adminUsername": {
            "type": "string",
            "value": "[parameters('adminUsername')]"
        }
    }
}

Similar to the test environment, deploying developed results to the production environment also needed to be performed manually by developers through copy/paste operations. Moreover, the process was unable to set updating schedules, and in most cases developers had to connect each server directly (via Remote Desktop) to verify the configuration was set as intended. If operating servers were powered off during deployment, unnecessary steps such as turning on, applying product, and turning off the power were inevitable. During the whole test period, every test was performed after deploying the product, as well as configuring and verifying settings about deployment. These steps needed about 2 days; the whole test time was 20 hours.

Each product can be deployed and configured by using different technologies. The way that deployment and configuration is usually decided is to consider the operating system of an existing infrastructure or technology preferences of the client.

  • Chef/Puppet for Linux platforms
  • Windows PowerShell Desired State Configuration (DSC) for Windows platforms

Because the web services of JYP fansites are provided by using Internet Information Services (IIS) on Windows Server, parts of deployment and configuration that have technical dependencies and are about installing product in VMs are made to be processed by PowerShell DSC.

From the start of development to deployment to operational server, the steps and technologies applied to reduce process time and securing reliability are as follows:

  • Feature development/integration/build: Visual Studio Team Services
  • Implementation of Infrastructure as Code and configuration management
  • Continuous deployment
    • Build to Test deployment and configuration: PowerShell DSC
    • After test is completed, Build to Product deployment and configuration: PowerShell DSC
  • Test
    • Smoke-test in production environment

By using Team Services, products built by developers can be brought to source servers in the production environment. Configuration for roles is used to differentiate between test environments and production environments.

Full workflow

The following PowerShell script sets the DSC server.

Install-Module xPSDesiredStateConfiguration 

configuration DscPullServer
{ 
    param  
    ( 
        [string[]]$NodeName = 'localhost', 

        [ValidateNotNullOrEmpty()] 
        [string] $certificateThumbPrint,

        [Parameter(Mandatory)]
        [ValidateNotNullOrEmpty()]
        [string] $RegistrationKey 
    ) 

    Import-DSCResource -ModuleName xPSDesiredStateConfiguration
    Import-DSCResource –ModuleName PSDesiredStateConfiguration

    Node $NodeName 
    { 
        WindowsFeature DSCServiceFeature 
        { 
            Ensure = 'Present'
            Name   = 'DSC-Service'             
        } 

        xDscWebService PSDSCPullServer 
        { 
            Ensure                   = 'Present' 
            EndpointName             = 'PSDSCPullServer' 
            Port                     = 8080 
            PhysicalPath             = "$env:SystemDrive\inetpub\PSDSCPullServer" 
            CertificateThumbPrint    = $certificateThumbPrint          
            ModulePath               = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Modules" 
            ConfigurationPath        = "$env:PROGRAMFILES\WindowsPowerShell\DscService\Configuration" 
            State                    = 'Started'
            DependsOn                = '[WindowsFeature]DSCServiceFeature'     
            UseSecurityBestPractices = $false
        } 

        File RegistrationKeyFile
        {
            Ensure          = 'Present'
            Type            = 'File'
            DestinationPath = "$env:ProgramFiles\WindowsPowerShell\DscService\RegistrationKeys.txt"
            Contents        = $RegistrationKey
        }
    }
}

JYP fansites were based on Windows, but their management architectures were composed in the form of workgroups, so additional work was needed to process authentication between servers. The requisite task were generating the authentication certificate based on the source server, extracting public and private keys, and installing the extracted keys on the target server. In addition, to encrypt network communication between source server and target server, the SSL certificate for the source server was generated and applied.

The following PowerShell script generates the certificate.

New-SelfSignedCertificate -DnsName JYP-FANS-Z -CertStoreLocation cert:\LocalMachine\My
$cert = New-SelfSignedCertificate -Type DocumentEncryptionCertLegacyCsp -DnsName 'DscEncryptionCert' -HashAlgorithm SHA256
$mypwd = ConvertTo-SecureString -String "Passw0rd" -Force -AsPlainText
$cert | Export-PfxCertificate -FilePath "C:\Certificate\DscPrivateKey.pfx" -Password $mypwd -Force
$cert | Export-Certificate -FilePath "C:\Certificate\DscPublicKey.cer" -Force
$cert | Remove-Item -Force
Import-Certificate -FilePath "C:\Certificate\DscPublicKey.cer" -CertStoreLocation Cert:\LocalMachine\My

When configuring the front-end servers of the fansites, the authentication certificate is applied to VMs, so additional work is not needed when scale-out occurs.

Separating deployment to VMs in the test environment and the production environment is accomplished using the ConfigurationName property of PowerShell DSC. In the production environment, the value “JYPEFANS” is assigned to ConfigurationName; in the test environment, the value “TestFANS” is assigned.

When each environment has a different database, the requirement arose that values in Web.Config can be modified manually. This requirement is also applied to DSC configuration; depending on the situation, applying different Web.Config files to the same product can be enabled.

The following PowerShell script contains the DSC settings for production.

[DSCLocalConfigurationManager()]
configuration PullClientConfigNames
{
    Node localhost
    {
        Settings
        {
            RefreshMode = 'Pull'
            RefreshFrequencyMins = 30
            ConfigurationMode = "ApplyAndAutoCorrect"
            RebootNodeIfNeeded = $true
            Certificateid = "<Certificate Thumbprint>"
        }
        ConfigurationRepositoryWeb JYPEFANS-PullSrv
        {
            ServerURL = 'https://<Source Server>:8080/PSDSCPullServer.svc'
            RegistrationKey = '<Registration Key>'
            ConfigurationNames = @('JYPEFANS') # For testing, please change this name to TestFANS
        }
    }
}
PullClientConfigNames

Set-DSCLocalConfigurationManager localhost -Path .\PullClientConfigNames –Verbose

When the build result is deployed to the source server, the MOF file is assigned by using ConfigurationName and a new checksum is calculated for it. Thus each server can bring and apply modifications periodically. Moreover, because ConfigurationMode is set to “ApplyAndAutoCorrect”, any differences between the existing settings and the modified values will be applied as JYP settings.

Team Services release management

The following PowerShell script creates the MOF file.

$ConfigData= @{ 
    AllNodes = @(     
        @{  
            NodeName = "*"
            CertificateFile = "C:\Certificate\<Certificate Public Key file>.cer"
            Thumbprint = "<Certificate Thumbprint>" 
        },
        @{
            NodeName = "<Node IP Address>"
        }
    );    
}

Configuration ApplySource
{
    param (
    [pscredential]$credential
    )

    Import-DscResource -ModuleName PSDesiredStateConfiguration

    Node $allnodes.nodename
    {
        Archive WebFiles
        {
            Ensure      = 'Present'
            Path        = '\\<Source Server IP Address>\Sources\Test.zip'
            Destination = '<Destination Folder>'
            Force       = $true
            Validate    = $true
            Checksum    = 'SHA-1'
            Credential = $credential
        }

        WindowsFeature WebServer
        {
            Name      = 'Web-Server'
            Ensure    = 'Present'
            IncludeAllSubFeature = $True
            DependsOn = '[Archive]WebFiles'
        }           
    }
}

$cred = Get-Credential
ApplySource -credential $cred -ConfigurationData $ConfigData

$destination = "C:\Program Files\WindowsPowerShell\DscService\Configuration\<File Name>.mof"
Copy-Item -Path .\ApplySource\<File Name>.mof -Destination $destination -PassThru
New-DscChecksum -Path $destination -Force -Verbose

By using PowerShell DSC, repeated tasks for configuring test environments are eliminated and reliable settings are enabled at the same time. This approach prevents both the human errors that developers can make and the problems caused by repetitive work that waste time.

After we completed the DevOps story, the JYP CIO raised another topic outside DevOps. Before the hackfest, JYP used ASP.NET to maintain session state in IIS. The CIO was concerned that the ASP.NET state server didn’t support high availability. And he wanted to use the solution on Windows platforms too.

Many people think that Redis is supported only on Linux. But Microsoft Open Technologies built a production-ready Windows port of Redis, including 64-bit support, an installer for Azure, NuGet support, Redis Sentinel for high availability, and much more. So we suggested that JYP adopt Redis on Windows. We quickly built Redis on Windows and tested high-availability scenarios.

And we didn’t forget to mention the Azure Redis Cache service. It will be easier to administer and implement than infrastructure as a service VMs. For now, though, they want to test within the infrastructure as a service environment. They promised it will be tested soon.

Conclusion

The results of the hackfest were quite satisfying:

  • Improved work time for configuring test environment: 4 hours → 15 minutes (reduction of 93.5 percent)
  • Improved work time for deploying and verifying test/production environment: 2–3 hours per server → 15 minutes (reduction of 87.5–91.7 percent)
  • Probability of reconstitution for test/production environment: 20 percent → zero
  • Improved work time for building additional servers and deploying production environment when service requests are flooded: 1 hour per server → 15 minutes (reduction of 75 percent)
  • Microsoft interoperability with open-source platform and service: Redis on Windows and Azure Redis Cache

“In cloud configuration, adding or removing servers is a common practice, so manual actions by PMs and developers are not efficient at all. Through this hackfest, developed code can be reflected faster and more accurate than before.

“Moreover, with DevOps processes, improved points and effects of improvements can be expressed numerically, so the level of contribution of the IT team to the whole business comes to light. From now on, even if traffic is flooded unexpectedly, fansites can provide ordinary services instead of showing error messages. It will also be very helpful to spread the Korean wave.”

—Chan Park, CIO of JYP Entertainment IT team

Before this hackfest, applying new or improved features demanded by fans and internal staff required 2–3 business days after development finished. However, the technologies discussed and utilized during this hackfest reduced that requirement to a maximum of 2 hours. In addition, building additional servers is now based on automatic processes, preventing fansites from being put out of service by an unexpected surge in page requests.

Future directions

During the hackfest, the following directions were recognized as future areas of focus for the JYP IT team:

  • Possibility to adopt other solutions, such as Chef on the JYP home-page server
  • Add container technology to front-end servers
  • Improve the monitoring capabilities of the system, such as Azure Application Insights, Azure Operations Management Suite, and Azure Security Center

In fact, small- and medium-sized companies in Korea have little interest in DevOps. Before this hackfest, JYP hesitated to adopt DevOps technologies, too. Now they have confidence in DevOps processes and related technologies. Microsoft Korea DX offered additional guidance in the form of an additional hack day if desired or necessary to develop additional infrastructure.

Additional resources

The sample code for this solution is available on GitHub: Azure/azure-quickstart-templates

Windows PowerShell DSC

Chef on Windows

Redis on Windows

Visual Studio Team Services