Redback Technologies is exploring ways to integrate blockchain technologies into its energy products and services. However, at this early stage, blockchain technology and frameworks are an extremely new field without many best practices defined yet. Most notably, there has been very little guidance about implementing DevOps around blockchain projects, with everything being done manually on developers’ machines.

Microsoft engaged Redback to implement an initial DevOps process that would allow developers on the team to modify their smart contracts, have a continuous integration (CI) build confirm there are no build errors, provision an in-memory test blockchain testnet to run the integration tests, and publish the test results back into Visual Studio Team Services.

This was groundbreaking work, with the learnings being published in blog posts, added to official documentation pages, and reused internally by other teams within the Microsoft Blockchain Technical Working Group.


VSTS Dashboard showing unit tests passing


Key technologies used:

Microsoft hack team participants:

  • David Burela – Melbourne, Australia, Senior Technical Evangelist, DX Australia
  • Paul Bouwer – Brisbane, Australia, Senior SDE, Partner Catalyst team
  • Shawn Cicoria – NY Metro, USA, Senior SDE, Partner Catalyst team
  • Eric Maino – Redmond, USA, Senior SDE, Partner Catalyst team
  • Ali Hajimirza – Redmond, USA, SDE, Partner Catalyst team
  • Tyler Gibson – Redmond, USA, Senior SDE, Partner Catalyst team

photo of the group

Customer profile

Redback Technologies is an Australian company located in Brisbane, Queensland. It focuses on the development of advanced, low-cost solar solutions for residential and commercial users. Its products include Smart Hybrid Solar Inverter System, which has intelligent technology that gives you the power to store, monitor, and manage your home’s solar energy—all in a compact, elegantly designed unit mounted either inside or outside your home. They have been featured in the media as an innovative energy solutions company.

redback company logo


Problem statement

Blockchain-based developments are complicated. To further exacerbate this, the current state of the tools and frameworks are in the very early stages of development. This means there is not much in the way of guidance or best practices available, and that requires us to come up with unique solutions to all traditional development practices.

Redback was engaged with Microsoft around integrating blockchain technology into its products and services. Early on, it was discovered that having multiple developers working on the same smart contract code could result in the typical range of issues prone to modern software development:

  • Merge conflicts.
  • Not all solution files being checked in.
  • New commits breaking changes in previously working code.
  • Not running tests frequently, especially before/after committing code.
  • Building on a developer’s local machine and deploying to production.

It was decided that putting a basic DevOps process around the critical inner portion would greatly increase the team’s confidence in the current state of the code and accelerate future development work.

Solution, steps, and delivery

It was quite a journey to get from a manual single-developer experience to a workflow and toolchain that scaled up to allow a DevOps flow for a team of developers. We looked at a number of tools and frameworks out in the community and tried to see which had enough mindshare to allow for easier support. We worked as a team to look through what the challenges were and how best to overcome the inefficiencies.

photo of the group during initial discussions


Blockchain development done manually

Writing blockchain “smart contracts” is done in a language called Solidity. The Microsoft team had previously used a number of manual tools on a developer’s machine to develop smart contract code in Solidity in order to compile, deploy, and so on.

initial value stream map


Our value stream mapping (VSM) exercise looked at what the impact would be if Redback were to adopt this development style. It exposed that using the tools as-is would be very wasteful. There were a lot of manual steps throughout the process, with many chances for defects to arise from misconfigured tools, incorrect blockchain network connection settings, and manual copy and pasting. The toolchain was optimized for a single developer and did not scale well with a team of developers. It also did not have a way to automate processes such as having a continuous integration build to check all committed code.

The VSM exercise identified potential improvements around automating builds and automating the testing. These would remove the time-intensive manual steps, as well as ensure that there is a repeatable process for building smart contracts. Automating this would also give direct benefits by giving an indication of the health of the current checked-in code.

photo of the group mapping out workflows


Improving blockchain development with DevOps

In order to improve the development experience for the team, we automated and improved the core check-in process. At a minimum, a CI build was needed and a way to automatically run unit/integration tests on blockchain code.

The problems we encountered during the engagement were:

  • Finding a blockchain toolchain that could be used on developer machines and on the Visual Studio Team Services build agent.
  • Having a reliable way to get global npm packages to be executable from the Visual Studio Team Services build agent.
  • Getting a temporary blockchain environment provisioned and torn down for each execution of the unit/integration tests.
  • Outputting the test results into a format that Visual Studio Team Services would accept.

Using open source blockchain tools

After evaluating a few different blockchain tool offerings, an open source framework—Truffle—was selected to assist with improving the workflow. Truffle allows a developer on their local machine to compile, run unit tests, and deploy the artifacts to a blockchain network. This tight “inner loop” integration rapidly improves general developer productivity and cuts down on manual steps.

Although running Truffle on a single developer’s machine helped improve their individual workflow, by itself it didn’t scale to automate the team’s DevOps flow. I then focused on finding a way to get Truffle running on a build agent that would allow Visual Studio Team Services to orchestrate a DevOps process around it.

Configuring a build agent

To support the Visual Studio Team Services build process, a Windows virtual machine was provisioned in Microsoft Azure and configured with the required build tools and a Visual Studio Team Services build agent. Because all the blockchain development tools require Node and npm to function, the first step was to install and configure the Windows versions of these tools. Next, the supporting build tools such as Visual C++ compilers and Python were installed automatically via the Windows build tools npm package. Once the prerequisite frameworks were installed, Truffle, TestRPC, and a MochaJS extension package were installed as global npm tools.

The work we did to find a repeatable way to install the prerequisites on a Windows desktop machine and Windows-based build server resulted in a set of tutorials that were put onto the official Truffle documentation site. (See Published walkthroughs.)

Configuring Truffle to output JUnit format

Truffle uses the Mocha test framework to run the smart contract unit tests. Normally, the test results are displayed on the developer’s machine when run locally, but we wanted to have the tests run on the build agent and the results uploaded into Visual Studio Team Services. Mocha can be configured to output the results in a variety of file formats, but none of the defaults worked well with Team Services. The Mocha JUnit Reporter was able to output the results into a format that Team Services understood and could import.

running Truffle test


VSTS Dashboard showing unit tests passing


Hosting a blockchain testnet for test execution

When Truffle executes the tests on the Visual Studio Team Services build agent, it needs a blockchain network to deploy the smart contracts to, so that the tests can be executed in the environment. Doing it on a real blockchain network would take an extraordinarily long time, because each operation would need to wait for the block to be mined before the next operation could take place. With each block taking 12 seconds, and each test requiring multiple operations, it could take tens of minutes to execute a test suite. TestRPC is an in-memory blockchain environment that allows for instant mining, perfect for development and test execution.

In order to take advantage of it on the build server, we needed to find a way to start a fresh environment, allow test execution to happen, and then tear down the instance after test completion. A series of PowerShell scripts were created to automate this process, allowing Visual Studio Team Services to orchestrate the test run. These scripts are now available on the official Truffle documentation site for reuse by others. (See Published walkthroughs.)


photo of the training the group to use Truffle


Automated DevOps solution

The final DevOps solution that was deployed involved using a mixture of Visual Studio Team Services combined with a number of open source blockchain tools and frameworks.

diagram of the workflow implemented in VSTS


The DevOps solution had the following workflow:

  1. Developer checks in new code.
  2. Visual Studio Team Services build is triggered.
  3. Build agent pulls down the source code.
  4. Truffle is used to compile the smart contracts.
  5. PowerShell script spins up a temporary in-memory blockchain by using the TestRPC tool.
  6. Truffle is used to run the unit tests. Results are output into a JUnit.xml file.
  7. PowerShell scripts tear down the TestRPC blockchain.
  8. Unit test results are uploaded to Visual Studio Team Services.

A followup value stream mapping exercise found that the introduction of better tooling, combined with the automation by Visual Studio Team Services, drastically reduced the process time. The removal of manual steps helped to speed things up and eliminated the defects that arose from manually moving from one tool to another.


VSM after DevOps


Deploying to the blockchain is still a manual step; however, now that the tooling has been improved, it is easier to take the artifacts and run Truffle to run the same migration script to deploy to a UAT or production blockchain on Azure Blockchain as a Service.

Conclusion

photo of the group reviewing the work


Redback is doing some great work with blockchain development to explore ways to improve its product offerings. It was great to be involved in helping the Redback team use a combination of Visual Studio Team Services and open source blockchain tooling to automate their team’s development. The learnings from this engagement were turned into new training materials that were published in blog posts, added to official documentation pages, and reused internally by other teams at Microsoft.

We introduced new DevOps practices around blockchain development by removing manual steps and using Visual Studio Team Services to implement the practices of continuous integration and automated testing. The introduction of DevOps via Team Services and having a visible dashboard showing the health of the build and unit tests are giving the team at Redback the confidence to develop their code quickly with a faster feedback loop, over the previous manual way of doing blockchain development.

Future work

There was only limited time to implement the basics of a working Visual Studio Team Services system. In the future, there is additional functionality that could be integrated in:

  1. Automated release management:

    Currently, the build artifacts need to be taken and then published to a blockchain via the truffle migrate command. Automating this via Team Services release management to push to an Azure Blockchain as a Service testnet or production network is the next logical step.

  2. Use of a Linux build server or Windows Subsystem for Linux:

    Visual Studio Team Services supports both Windows- and Linux-based build agents. The blockchain development tools are npm-based and will work on both platforms. Offering a build solution for teams that prefer Linux over Windows would be useful.

  3. Upgradable smart contracts:

    Smart contract logic is non-mutable once published. However, through the use of address variables within a smart contract, it is possible to point internal functions at other contracts, which can then be swapped out to perform an upgrade. Integrating this with the future release management work above is a natural tie-in.

Additional resources

Here are links to additional documentation walkthroughs and tools that were used throughout this project.

Published walkthroughs

Tools

Source code

To make it easy for others to reproduce our work, the Visual Studio Team Services steps were implemented as a small number of discreet PowerShell scripts. This helped keep each section small and easily debuggable.

The PowerShell scripts assume that you have npm as well as the Truffle and TestRPC packages installed on the build machine.

photo of the group reviewing the work


System version information

#assists with debugging and ensuring all tools are installed 
#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path
npm config set prefix $env:npm_path    #only needs to be set once, will update for user
#DEBUG
#$env:path
#npm list -g –depth=0
#Display system information
Write-Host “System version information”
Write-Host -nonewline    “node version: ” ; node -v
Write-Host -nonewline    “npm version: “; npm -v
Write-Host -nonewline    “npm prefix: “;  npm prefix -g
Write-Host -nonewline    “truffle: ” ;    truffle version


Config transform and previous test cleanup

# remove old test results
rm .\junitresults.xml -ea SilentlyContinue 

# Modify the Truffle test runner to use the JUnit reporter
Rename-Item .\truffle.js .\truffle_temp.js
cat .\truffle_temp.js | % { $_ -replace ‘reporter: “spec”‘, ‘reporter: “mocha-junit-reporter”‘ } | Out-File -Encoding ASCII .\truffle.js
rm .\truffle_temp.js


Truffle build

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path#Truffle build
truffle compile


Launch TestRPC

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path# launch the process
echo “launching TestRPC”
$testrpcProcess = Start-Process testrpc -passthru
# persist the PID to disk and display in logs
$testrpcProcess.Id | Export-CliXml testrpcPID.xml
cat testrpcPID.xml


Run Truffle tests

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path# Run the tests
truffle test


Shut down TestRPC

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path# retrieve the PID and kill the entire processs tree
cat testrpcPID.xml
$testrpcPID = Import-CliXml testrpcPID.xml
taskkill /pid $testrpcPID /F /T