Skip to main content

FAQ

Why the Virtual Client was Created?​

Partner teams in Azure created the Virtual Client because we care deeply about performance and quality for our customers. In fact, the Virtual Client team and close partner teams collectively own the charter for ensuring that Azure hardware systems offer the best performance and reliability experience to our customers. The Virtual Client was designed to enable our teams in Azure to accomplish this task. Virtual Client is used every day in the Azure cloud as part of the mission to build the best hardware systems on the planet. The Virtual Client allows our teams to cover the breadth of the customer experience running a wide range of benchmarks on systems all across the planet at a "cloud scale".

Why Microsoft Open-Sourced the Platform?​

  1. Learning a benchmark is hard. Learning hundreds of benchmarks is an impossible task for most teams. In Azure, we live with and understand the complexity of hardware systems on a daily basis. We are testing them by the 1000s on a daily basis. So, we understand how difficult it is to produce meaningful benchmarking results, or to debug performance degradations. We want to empower every performance-enthusiastic person to be able to benchmark services and systems with ease.

  2. We'd really like your contribution. Perhaps you are an expert in a particular workload or monitor. That expertise can travel further if we work together to integrate it into a common platform. This will enable anyone else around the world to leverage your experience, your expertise making the world of hardware validation and benchmarking a better place.

Why Should You Use The Virtual Client?​

  1. Minimize the learning curve. Virtual Client is much more than just a few scripts to automate workloads. VC was born out of the efforts and experience of engineers recognized in the cloud industry as experts in hardware and virtualization performance experts, folks that design and validate next generations of cloud hardware. We work with these folks on a daily basis to constantly infuse their expertise into the Virtual Client. It is no overstatement that the due diligence that goes into each workload or feature onboarded to the platform comes from decades of engineering experience building cloud-scale systems.
  2. The Virtual Client team is on a mission with partner teams across Microsoft and partner companies to make VC the industrial standard for benchmarking and monitoring hardware systems.

When VirtualClient is probably not a good fit​

  1. You only need one-off testing with a workload you already understand. You might have some scripts that doesn't need much maintainences.
  2. You are fine tuning for an absolutely best score from one particular workload, for benchmarking competitions. VirtualClient is designed to measure general experience from customers' perspective.

How is VC used at Azure?​

Quality is one of the top priorities for Azure. The Virtual Client is used to assess the performance and reliability of hardware systems across the world in the Azure cloud to protect the customer experience as changes/updates are made to the systems that could regress their reliability or performance. We run on 1000s of hardware systems every single day as part of our charter.

Azure runs large scale AB-testing on firmware, software updates that goes to the fleet. We block updates that measures a noticeable difference between baseline and control groups. This methodology has been protecting Azure customers' experience on Azure's offerings. Now, you can use the tool Azure used, to protect your hardware or software performance.

Will VC support commercial workloads like SPECcpu, GeekBench?​

Yes, the team plans to support commercial workloads in the future releases. The team is working to establish and document a model that will enable workloads that require licenses to be supported without having to distribute the workloads themselves. User will be able to readily incorporate the workloads themselves supplying their own licenses.

What if my workload is not supported by VC?​

  1. Take a look at our Roadmap to see if we are already planning to onboard the workload.
  2. Send us a Feature Request if it's not on our radar yet.
  3. If you would like to contribute, we welcome it! Please refer to our Development Guide to get started.
  4. You can even write your own extensions outside of this repo in your own public or private repo. Virtual Client supports an extensions model.

Does VC send any data to MSFT?​

No, the Virtual Client does not collect any of your benchmark data for upload to Microsoft. Even when benchmarking at scale, and leveraging the Virtual Client's telemetry capabilities, users supply connection strings etc. to storage resources they own and manage. Your data remains your own.