Create, Upload and Run Tests in VSAC

In this lab we will Create and configure a new Test run in Visual Studio App Center (VSAC), prepare and then upload the Tests, Run and then review them.

DevOps MPP Course Source

Pre-requisites:

  • Complete the earlier Mobile VSAC lab Getting Started in Visual Studio App Center
  • Add Getting Started Android app to Visual Studio App Center
  • Build the sample app Getting Started for Android
  • Sign the app with an Ad-Hoc keystore
  • Store the GettingStarted.apk in a folder on your workstation, referred to as GettingStartedFolder.

Lab Tasks:

  • Create and configure new test run
  • Install App Center CLI
  • Prepare Tests for Upload
  • Upload and Run Tests
  • Review Tests

Estimated Lab Time:

  • approx. 30 minutes

Task1: Create and Configure Test Run

  1. Select the Getting Started app for Android, which uses Xamarin.UITest.

  2. Under Build > New Test Runs, click the New Test Run button.

  3. Configure your test run by using the Test Run Wizard. Name the test run “First Tests”. Pick the Test Series default master. Choose a system language of English (United States), and the test framework Xamarin.UITest.

  4. Select five devices of your choosing to run your tests on.

  5. Copy the entire appcenter test run command for later use.

Task 2: Install App Center CLI

To submit tests, you must install App Center Command Line Interface (CLI) on your development computer. The CLI runs on Node.js, so it can be installed on a Windows or macOS machine. The installation requires two steps:

  1. In your browser, navigate to the Node.js download page and download and install Node.js. The LTS version should suffice.

  2. Invoke a command prompt and change the directory to the Node.js installation directory, typically C:\Program Files\nodejs on Windows. Install the appcenter-cli NPM package by using this command:

       npm install -g appcenter-cli

Task 3: Prepare Tests for Upload

Prepare the Getting Started Xamarin.Android app and UITest for submission to App Center Test. The Getting Started app already has the INTERNET permission granted in AssemblyInfo.cs.

  1. In the Getting Started app, confirm that the Shared Mono Runtime is disabled. The Shared Mono Runtime will prevent UITests from running in App Center Test.

    • On Visual Studio 2017, open project properties and select the Android Options tab. In the Packaging Properties section, uncheck the checkbox for the Shared Mono Runtime.

    • On Visual Studio for Mac, look in the Project Options, under Build > Android In the Packaging and Deployment section of the General tab, uncheck the checkbox for the Shared Mono Runtime.

Task 4: Upload and Run Tests

After installing the App Center CLI, log in to App Center and upload your tests. Follow these steps:

  1. Open a command prompt and navigate to the NuGet /package directory of your app solution.

  2. Log in to App Center by using the command appcenter login. You’ll be redirected to your browser where App Center will provide you with an authentication code. Copy that token and paste it back into the the command prompt window to authenticate.

  3. Upload your tests by using the appcenter test run command you copied from the final page in the New Test Wizard in App Center Test when you created a new test run. Your command will look something like this, which is for Xamarin.UITest:

appcenter test run uitest --app "Getting-Started" --devices devicesid --app-path pathToFile.apk  --test-series "First Tests" --locale "en_US" --build-dir pathToUITestBuildDir
  1. Populate the placeholder fields with your real values in parameters, including devices, app-path, and build-dir. Here are the main parameters in the appcenter test run call:
  • app – Name of the app in VSAC
  • app-path - Path to the app to be tested, ex. .apk, .ipa , ex. “\Development\GettingStarted\GettingStarted.Android\bin\Release\GettingStarted.Android.apk”
  • devices – ID or name of device set in VSAC to run tests on
  • test-series - Name of the test series in VSAC that this test run will belong to
  • build-dir – Directory path of your test assembly, ex. “\Development\GettingStarted\UITests\bin\Release”

Here is an example command:

appcenter test run uitest --app "Getting-Started" --devices 4c93991 --app-path "\Development\GettingStarted\GettingStarted.Android\bin\Release\GettingStarted.Android.apk"  --test-series "First Tests" --locale "en_US" --build-dir "\Development\GettingStarted\UITests\bin\Release"

When you provide the correct parameters and your tests are coded properly then the appcenter test run command will upload your tests to App Center Test and run them immediately. The command window will show you the list of test devices and high level results for each device. If you experience issues, follow the error messages and ensure that your artifacts are in the specified folders.

Task 5: Review Tests

After your tests have completed running by using the appcenter test run command, you can view the results online in App Center Test.

  1. Click on the VSAC menu Tests>Test Runs and choose the test run you’d like to review. If you’re using test series, flip between test series by using the Test Series button at the top of the screen. Review the test report for that test run.

  2. Drill down into specific tests for device screenshots.

  3. Click on a device for a large screenshot and additional details such as hardware usage and timings.

  4. Click the Details button to see the device-specific specifications. Review and exit details.

  5. Click the Logs button to review the device logs, test logs, and a stack trace.

Summary

In this lab you completed the following tasks

  • Created and configured new test run
  • Installed App Center CLI
  • Prepared Tests for Upload
  • Uploaded and Run Tests
  • Reviewed Tests