More about E2E test · React Native for Windows + macOS

The E2E tests are now part of RN Tester and these pages are deprecated.

Technical Decisions

`MSTest` vs Node test runner

A node test runner is the first choice since we started the investigation for E2E test. React Native apps are written in JavaScript and it's a good choose to select a JavaScript framework to author the test case. It would be more friendly to the community, so MSTest with C# is excluded in our option.

WinAppDriver + WebdriverIO + Jasmine

There is not existing example we can follow to setup E2E testing on Windows for React Native, and I spent weeks to investigate, test and prototype for our E2E. Hereafter I explain what kind of decisions I made and why I made these decisions.

Technical Decision

Why WinAppDriver

Detox or Cavy require a significant, ongoing investment from us.
WinAppDriver is owned by us.

Why WebdriverIO

Be released more frequently.
wdio-sync makes you write test case without using await.
wdio test runner makes the integration as quickly as possible.

Personally, I think WebdriverIO also introduced some difficulties:

Doesn't support Jest.
Another set of APIs different from selenium community.
wdio test runner simplified the steps for user to do the integration, but it also complicated the system. It have a lot of processes which makes debugging very hard.

Why Jasmine

wdio doesn't support Jest.
Mocha is used in my prototype, but it doesn't report test failure if exception is thrown in before function.
It's a piece of cake to switch from Mocha to Jasmine, and I finished this job in 10 minutes. See PR 3197

`wdio` Test Runner

WebdriverIO comes with its own test runner @wdio/cli. It simplified the steps for the integration, but it make it hard to be understood and be troubleshooted.

Below chart explains how different libraries are linked together when wdio command is launched. Appium or WinAppDriver are running in separate processes, and WebdriverIO talks to them by W3C WebDriver protocol.

wdioInternalDependencies

WinAppDriver

WinAppDriver talks to a test app by UIA, and WinAppDriver implements part the W3C WebDriver protocol which allows WebDriver to talk to him. So WinAppDriver could talk directly with WebdriverIO, and whole test framework could be either with or without Appium.

There are two possible setup in dev environment based on with/without Appium:

Option 1
Option 2

Option 1 is recommended and implemented by default. Appium and WinAppDriver are launched before spec is executed and they are killed after the spec is finished.

Option 2 is for advance user only. Each time we release a new WinAppDriver, we also need to update code on Appium. Before Appium has a new release, option 2 is the only way to verify the new features provides by WinAppDriver.

If yarn install is run as admin privilege, WinAppDriver would be installed automatically, otherwise you need to install WinAppDriver manually.

For the Azure pipeline, WinAppDriver is already installed on Windows Server 2019.

Bundle

Generally speaking, you don't need to care about it until you setup Azure pipeline, or the app is ready for production.

In E2E project, bundle has two meanings:

Just like react-native bundle helps to generate an offline bundle file, then the app can run without the Metro Bundler.
A macro to build the native project. ReactUWPTestApp provides six bundle configurations: ReleaseBundle|x86, ReleaseBundle|x64, ReleaseBundle|ARM, DebugBundle|x86, DebugBundle|x64, DebugBundle|ARM

Bundle is very close to the production environment. Here are commands to support Bundle in the E2E test:

Command	Description	Example
`bundle`	Create a bundle file which then be packaged to the native app	`yarn run bundle`
`buildbundleapp`	build the native app with BUNDLE macro. `--release` specify if it's a release version. `--arch [string]` The build architecture (`ARM`, `x86`, `x64`) (default: `x86`)	`yarn run buildbundleapp` `yarn run buildbundleapp --release` `yarn run buildbundleapp --arch x64`
`deploybundleapp`	Deploy the built test app, you can pair it with `--release` and `--arch`	`yarn run deploybundleapp` `yarn run deploybundleapp --release` `yarn run deploybundleapp --arch x64`
`e2ebundle`	Make a bundle, Build and deploy the solution, and run the testing	`Yarn run e2ebundle`

Locators to find UI Element

No matter what JavaScript framework you choose for native app testing, you have to use one of the locators which is described in mobile JSON wire protocol. Since locators are implemented significant different on iOS, Android and Windows, as below I only talk about the locators for Windows.

Locators WinAppDriver supports

WinAppDriver provides rich API to help locate the UI element. If testID is specified in React Native app for Windows, the locator strategy should choose accessibility id.

A unique accessibility id / testID per Window is recommended for React Native Windows E2E testing when authoring the test app and test cases. To ease the maintain effort, all testIDs are defined in Consts.ts, then be imported by test app and test page objects or test cases.

Client API	Locator Strategy	Matched Attribute in `inspect.exe`	Example
`FindElementByAccessibilityId`	`accessibility id`	`AutomationId`	`AppNameTitle`
`FindElementByClassName`	`class name`	`ClassName`	`TextBlock`
`FindElementById`	`id`	`RuntimeId` (decimal)	`42.333896.3.1`
`FindElementByName`	`name`	`Name`	`Calculator`
`FindElementByTagName`	`tag name`	`LocalizedControlType` (upper camel case)	`Text`
`FindElementByXPath`	`xpath`	`Any`	`//Button[0]`

Locators WebdriverIO supports

Client API by Example	Locator Strategy
`\$('~AppNameTitle')`	`accessibility id`
`\$('TextBlock')`	`class name`

Locators `selenium-appium` supports for `selenium-webdriver`

Client API by Example	Locator Strategy
`By2.nativeAccessibilityId('AppNameTitle')`	`accessibility id`
`By2.nativeClassName('TextBlock')`	`class name`
`By2.nativeXpath('//Button[0]')`	`xpath`
`By2.nativeName('Calculator')`	`name`
`By2.nativeId('42.333896.3.1')`	`id`

Timers

wdio.conf.js (see WebdriverIO Timeouts )

// Default timeout for all waitFor\* commands.
waitforTimeout:10000,

jasmineNodeOpts: {
        defaultTimeoutInterval: 60000,
    }

Session Implicit Wait Timeout

A session has an associated session implicit wait timeout that specifies a time to wait for the implicit element location strategy when locating elements using the findElement() or findElements() commands. Unless stated otherwise it is zero milliseconds. You can set this timeout via:

browser.setTimeout({ 'implicit': 5000 });

waitForPageTimeout.

This timer is provided by BasePage object.

  // Default timeout for waitForPageLoaded command in PageObject
  privatewaitforPageTimeout: number = 10000;

You can override the setting in test case by passing a timeout parameter when call waitForPageLoaded, for example:

LoginPage.waitForPageLoaded(15000)

`wdio.conf.js`

Capabilities

capabilities are the configuration which Appium/WinAppDriver used to identify the app and launch the app. Below configurations supports both with and without Appium in the setup.

appium:app is used by Appium when the setup is WebdriverIO <-> Appium <-> WinAppDriver,

app is used for directly connection between WebdriverIO <-> WinAppDriver.

  capabilities: [
  {
      maxInstances: 1,
      platformName: 'windows',
      'appium:deviceName': 'WindowsPC',
      'appium:app': 'ReactUWPTestApp_kc2bncckyf4ap!App',
      'deviceName': 'WindowsPC',
      'app': 'ReactUWPTestApp_kc2bncckyf4ap!App',
      'winAppDriver:experimental-w3c': true,
  },

Log Level

    logLevel: 'trace'

Timeout

    // Default timeout for all waitFor* commands.
    waitforTimeout: 10000,

Appium service

Below configuration lets the framework launch/kill Appium automatically during the testing, and logs are saved as reports\appium.txt.

    port: 4723,
    services: ['appium'],
    appium: {
        logPath: './reports/',
        args: {
            port: '4723',
        }
    },

But the log like below is not that readable because it has a lot of control characters which is used by terminal.

Bad Readability

You could get nice output if you launch the Appium by yourself:

Modify the configuration and remove appium from services

services: [],

Start Appium by yourself

.\node\_modules\.bin\appium

Test framework

framework: 'jasmine'

Reports

reporters: ['dot', ['junit', { outputDir : '.\\reports' }]],

`PageObject` Pattern

PageObject Pattern is recommended in the E2E test, and you can get more information from these links:

PageObjects

WebdriverIO PageObjects

Element Locator `By()`

private get submitButton() {
    return By('Submit');
}

this.submitButton.click();

You can easily to use By(string) to locate a element which associated with testID in the app.

It's recommended to define a get for each locator like above.