Track code and actor activity coverage
Standard code coverage tools record the percentage of code lines that are actually executed by test
cases. Coyote additionally defines the higher-level metric Actor Activity Coverage that reports
state transitions and the percentage of possible events that are actually executed during a run of
Coyote coverage options
coyote /? displays a summary of command-line options. Here is the section describing
options to report activity coverage:
Coverage options: ----------------------------------- -c, --coverage string : Generate coverage reports if supported for the programming model used by the test.
Detailed descriptions are provided in subsequent sections. The following provides a quick overview.
--coverageis specified, Coyote will collect the history of all events and state transitions for reporting the coverage of the actors and state machines being tested.
- Coverage is not currently available for programming models other than actors.
Output file locations
By default, at the end of testing the report files are written to a directory named
Output/[assemblyToTest]/CoyoteOutput in the directory specified by the
path argument. If
--outdir outputDirectory is specified, then the files are written to the directory
Details of the report files that are created for the separate coverage types are provided in subsequent sections.
Actor activity coverage
Actor activity coverage includes event coverage, which is defined in the following section, as well as a summary of states that were entered and exited and which state transitions occurred.
Definition of event coverage
(M, S, E) is said to be defined if state S of machine M is prepared to receive an event
of type E, i.e., it has an action defined for the event.
(M, S, E) is said to be covered by a test run if state S of machine M actually dequeues
an event of type E during an execution.
Event coverage is the number of tuples covered divided by the number of tuples defined in the program. The higher this metric, the better testing exercised all these combinations in the program. As with other coverage metrics, obtaining 100% coverage may be unrealistic as it depends on the particular test harness being used.
Activity coverage output files
If the option
--coverage activity, or
--coverage activity-debug is passed to
coyote, the following files will be written to the output directory (for
path equal to
PingPong.coverage.txt. This file contains the Event Coverage metric along with a breakdown per
machine and per state. It also summarizes other useful coverage information.
PingPong.dgml. This file contains the Event Coverage visualization as described below.
PingPong.coverage.ser. This is the serialized
CoverageInfo object for the test run. Such
.coverage.ser files from multiple runs can be passed to
create a merged report.
--coverage activity-debug was specified, then there will also be a Debug directory containing
the same files as above for each process, with the filename qualified by a sequential process id,
Note that while
--coverage is a shortcut for specifying both
--coverage code and
activity, you must specify
--coverage activity-debug explicitly.
Activity coverage visualization example
The activity coverage can additionally be displayed in DGML diagram format. Run
coyote as described in the
coyote examples section below. This produces
a file in the DGML format as described in the activity coverage output
files section. Open the file using Visual Studio. It captures
machines, states and transitions witnessed during the testing of the program. The file also contains
inter-machine transitions. These transitions are usually auto-hidden when opened in Visual Studio,
but visible when you click on a state.
Coyote test examples
First build the Coyote samples by following the instructions here.
coyote with one of the coverage flags, as well as the other options you want. Here are
some minimal examples:
coyote test ./bin/net6.0/Monitors.exe -i 10 --coverage
This will create the directory
./bin/net6.0/Output/Monitors.exe/CoyoteOutput/, then it
generates coverage files for code coverage which you can load into Visual Studio to see the results.
coyote test ./bin/net6.0/Monitors.exe -i 10 -coverage activity -o "/Coyote_Coverage/Monitors"
This will create the directory
/Coyote_Coverage/Monitors/CoyoteOutput, then it generates only