INF-0006 - Long Vector Execution Test Plan

• Impacted Projects: DXC

Introduction

This test plan covers testing all HLSL intrinsics that can take long vectors as parameters. And more specifically, it only covers testing scenarios which will get coverage from a graphics driver supporting DXIL.

These tests will verify that all DXIL opcodes and LLVM instructions which can be reached using valid HLSL in SM 6.9 can compile, run, and produce correct output when given long and native vector inputs. They will not verify that the generated DXIL is vectorized.

All tests are to be included in the HLK test binary which ships with the OS. This test binary is only built in the OS repo and based off of the ExecutionTests source code in the DXC repo. There is a script in the WinTools repo which generates and annotates the HLK tests.

We break coverage down into five test categories.

1. Implement DXIL OpCode tests:

   • At the bottom of this document there are tables containing all HLSL operators (more on those in ‘3. HLSL Operator Tests’) and HLSL intrinsics that can be used with long vectors. The HLSL intrinsics tables have a DXIL OpCode and LLVM instruction columns. These columns contain the intrinsic’s mapped DXIL OpCodes as well as their LLVM instructions. All intrinsics have at least one DXIL OpCode or one LLVM instruction.

     Many intrinsics have trivial mappings. Atan is an example of an intrinsic with a trivial mapping. Other intrinsics have multiple DXIL OpCodes. Some intrinsics will use all listed DXIL OpCodes and/or LLVM instructions, while others will have additional logic which determines which OpCodes/Instructions are used. If an intrinsic relies on additional logic to determine which OpCodes/Instructions are used then the OpCode/Instructions will be enclosed in ‘[]’ brackets. The sign intrinsic is an example of an intrinsic with additional logic. If an OpCode/Instruction is not enclosed in ‘[]’ then it is used in all paths for that intrinsic.

2. Implement LLVM Instruction tests:

   • These are the test cases for the LLVM Instructions listed in the table at the bottom of this document.
   • Because we will use HLSL intrinsics to get coverage for the DXIL OpCode tests we speculate that we will get most of the coverage needed for the LLVM Instruction tests. After implementing the DXIL OpCode tests we should be able to do a coverage audit and ammend test cases, or write simple additional ones, as needed.
   • Just as in ‘1. Implement DXIL OpCode Tests’ some cases have multiple instructions listed. ‘[]’ brackets are used in the same manner. And there may also be multiple instructions.
   • Additional OpCodes/Instructions are logic based (i.e float or int specific).

3. HLSL Operator tests:

   • HLSL Operators Table in this document lists the HLSL Operators which can take long vectors as arguments.
   • Many of these operators can and will get coverage by default in the DXIL OpCode tests. But we will audit coverage and ammend test case, or write simple additional ones, as needed.

4. Standard loading and storing of long vectors

   • These could be covered in test categories 1 and 2. But I propose we break out individual cases to ensure that we have more granular coverage.
   • Ensure we have some basic tests doing standard loading/storing of long vectors across Buffer types to test and Vector element data types to test.
   • Additionally, the above buffer types and data types should be tested by loading from a ResourceDescriptorHeap

5. ‘Creative’ test cases:

   • Sizes around alignments and boundaries. See Vector sizes and alignments
   • Odd (non even) number of elements in vector. See Test Sizes

Buffer types to test

• Raw Buffers (Byte Address Buffers)
• Structured Buffers (StructuredBuffer<T>)

Vector element data types to test

Testing will cover the following vector element data types:

• bool, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, float16_t, float32_t, float64_t, packed_int16_t, and packed_uint16_t.

Vector sizes and alignments to test

General sizes to test are in the range [3, 1024]. It is worth noting that the new form of rawBufferLoad will be updated to vectorize sizes < 5.

Test sizes

• vector<TYPE, 3> : Testing one below previous vector limit. Early testing found some issues here so it was added.
• vector<TYPE, 4> : Previous limit.
• vector<TYPE, 5> : Testing one above previous vector limit.
• vector<TYPE, 16> : This size of ‘vector’ previously only appeared as matrices.
• vector<TYPE, 17> : Larger than any vector previously possible.
• vector<TYPE, 35> : Arbitrarily picked.
• vector<TYPE, 100> : Arbitrarily picked.
• vector<TYPE, 256> : Arbitrarily picked.
• vector<TYPE, 1024> : The new max size of a vector.
• These sizes will be tested across Vector element data types to test

Some noteable alignment cases

• 128 bit boundaries : Memory access for Shader Model 5.0 and earlier operate on 128-bit slots aligned on 128-bit boundaries. An example is vector<half, 7>, vector<half, 8> and vector<half, 9>. 112 bits, 128 bits, and 144 bits respectively. This boundary will tested for with 32-bit and 64-bit sized values as well.

• Most GPUs operate on at least 32-bits at once, so what happens if you use 16-bit values and an odd number of elements. Could accessing the last element expose issues where we could overwrite the next variable if it is assuming alignment?

High level test design

1. The test will leverage the existing XML infrastructure currently used by the existing execution tests. There are two XML files. This general design pattern exists today in the execution tests.

   • 1st XML: Used to define shader source code and metadata about that shader code. This XML file is parsed using a private class. This private class helps facilitate creation of D3D resources and execution of the shader.
   • 2nd XML: Describes metadata about the specific test cases. Used by the TAEF infrastructure for TAEF Data Driven Testing

2. Test inputs will be hard coded in a c++ header file. This was chosen over definining inputs in the second XML as this is cleaner and easier to parse for different data types. This c++ header method also avoids needing to repeat the data set in the XML for each individual test cast. Inputs will use ‘value sets’ which will typically be much smaller than the desired vector test size. Values will be repeated cyclically until the vector is full. For example, a value set {1, 2, 3} used to populate a vector<int, 1024> will produce the pattern <1, 2, 3, 1, 2, 3, ...>, repeating the sequence until all 1024 elements are filled. This approach provides predictable test data while keeping input definitions manageable.

3. Expected outputs are computed for each test case at run time.

4. All new long vector test code is factored out into its own files.

Implementation phases

Do the test work in two simple phases.

1. Implement and validate (locally against WARP) for all test categories.
2. HLK related work:

• Add a SM 6.9 HLK requirement. Includes updating the HLK requirements doc.
• Update mm_annotate_shader_op_arith_table.py to annotate the new test cases with HLK GUIDS and requirements
• Add new tests to HLK playlist

Shipping

Note that because DXC and the Agility SDK are both undocked from Windows it is our normal operating behavior for the HLK tests to become available with a later TBD OS release. The good news is that this doesn’t prevent the tests from being available much earlier in the DXC repo. It just means that they are simply TAEF tests in the DXC repo. An HLK test includes an extra level of infrastructure for test gating, selection, and result submission for WHQL signing of drivers.

1. Tests will be shared privately with IHVs along with the latest DXC and latest Agility SDK for testing and validation. IHVs will also be able to build and run the tests from the public DXC repo themselves. If needed Microsoft can share further instructions when the tests are available.

2. The tests will ship with the HLK at a TBD date in a later OS release.

Test Validation Requirements

The following statements must be true and validated for this work to be considered completed.

• All new test cases pass when run locally against a WARP device
• All new test cases must verify applicable outputs for correctness.
• All new test cases are confirmed to be present in HLK Studio and selectable to be run when a target device satisfies the HLK ShaderModel 6.9 requirement.
• All new tests/test cases are added to the official WHQL HLK playlist for the OS release that the HLK tests will ship with.
• Tests will be annoated to show which DXIL OpCode, LLVM Instructions, and HLSL operators they are intended to get coverage for.

Notes

• Private test binaries/collateral will be shared with IHVs for validation purposes. This will enable IHVs to verify long vector functionality without waiting for an OS/HLK release.

HLSL-Operators

✅ - Means there was an explicit test case implemented for the intrinsic. ☑️ - Means the intrinsic gets coverage via other intrinsics. For example ’exp2’ just uses the DXIL Opcode for Exp.

HLSL Operators

These operators generate LLVM instructions which use vectors.

Operator table from Microsoft HLSL Operators

Mappings of HLSL Intrinsics to DXIL OpCodes or LLVM Instructions

Trigonometry

Math

Float Ops

Bitwise Ops

Logic Ops

Reductions

Derivative and Quad Operations

WaveOps

Type Casting Operations