Copilot Causal Toolkit

The Copilot Causal Toolkit enables you to run causal inference analysis with Viva Insights data. For example, this can be used to address:

Two key scenarios - Seller Productivity and Burnout Prevention - are covered in this toolkit. Both involve using Copilot usage (Total_Copilot_actions_taken) as a treatment variable, and evaluating the treatment effect of using Copilot on an outcome variable (e.g. external or after-hours collaboration hours).

You can access the toolkit here on GitHub.

Python and Jupyter notebooks (.ipynb) are used for running the analysis. It is therefore recommended that you read the pre-requisites section and ensure that all of these are satisfied.

Directory

The copilot-causal-toolkit directory is organized as follows:

copilot-causal-toolkit/
├── data/                          # Directory for input data files
│   └── (Place your CSV files here - Person Query or Super Users Report exports)
│
├── script/                        # Main analysis scripts and modules
│   ├── CI-DML_AftCollabHours_PQ.ipynb     # After-hours collab analysis (Person Query)
│   ├── CI-DML_AftCollabHours_SUR.ipynb    # After-hours collab analysis (Super Users Report)
│   ├── CI-DML_ExtCollabHours_PQ.ipynb     # External collab analysis (Person Query)
│   ├── CI-DML_ExtCollabHours_SUR.ipynb    # External collab analysis (Super Users Report)
│   └── modules/                   # Helper modules for DML analysis
│       ├── data_processor.py      # Data cleaning and preprocessing functions
│       ├── data_filter.py         # Data filtering and aggregation utilities
│       ├── estimator.py           # DML estimation and causal inference functions
│       ├── output_manager.py      # Output generation functions
│       ├── subgroup_analysis.py   # Subgroup identification and comparison
│       ├── sensitivity_analysis.py # E-values and Rosenbaum bounds
│       └── visualizations.py      # Plotting and visualization functions
│
├── output/                        # Directory for analysis outputs
│   └── (Analysis results, plots, and reports will be saved here)
│
└── README.md                      # Documentation file

Key directories:

This directory contains four main Jupyter notebooks (.ipynb) for running causal inference analysis on top of Viva Insights data. Two work directly with a Person Query (PQ) schema:

And two of these work from an output from the Super Users Report (SUR):

We recommend creating a copy of the template that you wish to use and renaming them to match the scenario of your analysis.

Getting Started

Step 1: Download or Clone the Repository

First, you’ll need to get a copy of the causal inference package on your local computer:

Option A: Download as ZIP (Recommended for beginners)

  1. Click the green “Code” button at the top of the repository
  2. Select “Download ZIP”
  3. Extract the ZIP file to a location on your computer (e.g., C:\Users\YourName\Documents\repository-name)
  4. Navigate to the copilot-causal-toolkit subdirectory within the extracted folder
  5. Remember this copilot-causal-toolkit location—it will be your working directory

Option B: Clone with Git (If you have Git installed)

git clone [repository-url]
cd [repository-name]/copilot-causal-toolkit

Note: The copilot-causal-toolkit folder may be located within a larger repository. Make sure you’re working from the copilot-causal-toolkit subdirectory specifically, as this is where the analysis notebooks and modules are located.

Step 2: Open the Project in Your Editor

Using Visual Studio Code (Recommended):

  1. Open VS Code
  2. Go to File → Open Folder...
  3. Navigate to and select the copilot-causal-toolkit folder specifically (not the parent repository)
    • If you cloned the full repository, make sure to navigate into the copilot-causal-toolkit subdirectory
    • Example path: C:\Users\YourName\Documents\repository-name\copilot-causal-toolkit
  4. VS Code will now show the copilot-causal-toolkit structure in the sidebar (with data/, script/, output/ folders)

Important: Make sure you open the copilot-causal-toolkit folder itself, not the parent repository folder. This ensures all relative file paths in the notebooks work correctly.

Once your project is open, you can proceed to set up the prerequisites and data.

Setting up

Pre-requisites

Before running the causal inference analysis, ensure you have the following installed on your computer:

1. Python Installation

You’ll need Python 3.8 or higher (Python 3.9 or 3.10 recommended).

To check if Python is already installed:

Open a terminal (Command Prompt or PowerShell on Windows, Terminal on Mac) and run:

python --version

If you don’t have Python or need to upgrade, download it from python.org. During installation, make sure to check the box that says “Add Python to PATH”.

2. Jupyter Notebook

Jupyter allows you to run .ipynb notebook files interactively. Install it using:

pip install jupyter

Alternatively, you can use Visual Studio Code with the Python and Jupyter extensions, which provides a more integrated experience.

3. Required Python Packages

The analysis requires several Python packages. Copy and paste the following command to install all required packages at once:

pip install numpy pandas matplotlib scipy scikit-learn econml vivainsights

Package purposes:

Installation time: This may take 5-10 minutes depending on your internet connection.

4. Verify Installation

To verify all packages are installed correctly, run:

import numpy
import pandas
import matplotlib
import scipy
import sklearn
import econml
import vivainsights
print("✓ All packages installed successfully!")

You can run this in a Python terminal or in a Jupyter notebook cell.

If you’re familiar with Python environments, we recommend creating a virtual environment to avoid package conflicts:

# Create a virtual environment
python -m venv venv

# Activate it (Windows)
venv\Scripts\activate

# Activate it (Mac/Linux)
source venv/bin/activate

# Then install packages
pip install numpy pandas matplotlib scipy scikit-learn econml vivainsights jupyter

Downloading the data

There are two means of obtaining data for this causal inference analysis:

  1. export a Person Query as a csv file from Viva Insights
  2. export a csv file from an existing Super Users Report (if available)

For best results, we generally recommend the first method as this ensures that we have comprehensive coverage of all the covariates required in order to produce a causal inference result that we can be confident in. The second method can be quicker and immediately extracted from a Super Users report as it does not require the running and setting up of a fresh query.

Method 1: Export query from Viva Insights

  1. Open: https://analysis.insights.viva.office.com/Analysis/CreateAnalysis
  2. Select Person Query → ‘Set up analysis’.
  3. Configure:
    • Time period: Last 6 months (rolling)
    • Group by: Week
    • Metrics: See sub-step 4 for required attribute selection.
    • Filter: Is Active = True (if available) - You can validate the number of employees here.
    • Attributes: Include Organization and Function Type (others optional) - this is the last box on this page.
  4. Select required metrics (see below section on what columns to include)
  5. Save and Run query. Wait until Status = Completed.

Method 2: Export query from Super Users Report

This second method assumes that you already have a Super Users Report (.pbit/.pbix) populated with data from Viva Insights. The Super Users Report contains pre-aggregated data that can be quickly exported without setting up a new Person Query.

Steps to export data from Power BI:

  1. Open your Super Users Report in Power BI Desktop
    • If you don’t have the report file, ask your Viva Insights admin or check your organization’s shared workspace
  2. Navigate to the data view
    • In Power BI Desktop, click the Data icon (table icon) on the left sidebar
    • This shows all tables in your report
  3. Find the appropriate data table
    • Look for a table containing person-level data with columns like:
      • PersonId
      • Date (the metric date/week)
      • Total_Copilot_actions_taken
      • Collaboration metrics (hours, meeting hours, etc.)
      • Organizational attributes
    • Common table names: PersonMetrics, UserData, or similar
  4. Export the table to CSV
    • Option A: Right-click on the table name → Select “Copy table” → Paste into Excel → Save as CSV
    • Option B: Click on the table → Go to Home tab → Click “Transform data” to open Power Query Editor → In Power Query: Right-click the table → “Export” → Choose CSV format
  5. Verify the export
    • Open the CSV file to ensure it contains:
      • Multiple weeks of data (ideally 12-26 weeks)
      • All required columns listed in the scenarios below
      • No excessive blank rows or formatting issues
  6. Save to the data folder
    • Move the exported CSV file to copilot-causal-toolkit/data/
    • Rename it something descriptive (e.g., SuperUsersReport_Export_2025.csv)

Important notes for Super Users Report data:

Alternative: Export from Power BI Service (Online)

If your report is published to Power BI Service (online):

  1. Open the report at powerbi.com
  2. Navigate to the relevant page showing the data table
  3. Click the “…“ (More options) on a visual showing the underlying data
  4. Select “Export data” → Choose “Underlying data” → “.csv” format
  5. Download and save to copilot-causal-toolkit/data/

Note: Some Power BI reports may have export restrictions depending on your organization’s policies. Contact your Power BI admin if you cannot export data.

Columns to include: Seller Productivity Scenario

In this scenario, the goal is to understand how increasing Copilot usage increases or decreases time spent collaborating with external stakeholders and customers.

Columns to include: Burnout Prevention Scenario

In this scenario, the goal is to understand how increasing Copilot usage increases or decreases after-hours work patterns, which can impact employee wellbeing and burnout risk.

To examine the organizational or HR attributes you have in your dataset, you can run the following scripts to explore them:

hrvar_str = vi.extract_hr(data, return_type = 'vars').columns

for hr_var in hrvar_str:
    hrvar_table = vi.hrvar_count(data, hrvar = hr_var, return_type = 'table')
    print(f"\nValue counts for {hr_var}:")
    print(hrvar_table)

for hr_var in hrvar_str:
    vi.hrvar_count(data = data, hrvar = hr_var, return_type = 'plot')

Changing parameters in the scripts

Before running the analysis, you’ll need to customize a few parameters in the notebook to match your data and organizational structure. Open your chosen notebook and look for these configuration sections:

1. File Paths (Setup and Imports)

Update the data file path to match your actual CSV filename:

# Update this line to match your data file name
data_file_path = os.path.join(script_dir, '..', 'data', 'PersonQuery.Csv')
# For example: 'MyCompany_PersonQuery_2025.csv' or 'SuperUsersReport_Export.csv'

# Update the output directory name
output_base_dir = os.path.join(script_dir, '..', 'output', 'Subgroup Analysis - [YOUR COMPANY]')
# Replace [YOUR COMPANY] with your organization name, e.g., 'Contoso' or 'Fabrikam'

2. Analysis Configuration (For After-Hours notebooks only)

For after-hours collaboration analysis, decide whether you want to find subgroups with negative effects (reductions) or positive effects (increases):

# Toggle to control whether to find subgroups with NEGATIVE or POSITIVE effects
FIND_NEGATIVE_EFFECTS = True  # Set to True for reductions, False for increases

3. Organizational Attributes (Variable Definitions)

Update these variable lists to match the column names in your data. Look for sections defining:

SUBGROUP_VARS - Organizational attributes for subgroup analysis and heterogeneity analysis:

SUBGROUP_VARS = [
    'FunctionType',      # e.g., 'Function', 'Department', 'Division'
    'IsManager',         # Typically consistent across organizations
    'LevelDesignation',  # e.g., 'Level', 'Grade', 'Band'
    'Organization'       # Update these to match your data
]

Note: SUBGROUP_VARS serves a dual purpose - these variables are used both for creating person-level aggregations and for identifying subgroups with heterogeneous treatment effects. Include 2-4 key demographic/organizational attributes that you want to analyze.

NETWORK_VARS - Network metrics (usually consistent, but verify):

NETWORK_VARS = [
    'Internal_network_size',
    'External_network_size',
    'Strong_ties',
    'Diverse_ties'
]

COLLABORATION_VARS - Behavioral metrics (usually consistent):

COLLABORATION_VARS = [
    'Collaboration_hours',
    'Available_to_focus_hours',
    'Active_connected_hours',
    'Uninterrupted_hours'
]

4. Treatment and Outcome Variables (Usually no changes needed)

These are typically standard, but verify they exist in your data:

5. Quick Checklist Before Running

Running the analysis

Once everything is in place, you can choose to either run the notebook in its entirety or cell-by-cell.

If you’re running this for the first time, we strongly recommend running cell-by-cell. This allows you to:

How to run cell-by-cell:

Running All Cells at Once

Once you’re confident the notebook is configured correctly, you can run all cells:

Expected runtime: 10-30 minutes depending on data size (larger datasets take longer).

Troubleshooting Common Errors

If you encounter errors while running the notebook, here’s how to resolve them:

Error: “FileNotFoundError: [Errno 2] No such file or directory”

Error: “KeyError: ‘ColumnName’” or “Column not found”

Error: “ValueError: could not convert string to float”

Error: “MemoryError” or notebook becomes unresponsive

Using GitHub Copilot for Help

If you have GitHub Copilot or Copilot Chat installed in VS Code:

  1. Select the error message or problematic code
  2. Open Copilot Chat (Ctrl+Shift+I or Cmd+Shift+I)
  3. Ask specific questions like:
    • “Why am I getting this error?”
    • “How do I fix this KeyError for the column ‘Organization’?”
    • “Explain what this cell is doing”
    • “How do I change this variable list to use different column names?”

Copilot can help explain error messages, suggest fixes, and guide you through modifying the code to match your data structure.

Still Stuck?

Interpreting the outputs

After running the analysis successfully, you’ll find results in the output/ directory. The structure looks like this:

output/
└── Subgroup Analysis - [YOUR COMPANY]/
    ├── significant_subgroups_[timestamp].csv
    ├── sensitivity_analysis_results_[timestamp].json
    └── [SubgroupName]/
        ├── ate_plot_[timestamp].png
        ├── ate_results_[treatment]_[timestamp].csv
        ├── definition.txt
        └── transition_matrix_[treatment]_[timestamp].csv

Main Output Files

1. Significant Subgroups CSV (significant_subgroups_[timestamp].csv)

This file summarizes which subgroups show the strongest treatment effects (positive or negative).

Key columns:

How to interpret:

Business interpretation:

2. Subgroup-Specific Directories

For each significant subgroup, you’ll find a dedicated folder with detailed results:

a. ATE Plot (ate_plot_[timestamp].png)

A visualization showing the dose-response curve: how the outcome changes at different levels of Copilot usage.

What to look for:

Example interpretation:

b. ATE Results Table (ate_results_[treatment]_[timestamp].csv)

Detailed numeric results for each treatment level analyzed.

Key columns:

How to use:

c. Definition File (definition.txt)

A plain-text description of:

Purpose: Makes it easy to communicate findings to stakeholders (“This analysis looked at Sales managers with 500+ employees…”)

d. Transition Matrix (transition_matrix_[treatment]_[timestamp].csv)

Shows how people in this subgroup move between different Copilot usage levels over time.

What it reveals:

Example: If the matrix shows most people moving from “Low” → “Medium” usage, this suggests successful adoption.

3. Sensitivity Analysis Results (sensitivity_analysis_results_[timestamp].json)

A JSON file containing robustness checks that test whether results hold under different assumptions.

What’s tested:

How to interpret:

Note: This is more technical—focus on the main ATE results unless you need to defend methodological choices.

Summary: What to Focus On

For a quick analysis:

  1. Start with significant_subgroups_[timestamp].csv to identify which groups show effects
  2. Dive into each subgroup’s ate_plot_[timestamp].png to visualize the dose-response
  3. Use ate_results_[timestamp].csv for precise numbers to share with stakeholders
  4. Reference definition.txt when explaining which populations were analyzed

Making Sense of the Results

Statistical Significance:

Practical Significance:

Next Steps After Reviewing Results:

Methodology

This analysis uses Double Machine Learning (DML) with two-way fixed effects residualization to estimate the causal effect of Copilot usage on workplace outcomes. This section provides an overview of the methodology for those interested in understanding the technical approach.

Overview: What is Causal Inference?

Traditional correlation analysis can tell us that Copilot users have different collaboration patterns, but it cannot tell us whether Copilot causes those differences. People who adopt Copilot early might already be different from non-adopters in ways that affect their work patterns.

Causal inference methods allow us to estimate what would happen if we increased someone’s Copilot usage, accounting for:

The Double Machine Learning (DML) Framework

We use two primary DML estimators in this analysis:

1. LinearDML - Average Treatment Effects (ATE)

Purpose: Estimate the average dose-response relationship across all individuals.

How it works:

  1. Uses machine learning to model both:
    • How Copilot usage depends on covariates (propensity modeling)
    • How outcomes depend on covariates (outcome modeling)
  2. Removes the predictable parts using these models (residualization)
  3. Estimates the causal effect from the remaining “unexplained” variation

Key features:

Output: A dose-response curve showing how outcomes change at different Copilot usage levels (e.g., 5, 10, 15, 20 actions/week)

2. CausalForestDML - Heterogeneous Treatment Effects (CATE)

Purpose: Identify which subgroups experience different treatment effects (heterogeneity analysis).

How it works:

  1. First applies the same residualization as LinearDML
  2. Uses a random forest to discover which combinations of demographic attributes predict larger/smaller effects
  3. Identifies subgroups automatically without pre-specifying which groups to analyze

Key features:

Output: Rankings of subgroups by treatment effect magnitude, with confidence intervals for each group

Data Aggregation Approach

Before running DML, the notebooks aggregate the longitudinal data by person:

Why this matters:

Aggregation approach: For each person, we:

  1. Calculate the mean of all numeric variables (treatment, outcome, controls) across all observed weeks
  2. Keep demographic attributes (which are time-invariant by nature)
  3. Result is one row per person with their average metrics

What this means for interpretation:

Key assumptions:

  1. Unconfoundedness: We observe all important factors that affect both Copilot usage and outcomes (addressed by including comprehensive behavioral and demographic covariates)
  2. Positivity: Every individual has some probability of any usage level (checked via treatment distribution plots)
  3. Stable Unit Treatment Value (SUTVA): One person’s Copilot usage doesn’t affect another’s outcomes (potential concern with team-level spillovers)

Model Specifications Used

Machine learning models for nuisance functions:

Treatment featurization:

Control variables included:

Interpreting the Results

What the ATE tells you:

What the CATE tells you:

Confidence intervals:

Robustness and Sensitivity

The analysis includes several robustness checks:

  1. Alternative model specifications: Re-run with different ML algorithms
  2. Different control variables: Test sensitivity to covariate selection
  3. Varying featurization: Compare spline vs. polynomial treatment modeling
  4. Subsampling: Check if results are driven by outliers or specific time periods

Results stored in sensitivity_analysis_results_[timestamp].json document how stable the findings are across these variations.

Limitations and Caveats

What this analysis CAN tell you:

What this analysis CANNOT tell you:

Recommended interpretation: