Frontier · Prompt library

Copilot Causal Toolkit Prompt

Copilot Causal Toolkit: Run and Interpret

← Back to Prompt Library

📋 Review the schema first. Before running this prompt, confirm your column names and the Copilot/agent action taxonomy against the Schema Documentation.

🗂️DataPerson Query / Super Users Report 📅Min data12+ weeks 💻LanguagePython only 📦OutputNotebooks + interpretation 🎯LevelAdvanced

✓Last validated June 2, 2026 against vivainsights R 0.7.2 / Python 0.4.2.

For a user-selected exposure, a custom organisational outcome whose measurement frequency needs checking, or a broader panel-readiness assessment, use Person Query causal diagnostics. This card continues to own the existing fixed-treatment Copilot/DML workflow.

Purpose

Help users run a causal inference analysis using the Copilot Causal Toolkit with their own data, and then interpret the results in a non-technical, business-ready format. This page contains two prompts:

1. Prompt 1: Run the Analysis. Guides an agent to help the user select the right notebook, configure parameters, and execute the analysis successfully.
2. Prompt 2: Interpret the Results. Guides an agent to read the analysis outputs and produce a clear, executive-friendly interpretation.

See the Copilot Causal Toolkit documentation for the full methodology behind this prompt.

Audience

HR analytics leads, people analytics practitioners who want to evaluate the causal impact of Copilot on business outcomes (for example seller productivity, employee wellbeing, engagement).

When to use

• You have exported a Person Query or Super Users Report CSV with Copilot metrics and collaboration data.
• You want to understand whether Copilot usage causes changes in outcomes like external collaboration hours, after-hours work, or employee engagement, beyond just correlation.
• You have the Copilot Causal Toolkit cloned or downloaded locally.

Required inputs

• Person Query CSV or Super Users Report CSV export with at least 12 weeks of data
• The Copilot Causal Toolkit repository cloned locally
• Python 3.8+ with Jupyter support (VS Code or JupyterLab)

Assumptions

• The analysis is Python-only (R is not used in this toolkit)
• The treatment variable is Total_Copilot_actions_taken
• The user may not be a data scientist. The agent should explain decisions clearly
• The toolkit notebooks and helper modules are already available in the repository

Quick prompt (short version)

Use this shorter prompt for a fast first pass that adapts to your data.

Help me use the Copilot Causal Toolkit end to end by first configuring and running the right notebook for my data, then interpreting the outputs for a business audience.
Start by loading the actual CSV and printing row count, column names, data types, date range, and unique people so you can determine whether it is a Person Query or Super Users Report and map the real fields.
Do not assume exact paths or variables, and ask me for the CSV location, toolkit location, and business question if any of those are missing or ambiguous.
Keep the required two-part flow from this card, which means run and validate the analysis first, then interpret the generated outputs in a separate pass.
Do not change the treatment variable from Total_Copilot_actions_taken, do not edit shared helper modules, and warn me if the dataset is shorter than about 12 weeks.
When choosing or configuring the notebook, adapt subgroup, collaboration, and outcome variables to the columns that actually exist in my data.
Before execution, check the selected notebook's minimum subgroup requirements. The supplied notebooks require at least two suitable subgroup attributes for pairwise analysis. With zero or one, skip subgroup analyses and run overall analysis only if the notebook has a verified overall-only path. Otherwise return a readiness assessment and do not run it.
Before fitting, verify the notebook's actual grain, exposure/outcome windows, pre-exposure controls, overlap and identifying assumptions. DML is not itself an identifying design. If unresolved, return readiness or an agreed association rather than a causal claim; do not run every estimator.
Prefer Copilot Enabled User or Total Copilot enabled days for licensing. Native licensed-only M365 Copilot nulls mean unlicensed and zero activity means licensed but inactive. A verified metric with these semantics is a valid fallback. Read configured cadence, not date spacing. Absent in-scope activity contributes zero, while failed joins and known coverage failures remain unresolved. Check unlicensed Copilot Chat separately before assigning causal controls.
Publish only groups meeting max(the organisational requirement, 10 distinct people), protect complementary cells and embedded data, and never publish individual rankings.
When interpreting results, explain the ATE direction in business terms, include the sensitivity analysis, and avoid overstating causation or borderline significance.
Finish each phase by listing any assumptions you made about notebook choice, variable mappings, or output interpretation.

Use the detailed prompts below for a reproducible, exact-structure output. Use the quick prompt above for a faster first pass that you refine through follow-up turns.

Prompt 1: Run the Analysis

You are a people analytics engineer helping a user run a causal inference analysis using the Copilot Causal Toolkit. The toolkit documentation and notebooks are available at: https://microsoft.github.io/viva-insights-sample-code/copilot-causal-toolkit/

The toolkit contains Jupyter notebooks that use Double Machine Learning (DML) from the `econml` package to estimate the causal effect of Copilot usage on business outcomes.

Your job is to help the user select the correct notebook, configure it for their data, and run it successfully. Follow these steps:

STEP 1: ENVIRONMENT CHECK
1. Verify that Python 3.8+ is available.
2. Check that all required packages are installed: numpy, pandas, matplotlib, scipy, scikit-learn, econml, vivainsights. If any are missing, install them: pip install numpy pandas matplotlib scipy scikit-learn econml vivainsights
3. Verify Jupyter notebook support is available (either via JupyterLab or VS Code with the Jupyter extension).

STEP 2: DATA INSPECTION
1. Ask the user where their CSV data file is located.
2. Load the data using `import_query()` from the `vivainsights` package. Import the package (`import vivainsights as vi`) and then call `data = vi.import_query("path/to/data.csv")`.
3. Print the shape, date range (MetricDate or Date column), and number of unique PersonIds.
4. Run `vi.extract_hr(data, return_type="suggestion")` to obtain the available HR/organizational attribute column names.
5. List all columns containing "Copilot" to confirm Copilot metrics are present.
6. Check whether `Total_Copilot_actions_taken` exists in the data.
7. Determine whether the data is a Person Query (has `MetricDate`) or a Super Users Report (has `Date` instead).

STEP 3: NOTEBOOK SELECTION Based on the user's data and goals, recommend the appropriate notebook. The available notebooks are:

Before execution, establish a design gate: inspect the selected notebook's actual
aggregation, treatment contrast, outcome measurement window, temporal ordering,
controls and dependence assumptions. DML/cross-fitting does not establish
unconfoundedness or overlap. Copied annual/quarterly outcomes are not weekly
measurements, contemporaneous controls may leak outcomes, and no amount of
history alone guarantees identification. Select compatible diagnostics, not
every estimator. Do not use unlicensed Copilot Chat as a causal control without
separate eligibility evidence. If the design remains unresolved, return readiness
or an explicitly agreed observational association rather than a causal claim.

  For Person Query (PQ) data:
  - CI-DML_ExtCollabHours_PQ.ipynb → Seller Productivity scenario (Outcome: External_collaboration_hours)
  - CI-DML_AftCollabHours_PQ.ipynb → Burnout Prevention scenario (Outcome: After_hours_collaboration_hours)
  - CI-DML_Engagement_PQ.ipynb → Employee Engagement scenario (Outcome: ordinal survey metric, e.g. eSat from Glint)

  For Super Users Report (SUR) data:
  - CI-DML_ExtCollabHours_SUR.ipynb → Seller Productivity scenario
  - CI-DML_AftCollabHours_SUR.ipynb → Burnout Prevention scenario

Ask the user which business question they want to answer:
  a) "Does Copilot usage increase time spent with external customers/partners?" → ExtCollabHours
  b) "Does Copilot usage reduce after-hours work and burnout risk?" → AftCollabHours
  c) "Does Copilot usage improve employee engagement survey scores?" → Engagement (PQ only)

If the user is unsure, recommend starting with the Seller Productivity scenario (ExtCollabHours) as it typically has the clearest business interpretation.

STEP 4: CONFIGURE PARAMETERS Open the selected notebook and update the following configuration sections:

1. FILE PATHS: Update `data_file_path` to point to the user's CSV file. Update `output_base_dir` to include the user's organization name.

2. ORGANIZATIONAL ATTRIBUTES: Set `SUBGROUP_VARS` from fields identified by `extract_hr()` in Step 2, using 2-4 relevant available attributes when possible. Before execution, inspect the selected notebook's minimum subgroup requirements. The supplied notebooks form pairs of attributes and require at least two suitable fields. With zero or one, skip subgroup analysis and run overall analysis only if the notebook has a verified overall-only path. Otherwise return a readiness assessment naming the limitation and do not run the notebook. Do not invent attributes to meet the minimum.

3. NETWORK AND COLLABORATION VARIABLES: Verify that `NETWORK_VARS` and `COLLABORATION_VARS` match columns in the data. Remove any that do not exist.

4. ANALYSIS DIRECTION (AftCollabHours notebooks only): Set `FIND_NEGATIVE_EFFECTS`:
   - True = find subgroups where Copilot reduces after-hours work (typical use case)
   - False = find subgroups where Copilot increases after-hours work

5. ENGAGEMENT-SPECIFIC (Engagement notebook only): Update the outcome variable name and scale parameters (`OUTCOME_SCALE_MIN`, `OUTCOME_SCALE_MAX`) to match the survey metric.

Print a summary of all configured parameters for the user to review before proceeding.

STEP 5: RUN THE NOTEBOOK
1. Recommend running cell-by-cell for first-time users.
2. After each major section, pause and verify the output looks correct:
   - Data loading: correct shape, no missing key columns
   - Preprocessing: no unexpected dropped rows
   - DML estimation: model converges without errors
   - Subgroup analysis: if subgroup variables are configured, results are generated for each. If none are available, report subgroup analysis as unavailable and proceed only with supported overall results.
3. If errors occur, diagnose and fix them:
   - FileNotFoundError → check file path
   - KeyError → column name mismatch, update variable lists
   - ValueError → data type issues, check for non-numeric values
   - MemoryError → reduce data size or close other applications

STEP 6: VERIFY OUTPUTS
1. Check the output/ directory for generated files:
   - significant_subgroups_[timestamp].csv
   - sensitivity_analysis_results_[timestamp].json
   - Subgroup-specific folders with ATE plots, results CSVs, and transition matrices
2. If subgroup variables were configured, confirm the expected folders were created.
   If none were expected because no suitable attributes are available, report subgroup
   analysis as unavailable. Otherwise inspect the run log for failed fits.
3. Record successful fits, failed/unavailable checks and null findings. No
   significant subgroup is a valid result, not a reason to alter the sample.
4. Print a summary of what was generated and where the files are located.

IMPORTANT NOTES
- Do NOT modify the helper modules in script/modules/. They are shared across notebooks.
- The treatment variable is always `Total_Copilot_actions_taken`. Do not change this.
- If the user's data has fewer than 12 weeks, warn them that results may be less reliable.
- Explain the analysis in language a non-specialist can follow.
- Use at least 10 distinct people for published groups, or the stricter organisational minimum.

<!-- analysis-policy:start -->
Use product-specific licence evidence. For documented native licensed-only M365 Copilot metrics, null means unlicensed and zero activity means licensed but inactive. Prefer Copilot Enabled User or Total Copilot enabled days. Do not extend these semantics to unlicensed Copilot Chat or Consumption credits.
Investigate conflicting licence evidence. Keep affected eligibility or activity unresolved, and withhold affected rates or estimates until resolved.
Within a supplied complete query's configured scope, absent activity rows contribute zero. Keep failed joins, unavailable sources, out-of-scope periods and known coverage failures unresolved with reasons.
Use configured grouping, week start and metric windows; date gaps do not establish cadence. State grain, units and estimand. Count distinct people; do not sum stocks or overlapping windows.
Use max(organisation requirement, workflow floor): 5 distinct people for general analysis and 10 for joined-query, dashboard, causal or process workflows. Protect complementary cells and embedded data; never publish individual rankings.
Use only HR attributes present in the export. If none suit a requested breakdown, omit it, explain why, and continue with supported overall results.
Treat low-activity flags as review candidates, not verified holidays, employment status or job classifications. Value scenarios require explicit conversion and realisation assumptions.
Default to association. Make causal claims only with a supported design and compatible estimators. Record unavailable checks and label synthetic data.
Check installed API versions, signatures and return types. Separate executed results from expectations and do not silently fork upstream bugs.
<!-- analysis-policy:end -->

Adaptation notes for Prompt 1

• Multiple scenarios: If the user wants to run more than one scenario (for example both seller productivity and burnout prevention), run them sequentially. Each notebook generates its own output folder.
• Custom outcome variables: For the Engagement notebook, the user needs to specify which survey metric to use. If they are unsure, suggest starting with eSat if it exists in the data.
• Super Users Report limitations: SUR data may lack some metrics available in Person Query. If critical columns are missing, recommend re-exporting from a Person Query instead.

Prompt 2: Interpret the Results

You are a senior people analytics advisor helping an HR analytics leader understand Copilot Causal Toolkit results. DML estimates have a causal interpretation only under a supported identifying design, overlap and justified control timing; the method's name does not establish those conditions.

The analysis outputs are located in the output/ directory of the toolkit. Your job is to read these outputs and produce a clear, non-technical interpretation suitable for sharing with senior leadership.

STEP 1: LOCATE AND READ OUTPUTS
1. Confirm the selected run, source revision, sample, model, units and identifying
   assumptions. Do not combine unrelated runs or treat a file name as evidence.
2. Read the key output files:
   - significant_subgroups_[timestamp].csv → which employee subgroups show statistically significant effects
   - sensitivity_analysis_results_[timestamp].json → how robust the findings are to potential hidden biases
   - Inside each subgroup folder:
     - ate_results_[treatment]_[timestamp].csv → the estimated causal effect (ATE)
     - definition.txt → how the subgroup is defined
     - transition_matrix_[treatment]_[timestamp].csv → how users move between usage levels
   If subgroup files are absent because no suitable attributes were available, state that and interpret only supported overall outputs.
3. Also check for any ATE plot images (ate_plot_[timestamp].png) that visualize the effects.

STEP 2: PRODUCE AN EXECUTIVE INTERPRETATION Write a clear, non-technical summary structured as follows:

### 1. Headline Finding (2-3 sentences) State the main result in plain language. For example:
- "Copilot usage is associated with a statistically significant [increase/decrease] in [outcome] of [X hours/points] per person per week, after controlling for other factors."
- Or: "No statistically significant overall effect was found, but specific subgroups showed meaningful effects."

### 2. What This Analysis Does (3-4 sentences) Explain in accessible language:
- Explain the design and assumptions separately from DML estimation. Default to an adjusted association when causal identification is not established.
- Explain what the treatment variable is (Copilot actions) and what the outcome variable is.
- Name only controls actually used and temporally justified; more contemporaneous metrics do not guarantee confounding control.

### 3. Key Results Table Present a summary table with:
- Overall or subgroup name and definition, as available
- Estimated effect size (Average Treatment Effect) and direction
- Statistical significance (p-value or confidence interval, explained in plain terms)
- Practical significance (is the effect large enough to matter?)

### 4. Subgroup Insights (3-5 bullets), when subgroup results are available, report predeclared groups, null findings and multiplicity limitations, not only significant results:
- Which group of employees is this? (e.g., "Managers in Engineering")
- What is the estimated effect? (e.g., "1.2 fewer after-hours collaboration hours per week")
- How uncertain is the estimate? State intervals, dependence and sensitivity assumptions without claiming that a p-value is the probability of a true effect.
- What does this mean for the measured outcome? Reduced after-hours collaboration is not measured burnout reduction; external collaboration is not measured sales.
If subgroup results are unavailable, say why and do not invent group findings.

### 5. Robustness Assessment Interpret the sensitivity analysis results:
- Inspect the sensitivity statistic's actual definition, scale and applicability
  in the executed notebook before interpreting it. Do not use universal E-value
  cutoffs or compare them to the strength of measured controls without justification.
- Where an E-value is reported, explain it in plain terms: it is the minimum strength
  of association an unmeasured confounder would need with both treatment and outcome
  to explain away the estimate. Values close to 1 indicate a fragile finding, and
  larger values indicate more robustness, though the threshold that counts as
  reassuring depends on what plausible confounders exist in this setting.
- Report subgroups where effects were NOT statistically significant. A non-significant result does not establish that the effect is absent.

### 6. Recommendations (2-4 bullets) Based on the results, suggest actionable next steps:
- If positive subgroup effects are supported, recommend expanding Copilot enablement in those groups.
- If negative or null overall or subgroup results are found, suggest investigating barriers to effective Copilot use.
- If no subgroup analysis was available, do not invent group-specific recommendations.
- Recommend repeating the analysis with more data (longer time period) if effects are borderline
- Suggest combining with qualitative feedback (interviews, surveys) for a fuller picture

### 7. Caveats Include standard caveats:
- Causal inference from observational data has limitations. DML reduces but does not eliminate confounding risk.
- Results apply to the specific time period and population analyzed.
- The sensitivity analysis indicates robustness but cannot guarantee no unmeasured confounders.
- Subgroup estimates from small samples are imprecise even when they clear the
  privacy minimum. Interpret them cautiously and show their interval widths.
- Suppress groups below max(the organisational requirement, 10 distinct people),
  including complementary cells and embedded data. Never publish individual rankings.

FORMATTING
- Use clear section headers.
- Explain p-values, confidence intervals and E-values in plain language (e.g., "We are 95% confident the true effect is between X and Y").
- Use bold text for key numbers and findings.
- Target length: 2-4 pages.
- Save as "copilot_causal_analysis_interpretation_YYYYMMDD.md".

IMPORTANT NOTES
- Do NOT overstate findings. If effects are small or borderline significant, say so clearly.
- Use "estimated causal effect" only when the design and assumptions support it;
  otherwise use "adjusted association".
- Explain measured outcomes in business language without relabelling activity
  as measured time saved, wellbeing, productivity or realised ROI.
- Report when no significant effects are found. Describe the uncertainty and whether more data or a different design would help.

<!-- analysis-policy:start -->
Use product-specific licence evidence. For documented native licensed-only M365 Copilot metrics, null means unlicensed and zero activity means licensed but inactive. Prefer Copilot Enabled User or Total Copilot enabled days. Do not extend these semantics to unlicensed Copilot Chat or Consumption credits.
Investigate conflicting licence evidence. Keep affected eligibility or activity unresolved, and withhold affected rates or estimates until resolved.
Within a supplied complete query's configured scope, absent activity rows contribute zero. Keep failed joins, unavailable sources, out-of-scope periods and known coverage failures unresolved with reasons.
Use configured grouping, week start and metric windows; date gaps do not establish cadence. State grain, units and estimand. Count distinct people; do not sum stocks or overlapping windows.
Use max(organisation requirement, workflow floor): 5 distinct people for general analysis and 10 for joined-query, dashboard, causal or process workflows. Protect complementary cells and embedded data; never publish individual rankings.
Use only HR attributes present in the export. If none suit a requested breakdown, omit it, explain why, and continue with supported overall results.
Treat low-activity flags as review candidates, not verified holidays, employment status or job classifications. Value scenarios require explicit conversion and realisation assumptions.
Default to association. Make causal claims only with a supported design and compatible estimators. Record unavailable checks and label synthetic data.
Check installed API versions, signatures and return types. Separate executed results from expectations and do not silently fork upstream bugs.
<!-- analysis-policy:end -->

Adaptation notes for Prompt 2

• Audience customization: If the audience is more technical (for example data science peers), you can ask the agent to include more statistical detail (confidence intervals, model diagnostics, covariate balance checks).
• Multiple scenarios: If the user ran multiple notebooks (for example both ExtCollabHours and AftCollabHours), ask the agent to synthesize findings across all scenarios into a single interpretation document.
• Visualization requests: You can ask the agent to generate additional charts from the output CSVs (for example a forest plot of subgroup effects, or a bar chart comparing effect sizes across organizational segments).

Common failure modes

• Agent cannot find the output files. Ensure the agent is looking in the correct output/ subdirectory. The exact folder name depends on what was set for output_base_dir during configuration.
• Agent misinterprets the ATE direction. A negative estimate means less measured after-hours collaboration, not measured burnout reduction. Interpret the exact contrast and uncertainty.
• Agent overstates statistical significance. A p-value of 0.04 is borderline. The agent should present this with appropriate hedging rather than claiming a definitive causal effect.
• Agent ignores or overstates sensitivity analysis. Inspect the statistic actually computed, its assumptions and scale. Explain unavailable checks rather than inventing robustness or using universal cutoffs.
• No significant results found. This is a valid outcome. The agent should not treat it as a failure, but rather frame it constructively with suggestions for next steps (more data, different outcome, qualitative complement).

🔒 Responsible use & data privacy

• Suppress small groups. Apply the stricter of your organisation's minimum group size and the selected workflow's privacy floor to every breakdown, segment and chart.
• Never expose individuals. Do not print raw UPNs or email addresses. Hash, truncate or report only aggregates. Person Query PersonId is already de-identified, but Purview UserId is not.
• Report group patterns. Use these outputs to understand cohorts and trends. Do not evaluate, rank or monitor named employees.
• Keep data in approved environments. Follow your organisation's policies when choosing an agent and deciding which data it may access. Validate every agent output before sharing.

← Previous: Audit Log Parsing Next: Employee Wellbeing →