A difference-in-differences “metric scan”

Introduction

A frequent request in Copilot Analytics is a single table that answers: across all our collaboration metrics, on which ones do heavier Copilot users change their behaviour, by how much, and which of those changes are statistically significant?

A naive version simply compares group means (Power users send more emails than Low users) but that is cross-sectional and confounded, because Power users may differ for many reasons. This notebook instead runs a within-person difference-in-differences (DiD) model per metric and assembles the results into one tidy, sortable table plus a forest plot. The design compares two both-licensed groups (Power vs Low Copilot users), so the contrast is usage intensity, not licence access.

For a full walk-through of a single-metric event-study and the parallel-trends assumption, see the companion event-study-did example. Here the focus is the scan across metrics and honest reporting of significance, including metrics that do not move.

This is an educational, homogeneous-effect simulation, not a supported real-data causal recipe. Groups and adoption timing are assigned by the simulator, not inferred from outcomes. Real observational intensity groups may be selected on post-treatment behaviour. First observed activity need not be actual adoption. Eligibility, coverage, selection, attrition, parallel trends, spillovers and heterogeneous staggered effects require a separate design review. Fixed effects and BH adjustment do not establish causal identification.

Set-up

Install fixest with install.packages("fixest") if needed.

knitr::opts_chunk$set(warning = FALSE, message = FALSE)
library(dplyr)
library(tidyr)
library(ggplot2)
library(scales)
library(purrr)
library(fixest)

Analysis configuration is independent of the simulator’s injected effects.

METRICS <- c("Emails_sent", "Chats_sent", "Meeting_hours",
             "Collaboration_hours", "After_hours_collaboration_hours",
             "Channel_message_posts")
WINDOW <- 8L

Simulate a Person-Query-shaped panel with two intensity groups

Everyone here is a licensed adopter with an adoption week; they differ in intensity, where Power users ramp to heavy Copilot use while Low users stay light. We simulate several standard Person Query collaboration columns and inject a different post-adoption change for Power vs Low on each metric. Some differentials are large, some small, and one is zero, so the scan will show a realistic mix of significant and non-significant results.

The simulated panel explicitly supplies person, calendar week, assigned group and adoption week. A Person Query alone does not establish these design inputs.

set.seed(202)

n_persons <- 500L
n_weeks   <- 40L
weeks     <- seq(as.Date("2024-01-01"), by = "week", length.out = n_weeks)

# ---- clearly-labelled illustrative effects --------------------------------
# DEMO ONLY: the extra post-adoption change for POWER users relative to LOW
# users, per metric. These values belong only to the simulator.
EFFECTS <- c(
  Emails_sent                     =  1.5,
  Chats_sent                      =  3.0,
  Meeting_hours                   =  0.4,
  Collaboration_hours             =  0.8,
  After_hours_collaboration_hours =  0.15,
  Channel_message_posts           =  0.0   # deliberately null
)
# ---------------------------------------------------------------------------

persons <- tibble(
  PersonId  = sprintf("P%04d", seq_len(n_persons)),
  group     = sample(c("Power User", "Low User"), n_persons,
                      replace = TRUE, prob = c(0.55, 0.45)),
  adopt_wk  = sample(10:26, n_persons, replace = TRUE),
  base_lvl  = rnorm(n_persons, 0, 1)          # person collaboration baseline
)

grid <- tidyr::crossing(PersonId = persons$PersonId, week_idx = seq_len(n_weeks)) |>
  left_join(persons, by = "PersonId") |>
  mutate(
    MetricDate = weeks[week_idx],
    post       = week_idx >= adopt_wk,
    is_power   = group == "Power User",
    season     = 1.2 * sin(2 * pi * week_idx / 26)
  )

# Baseline mean level per metric (rough Person Query scales)
base_mean <- c(Emails_sent = 22, Chats_sent = 30, Meeting_hours = 10,
               Collaboration_hours = 14, After_hours_collaboration_hours = 2,
               Channel_message_posts = 6)
noise_sd  <- c(Emails_sent = 5, Chats_sent = 7, Meeting_hours = 2.5,
               Collaboration_hours = 3, After_hours_collaboration_hours = 0.8,
               Channel_message_posts = 2)

simulate_metric <- function(m) {
  base_mean[[m]] + 3 * grid$base_lvl + grid$season +
    ifelse(grid$post & grid$is_power, EFFECTS[[m]], 0) +
    rnorm(nrow(grid), 0, noise_sd[[m]])
}

for (m in names(EFFECTS)) grid[[m]] <- simulate_metric(m)

panel <- grid |>
  select(PersonId, MetricDate, week_idx, group, adopt_wk, all_of(names(EFFECTS)))

head(panel)
## # A tibble: 6 × 11
##   PersonId MetricDate week_idx group      adopt_wk Emails_sent Chats_sent
##   <chr>    <date>        <int> <chr>         <int>       <dbl>      <dbl>
## 1 P0001    2024-01-01        1 Power User       16       37.2        34.4
## 2 P0001    2024-01-08        2 Power User       16       26.7        30.3
## 3 P0001    2024-01-15        3 Power User       16       32.8        42.3
## 4 P0001    2024-01-22        4 Power User       16       29.3        30.6
## 5 P0001    2024-01-29        5 Power User       16       19.0        32.5
## 6 P0001    2024-02-05        6 Power User       16        6.65       33.4
## # ℹ 4 more variables: Meeting_hours <dbl>, Collaboration_hours <dbl>,
## #   After_hours_collaboration_hours <dbl>, Channel_message_posts <dbl>

Build event time and the DiD indicators

Each person’s anchor is their adoption week; we keep a balanced ±8-week window. treated flags Power users, post flags weeks on/after adoption, and their product treat_post is the DiD term.

stopifnot(all(c("PersonId", "MetricDate", "week_idx", "adopt_wk", "group",
               METRICS) %in% names(panel)),
          !anyNA(panel[c("PersonId", "MetricDate", "week_idx", "adopt_wk", "group")]),
          !anyDuplicated(panel[c("PersonId", "MetricDate")]),
          setequal(unique(panel$group), c("Power User", "Low User")))

panel_es <- panel |>
  mutate(event_week = week_idx - adopt_wk) |>
  filter(event_week >= -WINDOW, event_week <= WINDOW) |>
  group_by(PersonId) |>
  filter(sum(event_week < 0) >= 4, sum(event_week > 0) >= 4) |>
  ungroup() |>
  mutate(
    treated    = as.integer(group == "Power User"),
    post       = as.integer(event_week >= 0),
    treat_post = post * treated
  )

panel_es |> distinct(PersonId, group) |> count(group)
## # A tibble: 2 × 2
##   group          n
##   <chr>      <int>
## 1 Low User     243
## 2 Power User   257

Run the DiD once per metric and collect the results

We fit y ~ post + treat_post | PersonId + MetricDate (person + week fixed effects, person-clustered SE) for every metric, then record the effect, its 95% CI, the p-value adjusted across the metric scan, significance stars, and the effect as a share of the Power group’s pre-adoption baseline.

The person-specific post main effect captures the common adoption change in both groups. With staggered dates it is not absorbed by calendar-week fixed effects. The time-invariant group main effect is absorbed by person fixed effects.

stars <- function(p) dplyr::case_when(
  p < 0.001 ~ "***", p < 0.01 ~ "**", p < 0.05 ~ "*", TRUE ~ "n.s."
)

run_did <- function(metric) {
  fit <- fixest::feols(
    as.formula(paste0(metric, " ~ post + treat_post | PersonId + MetricDate")),
    data = panel_es, cluster = ~PersonId
  )
  ct  <- fit$coeftable["treat_post", ]
  ci  <- confint(fit)["treat_post", ]
  base_pre <- mean(panel_es[[metric]][panel_es$treated == 1 & panel_es$post == 0],
                   na.rm = TRUE)
  tibble(
    metric          = metric,
    estimate        = ct[["Estimate"]],
    conf_low        = ci[[1]],
    conf_high       = ci[[2]],
    p_value         = ct[["Pr(>|t|)"]],
    baseline_pre    = base_pre,
    pct_of_baseline = ct[["Estimate"]] / base_pre
  )
}

results <- purrr::map_dfr(METRICS, run_did) |>
  mutate(
    p_adjusted = p.adjust(p_value, method = "BH"),
    sig = stars(p_adjusted)
  ) |>
  arrange(desc(estimate))

results |>
  mutate(
    estimate        = round(estimate, 3),
    `95% CI`        = sprintf("[%+.2f, %+.2f]", conf_low, conf_high),
    `% of baseline` = scales::percent(pct_of_baseline, accuracy = 0.1),
    p_value         = signif(p_value, 2),
    p_adjusted      = signif(p_adjusted, 2)
  ) |>
  select(Metric = metric, `Δ (units)` = estimate, `95% CI`,
         `% of baseline`, p = p_value, `p (BH)` = p_adjusted, Sig = sig) |>
  knitr::kable(caption = "Power vs Low DiD: one row per metric, sorted by effect")
Power vs Low DiD: one row per metric, sorted by effect
Metric Δ (units) 95% CI % of baseline p p (BH) Sig
Chats_sent 3.085 [+2.47, +3.70] 10.4% 0.0e+00 0.0e+00 ***
Emails_sent 2.044 [+1.62, +2.47] 9.4% 0.0e+00 0.0e+00 ***
Collaboration_hours 0.801 [+0.55, +1.05] 5.8% 0.0e+00 0.0e+00 ***
Meeting_hours 0.355 [+0.14, +0.58] 3.6% 1.6e-03 1.9e-03 **
After_hours_collaboration_hours 0.198 [+0.13, +0.27] 10.7% 1.0e-07 1.0e-07 ***
Channel_message_posts 0.153 [-0.02, +0.33] 2.6% 8.9e-02 8.9e-02 n.s.

Read the table as: within-person, Power users changed this metric by Δ more than Low users did after adoption. The Sig column separates signals from noise under this simulation’s assumptions. Even a deliberately null metric can be significant by chance; the generated table, not the injected effect, determines the displayed result. BH-adjusted p-values control false discoveries under their assumptions; the displayed 95% intervals are pointwise, not multiplicity-adjusted.

Forest plot

The forest plot shows each effect as a share of the Power baseline, with 95% confidence intervals. Intervals crossing the zero line are not statistically distinguishable from “no differential change”.

ggplot(results,
       aes(x = pct_of_baseline,
           y = reorder(metric, pct_of_baseline),
           colour = sig != "n.s.")) +
  geom_vline(xintercept = 0, colour = "grey50") +
  geom_pointrange(aes(xmin = conf_low / baseline_pre,
                      xmax = conf_high / baseline_pre),
                  linewidth = 0.7, size = 0.5) +
  scale_x_continuous(labels = scales::percent) +
  scale_colour_manual(values = c(`TRUE` = "#1b4965", `FALSE` = "#bc4b51"),
                      labels = c(`TRUE` = "Significant (BH p<0.05)", `FALSE` = "n.s."),
                      name = NULL) +
  labs(
    title    = "Power vs Low Copilot users: DiD effect by metric",
    subtitle = "Within-person change for Power relative to Low, as % of Power pre-adoption baseline (95% CI)",
    x = "Effect as % of pre-adoption baseline", y = NULL
  ) +
  theme_minimal(base_size = 12) +
  theme(legend.position = "top")

Wrapping up

The scan demonstrates estimation and multiplicity reporting under a known, homogeneous data-generating process. It does not validate a real-data causal effect or a heterogeneous staggered-adoption estimator. Do not simply replace the simulator with a query and interpret the output causally.