# Measuring behaviour change with an event-study / difference-in-differences

## Introduction

A common question in Copilot Analytics is *causal*: **did a person's
collaboration behaviour actually change after they started using Copilot**, or
are heavy users simply different people to begin with? A plain comparison of
users vs non-users cannot separate the two, because adoption is not random.

A **within-person event-study** paired with a **two-way fixed-effects (TWFE)
difference-in-differences (DiD)** model is a standard way to address this. The
design:

* aligns every adopter on their own **event time** (weeks relative to the week
  they adopted Copilot),
* compares adopters against a control group of non-adopters over the same
  calendar weeks,
* absorbs each person's baseline (**person fixed effects**) and every calendar
  week's common shocks (**week fixed effects**), and
* reads the treatment effect as the *within-person* change in the treated group
  relative to the control group.

Person Query exports do not ship with a clean adoption event, so this notebook
builds a **small seeded simulation** whose column names match a real Person
Query. The downstream modelling code therefore runs on a real export loaded
with `vivainsights::import_query()`, so you swap out the data-generation chunk and
map your outcome columns (shown below). The simulation injects a **clearly-labelled
illustrative effect** so the model has something to recover; this is for
demonstration only and is not a real result.

## Set-up

This example uses **fixest** for fast fixed-effects estimation; install it with
`install.packages("fixest")` if you do not already have it.

```{r setup, message=FALSE, warning=FALSE}
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
library(dplyr)
library(tidyr)
library(ggplot2)
library(scales)
library(fixest)   # fast fixed-effects estimation (feols, i(), iplot)
```

To run this notebook on your **own** export instead of the simulation, replace
the simulation chunk below with the following, then continue unchanged from the
*"Derive the adoption week"* section:

```{r real-data, eval=FALSE, purl=FALSE}
panel <- vivainsights::import_query("your-person-query.csv")
app_cols <- grep("^Copilot_actions_taken_in", names(panel), value = TRUE)
panel <- panel %>%
  mutate(across(all_of(app_cols), ~ tidyr::replace_na(.x, 0))) %>%
  mutate(Total_Copilot_actions_taken = rowSums(across(all_of(app_cols))))
# choose the outcome column(s) to model, e.g. Collaboration_hours, Chat_hours,
# Emails_sent -- these are standard Person Query columns.
```

## Simulate a Person-Query-shaped panel

The block below creates a weekly panel with the same shape as a Person Query:
one row per `PersonId` x `MetricDate`, a `Total_Copilot_actions_taken` column,
and a few collaboration outcomes (`Collaboration_hours`, `Chat_hours`,
`Emails_sent`). Treated people adopt Copilot at a **staggered** week; control
people never adopt. We add person baselines, calendar-week seasonality, and
noise so the panel looks realistic.

> **Replace this chunk for real data.** Load your export with
> `vivainsights::import_query()` and derive the adoption week from the first week
> of non-zero Copilot actions (see the next section). Everything downstream stays
> the same.

```{r simulate}
set.seed(100)

n_persons <- 400L
n_weeks   <- 40L
weeks     <- seq(as.Date("2024-01-01"), by = "week", length.out = n_weeks)

# ---- clearly-labelled illustrative effect ---------------------------------
# DEMO ONLY: this is the size of the behaviour change we inject into treated
# people after adoption. Set to 0 (or delete) when running on real data, so the
# model should estimate the effect, not have it baked in.
TREATMENT_EFFECT <- 0.8   # extra collaboration hours/week once adopted
# ---------------------------------------------------------------------------

persons <- tibble(
  PersonId    = sprintf("P%04d", seq_len(n_persons)),
  treated     = rbinom(n_persons, 1, 0.6),
  # person baseline (some people simply collaborate more than others)
  base_collab = rnorm(n_persons, mean = 12, sd = 3)
)

# Treated people adopt at a staggered week (10..28); controls never adopt.
persons <- persons %>%
  mutate(adopt_week = ifelse(
    treated == 1,
    sample(10:28, n_persons, replace = TRUE),
    NA_integer_
  ))

# calendar-week seasonality shared by everyone
week_shock <- tibble(
  week_idx  = seq_len(n_weeks),
  MetricDate = weeks,
  season    = 1.5 * sin(2 * pi * seq_len(n_weeks) / 26)
)

panel <- tidyr::crossing(PersonId = persons$PersonId, week_idx = seq_len(n_weeks)) %>%
  left_join(persons, by = "PersonId") %>%
  left_join(week_shock, by = "week_idx") %>%
  mutate(
    adopted_now = treated == 1 & week_idx >= adopt_week,
    # Copilot actions: 0 before adoption, positive afterwards (treated only)
    Total_Copilot_actions_taken = ifelse(
      adopted_now, rpois(n(), lambda = 25), 0L
    ),
    # primary outcome: baseline + season + injected effect (post-adoption) + noise
    Collaboration_hours = base_collab + season +
      ifelse(adopted_now, TREATMENT_EFFECT, 0) +
      rnorm(n(), 0, 2),
    # two correlated collaboration outcomes for the composite index later
    Chat_hours  = 0.35 * Collaboration_hours + rnorm(n(), 0, 0.8),
    Emails_sent = 20 + 1.2 * Collaboration_hours + rnorm(n(), 0, 4)
  ) %>%
  select(PersonId, MetricDate, week_idx, treated,
         Total_Copilot_actions_taken,
         Collaboration_hours, Chat_hours, Emails_sent)

head(panel)
```

## Derive the adoption week and event time

On a real export you would not know who is "treated"; you would infer it from
the data. Here we reproduce that realistic step: a person is an **adopter** if
they ever record a non-zero Copilot action, and their **adoption week** is the
first such week. Control people (never any actions) are assigned a **placebo**
adoption week (the median adopter week), so they contribute a comparable
event-time window.

```{r event-time}
adopt <- panel %>%
  filter(Total_Copilot_actions_taken > 0) %>%
  group_by(PersonId) %>%
  summarise(adopt_date = min(MetricDate), .groups = "drop")

placebo_date <- as.Date(median(adopt$adopt_date))

panel <- panel %>%
  left_join(adopt, by = "PersonId") %>%
  mutate(
    is_adopter = !is.na(adopt_date),
    anchor     = dplyr::coalesce(adopt_date, placebo_date),
    event_week = as.integer((MetricDate - anchor) / 7)
  )

# Keep a balanced +/- 8-week window around each person's anchor
WINDOW <- 8L
panel_es <- panel %>%
  filter(event_week >= -WINDOW, event_week <= WINDOW) %>%
  group_by(PersonId) %>%
  filter(sum(event_week < 0) >= 4, sum(event_week > 0) >= 4) %>%
  ungroup() %>%
  mutate(
    treated_grp = as.integer(is_adopter),
    post        = as.integer(event_week >= 0),
    treat_post  = post * treated_grp
  )

panel_es %>%
  distinct(PersonId, treated_grp) %>%
  count(group = ifelse(treated_grp == 1, "Adopter (treated)", "Non-adopter (control)"))
```

## The headline TWFE difference-in-differences

The core model is:

$$y_{it} = \beta\,(\text{post}_{it}\times\text{treated}_i) + \alpha_i + \gamma_t + \varepsilon_{it}$$

where $\alpha_i$ are person fixed effects, $\gamma_t$ are calendar-week fixed
effects, and standard errors are clustered by person. The coefficient $\beta$ on
`treat_post` is the difference-in-differences estimate, the within-person change
for adopters, net of the control group and of anything common to a given week.

```{r twfe}
did <- fixest::feols(
  Collaboration_hours ~ treat_post | PersonId + MetricDate,
  data    = panel_es,
  cluster = ~PersonId
)

summary(did)

did_beta <- coef(did)[["treat_post"]]
cat(sprintf(
  "\nDiD estimate: %+.2f collaboration hours/week (injected demo effect was %+.2f)\n",
  did_beta, TREATMENT_EFFECT
))
```

The recovered coefficient should land close to the injected `TREATMENT_EFFECT`,
confirming the model is working. On real data you would *not* know the true
value, and that is the whole point of estimating it.

## The event-study: check for pre-trends and see the effect emerge

A single DiD number hides a crucial assumption: **parallel trends**. The
event-study version estimates a separate coefficient for each event week
(relative to week -1, the week before adoption). If the design is valid, the
**pre-adoption** coefficients should sit near zero (no divergence before
treatment), and the **post-adoption** coefficients should step up to the effect.

`fixest::i()` builds these event-time interactions directly, with `ref = -1` as
the omitted reference week.

```{r event-study, fig.width=9, fig.height=5}
es <- fixest::feols(
  Collaboration_hours ~ i(event_week, treated_grp, ref = -1) |
    PersonId + MetricDate,
  data    = panel_es,
  cluster = ~PersonId
)

# Tidy the event-time coefficients for a ggplot
es_tab <- as.data.frame(coeftable(es))
es_tab$term <- rownames(es_tab)
es_coefs <- es_tab %>%
  filter(grepl("event_week::", term)) %>%
  mutate(
    event_week = as.integer(sub(".*event_week::(-?\\d+).*", "\\1", term)),
    lwr = Estimate - 1.96 * `Std. Error`,
    upr = Estimate + 1.96 * `Std. Error`
  )

# Add the reference week (-1) at exactly 0
es_coefs <- bind_rows(
  es_coefs,
  tibble(event_week = -1, Estimate = 0, lwr = 0, upr = 0)
) %>% arrange(event_week)

ggplot(es_coefs, aes(event_week, Estimate)) +
  geom_hline(yintercept = 0, colour = "grey60") +
  geom_vline(xintercept = -0.5, linetype = "dashed", colour = "grey40") +
  geom_ribbon(aes(ymin = lwr, ymax = upr), fill = "#5fa8d3", alpha = 0.25) +
  geom_line(linewidth = 1, colour = "#1b4965") +
  geom_point(size = 2, colour = "#1b4965") +
  scale_x_continuous(breaks = seq(-WINDOW, WINDOW, 2)) +
  labs(
    title    = "Event-study: collaboration hours around Copilot adoption",
    subtitle = "Coefficients vs event week (ref = week -1), 95% CI. Flat before, steps up after.",
    x = "Weeks relative to adoption", y = "Effect vs week -1 (hours/week)"
  ) +
  theme_minimal(base_size = 12)
```

A flat pre-period and a clean post-period step is the visual signature of a
credible DiD. If the pre-period coefficients trended, you would be worried that
adopters were already diverging before Copilot and would treat the DiD estimate
with caution.

## Bonus: run the same design on a composite index

Individual metrics can be noisy. A robust alternative is to combine several
related collaboration metrics into a single **z-scored composite index**, then
run the identical DiD on the index. Each component is standardised across the
panel (mean 0, sd 1) and averaged, so no single metric's scale dominates.

```{r composite, fig.width=9, fig.height=4}
components <- c("Collaboration_hours", "Chat_hours", "Emails_sent")

zscore <- function(x) (x - mean(x, na.rm = TRUE)) / sd(x, na.rm = TRUE)

panel_es <- panel_es %>%
  mutate(across(all_of(components), zscore, .names = "z_{.col}")) %>%
  mutate(collab_index = rowMeans(across(starts_with("z_")), na.rm = TRUE))

did_index <- fixest::feols(
  collab_index ~ treat_post | PersonId + MetricDate,
  data    = panel_es,
  cluster = ~PersonId
)

summary(did_index)

# Event-time trajectory of the composite, treated vs control
idx_traj <- panel_es %>%
  mutate(grp = ifelse(treated_grp == 1, "Adopter", "Control")) %>%
  group_by(grp, event_week) %>%
  summarise(idx = mean(collab_index), .groups = "drop")

ggplot(idx_traj, aes(event_week, idx, colour = grp)) +
  geom_vline(xintercept = -0.5, linetype = "dashed", colour = "grey40") +
  geom_line(linewidth = 1) +
  geom_point(size = 1.8) +
  scale_colour_manual(values = c("Adopter" = "#1b4965", "Control" = "#bc4b51"),
                      name = NULL) +
  labs(
    title    = "Composite collaboration index around adoption",
    subtitle = "Mean z-scored index (Collaboration hours + Chat hours + Emails sent)",
    x = "Weeks relative to adoption", y = "Composite index (z-units)"
  ) +
  theme_minimal(base_size = 12) +
  theme(legend.position = "top")
```

## Wrapping up

The event-study + TWFE DiD pattern is a portable way to ask *"did behaviour
change after adoption, within the same person?"* without being fooled by the
fact that adopters differ from non-adopters. To apply it to your own data,
replace the simulation chunk with `vivainsights::import_query()`, derive the
adoption week from the first non-zero Copilot action, set `TREATMENT_EFFECT`
aside entirely, and let the model estimate the effect. Always inspect the
event-study pre-trends before trusting the single DiD number.
