knitr::opts_chunk$set(warning = FALSE, message = FALSE)
library(tidyverse)
library(vivainsights)

Evaluating a Workplace Intervention

Introduction

Organisations frequently run interventions aimed at improving how people work: a protected “focus day”, a meeting-reduction push, or a Microsoft 365 Copilot enablement wave for one team. The natural question is did it actually work? - and, just as importantly, did anything simply move somewhere else?

Answering this credibly needs more than a before-and-after comparison for the group that took part. A company-wide trend, a seasonal effect, or a change in how a metric is calculated can all masquerade as a programme effect. The antidote is a carefully designed comparison. The following simulation illustrates the arithmetic:

  • Compare a treated group (who received the intervention) against a control group (who did not).
  • Split time into Before, During, and After windows.
  • Use a difference-in-differences read: the treated group’s change minus the control group’s change.
  • Compare both groups’ changes, while remembering that common trends and intervention spillovers cannot be separated by subtraction alone.

We demonstrate the arithmetic on the built-in pq_data sample Person Query with an injected effect. This is an educational simulation, not a supported real-data causal recipe. No actual intervention is observed, and no confidence interval or identification check is implemented. Parallel trends, stable composition, treatment selection, coverage and absence of spillovers would need a separate study design.

Data preparation

pq_data is a weekly Person Query. We assign a treated group (here, one organisation that we imagine received the intervention) and a control group (everyone else), then split the weeks into three equal windows.

data("pq_data", package = "vivainsights")

pq <- pq_data %>% mutate(MetricDate = as.Date(MetricDate))

# Treated vs control. In practice you would flag the population that actually
# received the intervention; here we use one organisation for illustration.
TREATED_ORG <- "IT"
pq <- pq %>%
  mutate(group = ifelse(Organization == TREATED_ORG, "Treated", "Control"))

# Before / During / After as three equal windows across the date range.
rng  <- range(pq$MetricDate)
cuts <- rng[1] + diff(rng) * c(1/3, 2/3)
pq <- pq %>%
  mutate(period = factor(
    case_when(
      MetricDate <  cuts[1] ~ "Before",
      MetricDate <  cuts[2] ~ "During",
      TRUE                  ~ "After"
    ),
    levels = c("Before", "During", "After")
  ))

count(pq, group, period)
## # A tibble: 6 × 3
##   group   period     n
##   <chr>   <fct>  <int>
## 1 Control Before  1856
## 2 Control During  1624
## 3 Control After   1856
## 4 Treated Before   544
## 5 Treated During   476
## 6 Treated After    544

The sample data contains no real intervention, so purely for demonstration we inject a modest, clearly labelled reduction in multitasking for the treated group during and after the imaginary programme. This injected change is a teaching device, not an observed programme effect.

pq <- pq %>%
  mutate(Multitasking_hours = case_when(
    group == "Treated" & period == "During" ~ Multitasking_hours * 0.92,
    group == "Treated" & period == "After"  ~ Multitasking_hours * 0.80,
    TRUE                                     ~ Multitasking_hours
  ))

Two-stage aggregation

A robust group summary aggregates in two stages: first to a typical value per person (so a few very heavy or very light weeks do not dominate), then to a mean across people within each group and period. We wrap this in a small reusable function.

two_stage_summary <- function(data, metric,
                              id = "PersonId", group = "group",
                              period = "period") {
  # Stage 1: person-level mean within each period
  s1 <- data %>%
    group_by(.data[[id]], .data[[group]], .data[[period]]) %>%
    summarise(person_mean = mean(.data[[metric]], na.rm = TRUE), .groups = "drop")

  # Stage 2: group mean across persons within each period
  s1 %>%
    group_by(.data[[group]], .data[[period]]) %>%
    summarise(value = mean(person_mean, na.rm = TRUE),
              n_persons = n(), .groups = "drop")
}

summ <- two_stage_summary(pq, metric = "Multitasking_hours")
summ
## # A tibble: 6 × 4
##   group   period value n_persons
##   <chr>   <fct>  <dbl>     <int>
## 1 Control Before  4.89       232
## 2 Control During  5.27       232
## 3 Control After   5.20       232
## 4 Treated Before  4.70        68
## 5 Treated During  4.47        68
## 6 Treated After   4.38        68

Difference-in-differences

Now the key comparison. We look at each group’s change from Before to After, and take the difference between them. The control group’s change captures whatever was happening in that group; subtracting it produces a descriptive contrast. It isolates a causal effect only under assumptions not established by this example.

wide <- summ %>%
  select(group, period, value) %>%
  pivot_wider(names_from = period, values_from = value) %>%
  mutate(change_before_after = After - Before)

wide
## # A tibble: 2 × 5
##   group   Before During After change_before_after
##   <chr>    <dbl>  <dbl> <dbl>               <dbl>
## 1 Control   4.89   5.27  5.20               0.314
## 2 Treated   4.70   4.47  4.38              -0.322
did <- wide$change_before_after[wide$group == "Treated"] -
       wide$change_before_after[wide$group == "Control"]

did  # difference-in-differences (treated change minus control change)
## [1] -0.6359998

A picture makes the story immediate: the treated line should step down while the control line stays broadly flat.

ggplot(summ, aes(x = period, y = value, colour = group, group = group)) +
  geom_line(linewidth = 1) +
  geom_point(size = 2) +
  labs(
    title = "Weekly multitasking hours by period",
    subtitle = "Treated vs control, two-stage person-then-group means",
    x = NULL, y = "Multitasking hours / person / week", colour = NULL
  ) +
  theme_minimal(base_size = 12)

Reading the result honestly

The descriptive difference-in-differences is more informative than the treated group’s before/after change alone, but it is not automatically causal. A similar control change may reflect a common trend, measurement change or spillover; this comparison alone cannot distinguish them.

Two further checks are worth building in as a habit:

  • Look for displacement. Re-run the same summary on adjacent behaviours (for a meeting-reduction programme: after-hours and weekend collaboration hours) to confirm the load did not simply move elsewhere. “It went down and did not come back somewhere else” is the claim a leader actually needs.
  • Prefer two-stage aggregation. Person-then-group means stop a handful of extreme individuals or unequal week counts from skewing the comparison.

Transferable practices: always include a control population; structure the data as Before / During / After from the weekly person query; aggregate in two stages; and discount any signal that also moves in the control.

These summaries teach person-weighted aggregation and a descriptive contrast. Do not substitute a tenant query and claim an intervention effect without a separately reviewed design and an appropriate uncertainty estimate.