Copilot Analytics Lab Β· PANDAS Team Β· November 2026
Causal Questions About AI Usage
Most AI rollouts are staggered, product-specific and led by function. This guide asks a short series of questions about your data and points you to a design that fits, or says plainly when none does.
Leaders ask a plain question: did AI use change how work gets done? The honest answer depends on how the data is structured. AI use now spans Microsoft 365 Copilot, Cowork, Agents and GitHub Copilot, each measured in a different query with different units, start dates and eligible people. Whether you can say anything causal depends on who was exposed, when, and what you can fairly compare them with.
The TL;DR
- Choose the outcome before the method. The outcome decides what a credible result can say.
- Staggered, product-specific, function-led rollouts are the normal case. They need estimators built for them, and comparisons within function wherever timing varies inside functions.
- Run the standard checks on every route where they apply, and say where they cannot be run.
- Report results on Viva Insights proxies as behavioural outcomes or candidate leading indicators, and keep business impact claims for outcomes that measure it.
The usual starting point
Most analyses of AI use share a setting. Licences are product-specific, and rollout tends to follow function, so function shapes both when people get access and how their outcomes move. The data is typically a person-week panel joined onto the Person Query. Copilot has been available for over two years, so many early adopters have no observed pre-period in the export, and before-and-after comparisons are limited to later cohorts and newer tools.
This article is for people analytics practitioners who hold Viva Insights exports and know regression, and who may be new to modern difference-in-differences. It routes you to a design and does not catalogue every method. It replaces the site's earlier technique-by-technique guide.
Key terms
Each term is also underlined where it first appears in the text.
- Anticipation
- People changing behaviour before access starts, for example after an announcement, which makes pre-period weeks partly treated.
- Clustering
- Allowing for correlated outcomes within a group, such as a function or team. Standard errors are usually clustered at least at the level at which exposure was assigned.
- Cohort and not-yet-treated
- A cohort is the group of people whose exposure started in the same week. Not-yet-treated people start later, so they are valid comparisons only for weeks before their own start.
- Conditional parallel trends
- The assumption that, among comparable people (here, within the same function), treated and comparison groups would have moved in parallel without treatment.
- Compliers (local average treatment effect)
- Compliers are the people whose use changes because of their assignment. Their effect, using assignment as an instrument, is the local average treatment effect.
- Estimand
- The exact quantity you want to estimate, such as the average effect of access for later licence cohorts in one function over stated weeks.
- Event study and leads
- A model that estimates effects for each week relative to the start date. Leads are the estimates for weeks before the start, and should be close to zero if the comparison is fair.
- Exclusion restriction and monotonicity
- Two assumptions for using assignment as an instrument. Exclusion says assignment affects the outcome only through use. Monotonicity says assignment never pushes anyone to use less.
- Exposure and treatment
- Exposure is a person's contact with an AI tool, such as a licence, first use or an amount of use. Treatment is the exposure whose effect you are estimating.
- Intention to treat
- Comparing people by assigned group, whether or not they used the tool. It estimates the effect of giving access.
- Negative control
- An outcome the tool should not plausibly change. An apparent effect on it signals confounding or a shared shock.
- Surrogate
- A short-term measure standing in for a longer-term outcome. It is valid only if it captures all of the effect on that outcome.
- Two-way fixed effects (TWFE)
- A regression with person and period fixed effects. With staggered start dates, one TWFE coefficient can use already-treated people as comparisons and give misleading weights.
- Unconfoundedness and overlap
- Unconfoundedness says that, given the covariates, exposure is unrelated to the outcome a person would have had without it. Overlap says comparable people exist at each exposure level.
Choose the outcome first
Pick the outcome before you pick a method. A method cannot repair an outcome that sits too far from the exposure. In practice the choice is between a business KPI and a Viva Insights proxy.
Business KPIs
Revenue, cases closed and cycle time are what leaders want to hear about, and they are the hardest outcomes to tie to AI use. Many functions contribute to a KPI, so one person's or function's AI exposure is a single contribution among many. The lag between exposure and the KPI can exceed the window you have. KPIs are usually recorded for teams or business units, monthly or quarterly, while exposure is individual and weekly, which leaves little statistical power. Attribution is rarely defensible when other tools, reorganisations and market conditions moved at the same time.
Viva Insights proxies
Measures such as External collaboration hours, After hours collaboration hours and Meeting hours are recorded per person, per week, and line up with exposure dates. That supports panel designs with many units and many periods. They carry their own risks.
- They measure behaviour. More
External collaboration hoursmay reflect more customer contact, but it is not necessarily more sales. It could equally reflect more rework or longer sales cycles. - Direction is ambiguous. Fewer meeting hours could mean efficiency or disengagement. Declare the expected direction before looking at results.
- The tool can move the metric directly. Meeting recap features can change meeting attendance, for example. Prefer proxies the tool cannot generate itself.
- Targets change behaviour. Once people are asked to improve a proxy, it can start to measure something different.
Proxies are often the only outcomes with enough weekly variation to analyse, so the task is to label them honestly. Once the outcome is settled, the decision guide helps you pick a design.
Find your route
Answer one question at a time. Each answer leads to the next question or to a route in Part 3, and you can step back to change an answer. With JavaScript turned off, the questions appear as a nested list.
Answer a few questions
Answer a few questions about your data and the guide points to a design. Every question and answer is listed below if scripting is unavailable.
-
What do you want to learn?
- Who uses AI, how much, and which tools: Describe use without a causal claim
- Whether giving people access changed an outcome: next question
- Whether using AI more changed an outcome: next question
-
Is your outcome measured repeatedly, before and after exposure, at a grain that matches it?
Weekly Viva Insights metrics usually qualify. Annual ratings copied to every week do not. Read more
- Yes, a weekly or monthly Viva Insights metric: next question
- Yes, a team-level KPI with enough teams and periods: next question Analyse it at team-period level. Do not copy team outcomes onto every person.
- No, it is measured once or only after exposure: next question
-
Your outcome is observed once. Was access assigned at random, or by a cut-off?
A single post-exposure outcome can still support a randomised comparison or a regression discontinuity.
- At random, for example a pilot lottery or randomised waves: Compare by assignment
- By a cut-off on a score or eligibility rule: Regression discontinuity
- Neither: Selection on observables
-
Can you build the person-week exposure panel, resolve each tool's coverage, and aggregate to your outcome's unit and frequency?
- Yes: next question
- Not yet: Build the joined panel first
-
Was access assigned at random, or by an arbitrary rule?
- At random, for example a pilot lottery or randomised waves: Compare by assignment
- By a cut-off on a score or eligibility rule: Regression discontinuity
- Neither, it was a business decision:
-
Among people eligible for the tool, were some not yet, or never, treated during your window?
Use of other AI tools does not rule out a comparison group. It changes the question to this tool versus other AI use. Read more
- Yes: next question
- No, everyone eligible was treated at about the same time: next question
-
Did licence dates vary within functions?
- Yes, several waves within the same function: Staggered DiD within function
- No, each function was licensed at once, at different times: Between-function comparison, with caution
- Only a few groups were treated, with a long history before: Synthetic control or controlled ITSA
-
There is no comparison group. What variation is left?
- A long, stable history and a sharp start date: Single-series ITSA, as weak evidence
- People differ in how much they use the tool: Dose-response within person This changes the question from the effect of access to the effect of usage intensity.
- Neither: Report descriptively
-
Can you compare heavy and light users within function, with rich pre-exposure information?
- Yes, and I observe each person over time: Dose-response within person With credible pre-exposure adjustment and overlap, selection on observables may also apply.
- Yes, but I observe each person once: Selection on observables
- No: Report descriptively Consider studying the effect of access instead.
If your situation sits between two routes, read both and run the stricter set of checks.
The routes
Each route states when it fits, the assumption it rests on, the main threats to that assumption, and where to run it.
Describe use without a causal claim
- Use when
- The question is who uses what, how much, and how that changes over time.
- Key assumption
- None about causes. The claim is a description of the data you hold.
- What would break it
- Wording that drifts into "because of AI". Coverage gaps that are treated as zero use.
- Where to run it
- The AI Adoption Across Tools prompt.
Read the full route: Describe use without a causal claim
If the question is about the spread of use across Microsoft 365 Copilot, Cowork, Agents, Consumption credits and GitHub Copilot, you do not need a causal design. Report counts, shares and trends by function and tool, keep unlike units separate, and treat unresolved coverage as unknown.
Randomised access
- Use when
- Access was assigned at random, for example a pilot lottery or waves randomised within function.
- Key assumption
- Assignment is as good as random. To estimate the effect of usage, assignment must affect the outcome only through use (exclusion) and must not push anyone to use less (monotonicity).
- What would break it
- Assignment that was not truly random, unequal attrition between arms, or assignment that changes behaviour without any use, for example through training or announcements.
- Where to run it
fixest(feolswith its IV syntax),pyfixest, orlinearmodelsIV. Seek specialist review if the randomisation was partial.
Read the full route: Randomised access
Compare people by the group they were assigned to, whether or not they used the tool. This intention-to-treat comparison answers what giving access did. Waves randomised within function are the strongest design you are likely to meet, because function is balanced by construction.
For the effect of usage, assignment can serve as an instrument. For a binary measure such as any use, it identifies a local effect for compliers, provided exclusion and monotonicity hold [3]. For continuous usage intensity, the instrumental-variable estimate is a weighted average of effects across levels of use and needs its own stated assumptions. Randomised access does not make chosen usage random, so keep the assignment effect and the usage effect as separate results. Instruments that were not randomised, such as a manager's early adoption, need a separate argument that they are as good as random and affect the outcome only through use. That is rarely convincing in workplace data, so treat them as specialist work.
An eligibility cut-off
- Use when
- Access depended on a threshold, such as a score or a start date, and many people sit close to it on either side.
- Key assumption
- Outcomes would have varied smoothly through the cut-off without treatment. Nothing else changes at the cut-off, and nobody could sort themselves precisely across it.
- What would break it
- Too few people near the cut-off, or manipulation of the score or date.
- Where to run it
- Rare here. Flag for specialist review.
Read the full route: An eligibility cut-off
Regression discontinuity compares people just either side of the threshold [4]. Rollouts here rarely use a sharp rule, so confirm first that one exists. If it does, check how many people fall near the cut-off and whether managers could adjust who sat on which side.
Staggered rollout, compared within function
- Use when
- Licences or access arrived in waves, several waves or both treated and not-yet-treated people exist inside the same function, and you hold repeated pre-adoption weeks.
- Key assumption
- Within function, people treated in a wave would have followed the same path as comparable people not yet treated, had they not been treated (conditional parallel trends).
- What would break it
- Licences going first to people already on a different path, comparison people taking up other AI tools at a different rate from the treated group during the window, behaviour changing before the licence date, or too few clusters.
- Where to run it
didin R,didimputation,did2s,fixest::sunab()orpyfixest. In Python,differencesorcsdid. The repository notebook is a starting point only.
Read the full route: Staggered rollout, compared within function
This is the usual case here. Build the comparison from the population that could have received the tool, such as developers without a GitHub Copilot licence yet. Label each person as treated with a dated start, not yet treated, or never treated in the window. Not-yet-treated people are valid comparisons only for the weeks before their own start, and only inside an estimator built for that.
Avoid reading the effect from a single two-way fixed-effects coefficient. When start dates differ, it is a weighted average of many two-group comparisons, and some use already-treated people as the comparison [5]. If effects differ across cohorts or over time, it can place negative weight on some of the underlying effects [6] [7]. For a wider orientation, see [8]. Pick one of these estimators, with function in the conditioning set:
- Group-time estimates. Estimate an effect for each start cohort and week, then summarise [9]. In R, use
did::att_gt()withxformlato put function in the conditioning set andcontrol_group = "notyettreated". - Imputation. Fit the untreated outcomes with person and function-by-period fixed effects, then compare treated people with their predictions [10].
- Sun and Abraham event studies. Use
fixest::sunab()for an event study that keeps cohorts apart [7]. - Stacked DiD. Build a data set for each cohort, with a clean comparison window and comparison people from the same function, then combine them with the weights needed for the average effect you want to report [11].
Report an event study with pre-period leads, by function wherever cells meet the privacy threshold. Where a function falls below the threshold or lacks power, pool with a stated justification or report it as unidentified. Cluster at the level at which licences were assigned, which is often function or team and seldom the person [12] [13]. With few clusters, consider small-sample corrections or randomisation-based inference.
Two data choices change the estimand. Licence date as the start supports an assignment effect, while first week of use follows a chosen behaviour and carries selection. These estimates also cover only cohorts with an observed pre-period (see when there is no before). For a published example of a staggered AI rollout, see Brynjolfsson, Li and Raymond on an AI assistant introduced to customer-support agents at different times [14].
Whole functions treated at once
- Use when
- Every person in a function started together, and timing differs only between functions.
- Key assumption
- Functions treated at different times would have trended in parallel without treatment, and you can show pre-period evidence for it.
- What would break it
- Function-specific shocks such as reorganisations, targets or seasonality that coincide with the start.
- Where to run it
- The staggered estimators above with function as the unit of comparison, and
HonestDiDfor sensitivity. Seek specialist review.
Read the full route: Whole functions treated at once
When a whole function is treated at once, its effect is not separately identified from whatever else changed in that function at that time. Only cross-function comparisons remain. They need pre-period evidence that the functions moved together, and a sensitivity analysis showing how far the conclusion survives violations of parallel trends [15]. Otherwise report the result as unidentified. First check whether a sub-function, such as a team or region, has variation in timing, in which case the within-function route applies at that level.
A few treated groups with a long history
- Use when
- One or a few groups were treated, you have a long pre-period, and many untreated groups can serve as donors.
- Key assumption
- A weighted mix of untreated groups reproduces the treated group's pre-period path and would have continued to track it.
- What would break it
- A short or unstable pre-period, too few donor groups, or donors that were themselves affected by the tool.
- Where to run it
tidysynthorpysynconfor synthetic control,synthdidfor synthetic DiD, and segmented regression for controlled ITSA.
Read the full route: A few treated groups with a long history
Synthetic control builds a comparison from a weighted set of untreated groups [16]. Synthetic DiD combines that weighting with a DiD comparison [17], but synthdid is available from GitHub only, is not actively maintained and assumes block adoption. Controlled ITSA fits a pre-existing trend and a change at the start date for the treated group and a comparison group [18]. Inference with few treated units is limited, so use placebo-in-space, which assigns the treatment to each donor in turn, and report it with the pre-period fit.
Everyone treated at once: single-series ITSA
- Use when
- Everyone in scope was treated in the same week, there is no usable comparison group, and you have a long, stable pre-period with a sharp start date.
- Key assumption
- Without the change, the pre-period trend would have continued unchanged through the post-period.
- What would break it
- A concurrent event, seasonality, a reorganisation or another AI tool launching around the same date. Any of these can produce the same break.
- Where to run it
- Segmented regression with
statsmodelsOLS and Newey-West (HAC) errors, or the single-series and controlled ITSA examples in the repository notebook.
Read the full route: Everyone treated at once: single-series ITSA
Single-series interrupted time series fits a level and a slope before the start date, then asks whether either changes afterwards [19]. It needs many weekly points before the change, and anything else that happened that week is folded into the estimate, so treat the result as weak evidence.
Period fixed effects are unavailable here, because they absorb the treatment completely. Instead, run a negative-control outcome, and place several false breakpoints inside the pre-period. If breaks of similar size appear often, the series is too noisy for a single-series claim.
Variation in how much people use AI
- Use when
- The question is about intensity of use, and you can observe usage repeatedly for the same people.
- Key assumption
- Changes in a person's usage are unrelated to other changes in their work that also move the outcome.
- What would break it
- Usage responding to workload, deadlines or projects (time-varying confounding), or the outcome driving usage (reverse causality).
- Where to run it
- Within-person fixed effects (M2 in the causal Skill),
contdidfor continuous-treatment DiD (early release), orDIDmultiplegtDYNfor non-binary treatment.
Read the full route: Variation in how much people use AI
A within-person model compares the same person in heavier and lighter weeks. It removes stable differences such as role or working style, but not changes that move usage and the outcome together. Busy weeks may raise both Copilot use and meeting load, for example. Continuous-treatment DiD compares groups by dose and needs stronger parallel-trends assumptions than the binary case [20]. Both packages are young, so confirm the estimator supports doses that change week to week. Timing checks are diagnostics only, and passing them does not establish a causal dose effect. Without a strategy for time-varying confounding, report associations.
Avoid adding a lagged outcome to a short fixed-effects panel, because it induces dynamic panel bias [21]. Actions, tasks, sessions and credits each get their own dose measure, and are never summed into one AI score.
Selection on observables
- Use when
- You can credibly adjust for confounders measured before exposure, and comparable people exist across exposure levels within function. The outcome may be observed only once.
- Key assumption
- After conditioning on the covariates, exposure is unrelated to the outcome you would have seen without it (unconfoundedness), and comparable people exist in both groups (overlap).
- What would break it
- Unmeasured drivers of adoption such as motivation, manager support or project type, and heavy users who exist only in one function.
- Where to run it
- Regression, matching, doubly robust estimation or double machine learning [22] with
DoubleMLoreconml. For Copilot actions, see the Copilot Causal Toolkit.
Read the full route: Selection on observables
The covariates must be measured before exposure and must include function. Check overlap within function using the distribution of estimated scores in each group. If heavy users sit in one function only, the comparison extrapolates. When access was neither randomised nor set by a defensible cut-off, this is also the route for an outcome observed once, with heavy caveats, because no fixed effect or timing check is available. No universal E-value threshold turns a sensitivity number into a pass, so report the sensitivity analysis and let readers judge it.
When no design fits
- Use when
- No route above has its assumptions met.
- Key assumption
- None. You make no causal claim.
- What would break it
- Wording that slips an effect into a descriptive result.
- Where to run it
- The AI Adoption Across Tools prompt for descriptive reporting.
Read the full route: When no design fits
Say plainly that a causal estimate is unavailable, and report what you can describe. Then list what would change the answer, such as randomised waves, staggered waves inside a function, a longer pre-period, or an outcome that repeats weekly.
Build the joined panel
Most routes start from the same exposure table, so build it once and carefully. The Person Query is the spine, at person-week grain, and other sources join onto it after they have been reduced to the same grain. For a team or business-unit outcome, aggregate exposure to that unit and period before joining, because copying a team outcome onto every person creates repeated copies of one observation.
Agent data arrives at person-agent-week, and Consumption data may be daily, so aggregate both to person-week first. The query join guide lists the native keys, aligned week starts and scoped missing-row rules. After every join, confirm that the row count and the person-week keys match the spine, and stop if cardinality is ambiguous. Keep the raw header names, such as Cowork tasks submitted and Total GitHub AI Credits used, so every number traces back to its export.
Eligibility and treatment status, per tool
Restrict each tool's analysis to the people who could have received it, then classify every eligible person for that tool. Absent activity alone does not establish that someone was never treated.
| Status | Definition | Role in estimation |
|---|---|---|
| Treated in window | Exposure began on a dated week in the window | Treated cohort, defined by start week |
| Already treated before the window | Access or adoption predates the export | Excluded from untreated comparison groups. They can support within-person intensity analyses |
| Not yet treated | Eligible, exposure begins later in the window | Valid comparison only for weeks before their own start |
| Never treated | Eligible, with licence or access records confirming no exposure through the window | Comparison group, with a note that later exposure is possible |
| Ineligible or unresolved | Not in the tool's population, or status cannot be established | Excluded from that tool's estimate |
State the exposure definition for each tool. A licence start date measures access, first observed use measures adoption, and an intensity measure captures dose. These answer different questions.
Contamination across tools
Check every AI tool in the joined data, including those outside your study. Unlicensed Copilot Chat, Cowork and Agents can make a nominal comparison group partly treated. Decide whether the estimand is this tool versus no AI, or versus other AI use, and build groups to match. Tools adopted together cannot be separated, so report a joint effect. Untreated colleagues can also benefit from a treated colleague's drafts or meeting recaps, so if spillovers are likely, define direct and spillover effects separately, or move to a team-level design.
Missing is not zero
Unresolved coverage stays unknown, and an omitted source or a failed join says nothing about use. Within a complete query's configured scope, absent activity rows do contribute zero. For documented licensed-only native Microsoft 365 Copilot metrics, a null means unlicensed and a zero means licensed but inactive. Do not extend that rule to unlicensed Copilot Chat or to Consumption credits, or infer a licence from credits or Agent sessions. Publish only groups that meet the stricter of your organisation's minimum or 10 people.
Function as a confounder
Rollout usually follows function. An organisation might license Engineering first, Sales later and Finance last, often for good business reasons. Function then determines when people get the tool and how their outcomes move, so a comparison that ignores it mixes the two. In this hypothetical, Engineering licensed in three waves supports within-function comparisons of earlier against later waves. Sales licensed in a single week has only between-function comparisons, against Engineering or Finance, and they depend on Sales trending like the comparison function. Finance, not yet licensed, can serve as a comparison but has no effect of its own to estimate.
Prefer within-function comparisons where suitable untreated observations remain. They still need conditional parallel trends, no anticipation and a clear treatment definition. Between-function comparisons rest on stronger assumptions, because a Sales-wide reorganisation looks identical to a Sales-wide AI effect.
Selection inside a function
Within a function, licences may still go first to particular roles, levels or high performers, or after a dip in performance, which creates a pre-programme dip [23] and makes recovery look like an effect. Compare future adopters with later and never adopters before treatment (see the before-adoption comparison). Small cells may need pooling, with a stated justification.
Standard checks
Run these five checks on the observational routes where they apply. Each one says where it does not. Randomised and regression discontinuity designs have their own diagnostics, such as balance checks between arms and density tests at the cut-off.
A fixed-effects ladder
Report the estimate under increasingly demanding specifications, using the same sample, target population and aggregation weights each time. Start from a basic comparison with person and period adjustment, then add function-by-period fixed effects, and where useful also show the pooled estimate without person effects. A large movement when person effects are added suggests that adopters differed from non-adopters from the start. A large movement when function-by-period effects are added suggests that function-specific shocks coincided with rollout. Plain period effects only absorb shocks that hit the whole organisation at once, so function-by-period effects are the default.
Three limits apply.
- Under staggered adoption, build this structure inside a staggered-adoption estimator (see the staggered DiD route).
- Person effects need repeated outcomes and some exposure variation in the sample. Never-treated comparison people, whose exposure never changes, still help estimate the period effects.
- Period effects are unavailable without a comparison group, which rules them out on the single-series ITSA route.
Fixed effects remove stable differences between people but do not detect them, and they do nothing about differences that change over time, such as a person who is becoming more engaged and also more likely to be licensed. The checks below address that. Useful packages are fixest in R, and pyfixest or linearmodels in Python.
Time-reversal placebos
Test whether future exposure "predicts" earlier outcomes. If it does, the usual explanations are licences targeted at people who were already improving, reverse causality, or allocation after a dip in performance [23]. For dated licence events, the leads of an event study are this check. Plot the lead coefficients with their intervals, and as a summary run a joint Wald (F) test of the leads. For usage intensity, include current exposure alongside the lead, because usage persists from week to week. Leads need dated exposure and several periods before it, so they are not available with a single series.
People may change behaviour before access arrives. Specify an anticipation window and keep those weeks out of the untreated comparison and the reference period, or move the treatment start to the announcement when that matches the question.
Judge lead estimates by their size and precision. Pre-trend tests often have low power, and a non-significant result does not establish parallel trends [24]. Where you can, show how far the conclusions survive bounded violations of parallel trends, using HonestDiD in R [15]. It takes the event-study coefficients and covariance from any estimator.
Compare adopters before adoption
Compare future adopters with later adopters and never adopters, within function, on outcomes and characteristics measured before treatment, such as level, tenure, role and earlier outcome trends. If future adopters already differed on the outcome or its direction of travel, the estimate needs a stronger argument or a different design.
A negative-control outcome
Estimate the same model on an outcome the tool should not plausibly change [25]. An apparent effect there points to residual confounding or a shared shock. A null result is reassuring but does not rule out confounding that affects only your main outcome. A useful negative control has no plausible route from the tool and shares the confounding or measurement processes that threaten the main outcome, such as the same seasonal pattern or reorganisation. Write both reasons down before the analysis. Meeting measures rarely qualify, because AI tools can change meeting attendance. If no candidate qualifies, say so in the write-up.
Leave one function out
Re-estimate the model excluding each function in turn, and show the estimates side by side. If one function dominates, report that concentration and estimate function-specific effects where the data support them. A loss of significance after dropping a function does not show that only that function benefited. Where cohorts are few, also leave one cohort out, since a single licence wave can dominate a pooled estimate.
Not every check applies to every route. The single-series ITSA route has no comparison group, so it cannot use the fixed-effects ladder, the before-adoption comparison or leave-one-out. A negative control still helps there. For synthetic and controlled ITSA designs, the comparisons run against donors, and leave-one-out means leaving out one donor. Between-function designs have few functions, so leave-one-out and the before-adoption comparison are coarse, and function-by-period fixed effects are unavailable because they absorb a start that covers a whole function. For selection on observables, the ladder needs repeated outcomes and the placebo needs outcomes observed before exposure, while covariate balance and overlap can be checked without earlier outcomes.
Report what you can claim
Report four things separately: the effect, its precision, the identification argument and the limitations. A sentence that blends them hides which part is weak.
Say whether the outcome is a proxy or a KPI, and how the proxy relates to the KPI, or that this could not be checked. Name the estimand: what is compared, for which cohorts and functions, and over which weeks.
Report the clustering level and the number of clusters in each model sample, at the level at which exposure was assigned [12]. With few clusters, standard cluster-robust inference can be unreliable [13], so say which correction you used, or that uncertainty cannot be reliably quantified.
Causal wording needs the identification argument and the checks above to hold up together. A pair of hypothetical sentences shows the difference.
- Weak: "Copilot reduced after-hours work in Sales."
- Careful: "Among later licence cohorts in Sales, weeks after licensing are associated with fewer
After hours collaboration hours. The estimate is consistent with a reduction, and the leads show no clear pre-trend, but the interval is wide and function-level shocks cannot be ruled out."
If you test many outcomes, functions or cohorts, say how many, and adjust or flag the multiplicity. Apply privacy thresholds to every published cell, including complementary cells that could be subtracted, and do not publish rankings of individuals.
Finally, list what you did not do, such as unavailable checks, functions dropped for privacy reasons and outcomes you examined but did not report. Keep the headline claim no stronger than the weakest link from exposure to outcome. A licence supports a claim about access, and chosen use supports a claim about people who chose to use the tool.
Run it yourself
These resources cover parts of the workflow, and none runs every route in this article.
- The causal Skill checks that a panel is ready and runs within-person diagnostics. It returns a proposed plan for DiD, event studies and instruments rather than running them.
- The Copilot Causal Toolkit runs a double machine learning workflow for Copilot, and suits the selection-on-observables route.
- The event-study scripts in R and Python fit a staggered-adoption event study on synthetic data, with
fixest::sunab()in R and a saturated event study inpyfixest. They include a synthetic function variable and function-by-week fixed effects, matching the within-function comparison in the staggered route. - The difference-in-differences metric scans in R and Python compare heavy and light users across many metrics. They report Holm-adjusted p-values and remain exploratory, because groups defined by chosen usage carry selection.
- The DiD/ITSA notebook implements 2x2 DiD with pooled OLS, an event-study plot of week-by-week group-mean differences, and controlled and single-series ITSA. It is not a staggered-robust estimator and does not use clustered errors.
- The query join guide explains how to join other sources onto the Person Query.
For routes the repository does not implement, these maintained packages are the starting point.
| Route | R | Python |
|---|---|---|
| Staggered DiD | did | differences, csdid |
| Imputation or two-stage | didimputation, did2s | pyfixest |
| Event study with fixed effects | fixest | pyfixest |
| Non-binary or non-absorbing treatment | DIDmultiplegtDYN | None found |
| Continuous dose DiD | contdid (early release) | None found |
| Pre-trend sensitivity | HonestDiD | No maintained port verified |
| Synthetic control | tidysynth, Synth | pysyncon |
| Synthetic DiD | synthdid (not actively maintained) | Port is stale |
| Controlled or single-series ITSA | Base R or fixest | statsmodels |
| Fixed-effects ladder | fixest | pyfixest, linearmodels |
| Double machine learning | DoubleML | DoubleML, econml |
References
- Athey, S., Chetty, R., Imbens, G. W. and Kang, H. (2025). The surrogate index: combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Review of Economic Studies. Earlier version: NBER Working Paper 26463 (2019).
- Prentice, R. L. (1989). Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in Medicine, 8(4), 431β440.
- Imbens, G. W. and Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467β475.
- Lee, D. S. and Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281β355.
- Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254β277.
- de Chaisemartin, C. and D'HaultfΕuille, X. (2020). Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9), 2964β2996.
- Sun, L. and Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175β199.
- Roth, J., Sant'Anna, P. H. C., Bilinski, A. and Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218β2244.
- Callaway, B. and Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200β230.
- Borusyak, K., Jaravel, X. and Spiess, J. (2024). Revisiting event-study designs: robust and efficient estimation. Review of Economic Studies, 91(6), 3253β3285.
- Wing, C., Freedman, S. M. and Hollingsworth, A. (2024). Stacked difference-in-differences. NBER Working Paper 32054.
- Abadie, A., Athey, S., Imbens, G. W. and Wooldridge, J. M. (2023). When should you adjust standard errors for clustering?. The Quarterly Journal of Economics, 138(1), 1β35.
- Cameron, A. C. and Miller, D. L. (2015). A practitioner's guide to cluster-robust inference. Journal of Human Resources, 50(2), 317β372.
- Brynjolfsson, E., Li, D. and Raymond, L. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889β942.
- Rambachan, A. and Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555β2591.
- Abadie, A., Diamond, A. and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493β505.
- Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W. and Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088β4118.
- Lopez Bernal, J., Cummins, S. and Gasparrini, A. (2018). The use of controls in interrupted time series studies of public health interventions. International Journal of Epidemiology, 47(6), 2082β2093.
- Lopez Bernal, J., Cummins, S. and Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: a tutorial. International Journal of Epidemiology, 46(1), 348β355.
- Callaway, B., Goodman-Bacon, A. and Sant'Anna, P. H. C. (2024). Difference-in-differences with a continuous treatment. NBER Working Paper 32117.
- Nickell, S. (1981). Biases in dynamic models with fixed effects. Econometrica, 49(6), 1417β1426.
- Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W. and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1βC68.
- Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. The Review of Economics and Statistics, 60(1), 47β57.
- Roth, J. (2022). Pretest with caution: event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305β322.
- Lipsitch, M., Tchetgen Tchetgen, E. and Cohen, T. (2010). Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology, 21(3), 383β388.