Hierarchical Forecasting • finnts

The finnts package leverages the great work of the hts package. It’s currently retired but we think old dogs can still learn new tricks! It’s still of great use to finnts and allows for both standard and grouped hierarchical forecasting.

Selecting a Hierarchy

Use detect_hierarchy() before a non-agentic forecast to choose a structurally valid approach. A unique bottom-level ID alone does not prove nesting: intermediate labels can still occur under multiple parents. Detection checks every adjacent parent-child relationship in the same cardinality-based order used by the engine.

decision <- finnts::detect_hierarchy(
  input_data = data,
  combo_variables = c("Region", "Country", "Site"),
  target_variable = "Revenue",
  combo_cleanup_date = as.Date("2025-08-01"),
  hist_end_date = as.Date("2026-08-01")
)
decision$forecast_approach
decision$conflicts
# Pass decision$forecast_approach as forecast_approach to prep_data(),
# using the same input data and cleanup settings.

The function returns diagnostics in memory and does not save files or change the caller’s input. Without a cleanup date it considers all supplied tuples; with a cleanup date it uses the existing target-sum cleanup policy and excludes future targets from activity. One retained series selects bottoms_up, and no retained series produces an error. Character-like labels are trimmed consistently with preprocessing; invalid labels and ambiguous combo identities fail explicitly. Collect Spark data frames before detection.

Detection describes observed data, not unobserved business relationships. Inspect hierarchy_order and conflicts when validating the result. Existing explicit prep_data() settings are not overridden. Agent setup and EDA share the structural analysis, with an intentional single-column bottoms-up policy. Adopting a different approach changes the prepared aggregates and hierarchy metadata; use a new run rather than reusing artifacts from a different hierarchy.

New Agent versions with only one retained series use bottoms_up, even with multiple combo columns. EDA reports none, and global reasoning allows only bottoms-up for this scope. Pre-expanded hierarchy-level ID inputs also use bottoms-up internally, but retain their configured outer reconciliation.

Resuming an existing Agent version preserves its saved forecast approach and previously uploaded inputs; a package upgrade does not reclassify that version. API request-ID retries select the version belonging to that request, even if a newer request has since created another version. Logged settings such as dates, horizon, regressors, and forecast flags must still match. Supplied replacement data values are not uploaded on a resume; create a new version to use changed history, drivers, or series. Missing required saved artifacts remain errors rather than triggering a hierarchy migration. Legacy bottoms-up run logs without allow_hierarchical_forecast cannot reveal whether expansion was disabled or the single-column policy selected bottoms-up; these retain the existing flag-comparison fallback. A saved hierarchical approach proves expansion was enabled. No new log columns are added.

A new version uses current detection on its cleaned population. If its outer approach differs from the latest completed predecessor (for example, standard to grouped, or hierarchical to single-series bottoms-up), update_forecast() rejects reuse and directs the user to iterate_forecast(). The allow_iterate_forecast performance fallback does not bypass this compatibility check. Existing predecessor artifacts are not rewritten.

Standard Hierarchy

Hierarchical external regressors retain the existing additive convention. A level-mapped driver must have at most one nonmissing value for each source tuple and date. Inference first checks single columns, then progressively wider tuples only if no narrower grain is valid. At the first valid width, it selects the grain with the fewest distinct grain/date/value tuples; equally coarse candidates must identify equivalent source partitions. Inference checks at most 1,024 candidate grains per driver. Each candidate-width batch is evaluated completely or rejected before scanning. If the search budget is exhausted, preparation fails before hierarchy artifacts are written; it does not silently fall back to bottom-level summation. Reduce redundant combo columns or prepare the driver at its known source grain before retrying. Aggregates count each source tuple once within the node’s actual bottom-series membership, including separate tuples with equal numeric values. Original bottom observations are preserved; missing-only level aggregates stay missing. Date-global drivers are copied, while full-bottom drivers retain the existing bottom-level summation and missing-value handling. Equally coarse candidate levels must describe the same partition or preparation fails with an ambiguity error. Observed equality cannot establish business meaning: validate the source grain and additive assumption before preparing a production forecast. These corrections can change newly prepared driver values. Use a new run or Agent version; existing saved inputs and fitted models are not migrated.

A standard hierarchy of data is pretty straight forward. Each combo variable can be aggregated into the next combo variable when producing forecasts at higher aggregations of the data. This structure resembles a pyramid, with the bottom being the lowest granularity of time series and the top being a single time series of the grand total of the data. Below is a good example with geographical combo variables that can be aggregated into one another when building a standard hierarchy forecast.

#> Standard Hierarchical Time Series Data
#> # A tibble: 9 × 5
#>   Continent     Country       City        Date       Target
#>   <chr>         <chr>         <chr>       <date>      <dbl>
#> 1 North America United States Kansas City 2020-01-01    100
#> 2 North America United States Kansas City 2020-02-01    250
#> 3 North America United States Kansas City 2020-03-01    320
#> 4 North America United States Seattle     2020-01-01     80
#> 5 North America United States Seattle     2020-02-01    200
#> 6 North America United States Seattle     2020-03-01    270
#> 7 North America Mexico        Mexico City 2020-01-01     50
#> 8 North America Mexico        Mexico City 2020-02-01     80
#> 9 North America Mexico        Mexico City 2020-03-01    120

In the above example, “City” was the lowest level of the hierarchy, which feeds into “Country”, which then feeds into “Continent”. For a non-agentic run, Finn forecasts each model at the prepared hierarchy levels, selects the best forecast at each level, and reconciles both the selected combination and the individual models back down to the lowest level. get_forecast_data(run_info) returns every successfully saved per-model reconciled forecast plus Best-Model. Models that fail to produce a reconciled artifact are not fabricated in the result. Filter Best_Model == "Yes" to retain only the reconciled selected forecast.

Agentic runs use a different output contract. During iterate_forecast(), different series can run and select different model sets, recipes, or averages. Finn therefore reconciles the selected forecasts together, and get_agent_forecast(agent_info) returns only that final reconciled Best-Model forecast, not a reconciled comparison table for every model. Best-Model can combine different winners across series; it does not mean that the same model family won everywhere. This distinction applies to both standard and grouped hierarchies and to Agent forecast updates.

Grouped Hierarchy

Grouped hierarchies are very different than the traditional hierarchy approach described above. There are some data sets that can be aggregated in various ways, meaning they need to follow another approach the hts package calls “grouped”. A good example is a data set that contains historical time series by geography, customer segment, and product.

#> Grouped Hierarchical Time Series Data
#> # A tibble: 12 × 5
#>    Country       Segment       Product Date       Target
#>    <chr>         <chr>         <chr>   <date>      <dbl>
#>  1 United States Enterprise    Coffee  2020-01-01     10
#>  2 United States Enterprise    Coffee  2020-02-01     20
#>  3 United States Enterprise    Coffee  2020-03-01     30
#>  4 United States Public Sector Coffee  2020-01-01      5
#>  5 United States Public Sector Coffee  2020-02-01      8
#>  6 United States Public Sector Coffee  2020-03-01     11
#>  7 Mexico        Enterprise    Coffee  2020-01-01     20
#>  8 Mexico        Enterprise    Coffee  2020-02-01     23
#>  9 Mexico        Enterprise    Coffee  2020-03-01     27
#> 10 Mexico        Enterprise    Tea     2020-01-01     50
#> 11 Mexico        Enterprise    Tea     2020-02-01     55
#> 12 Mexico        Enterprise    Tea     2020-03-01     60

It would be hard to aggregate the above data in a traditional hierarchy. The same products are found in different segments and countries, also the same segments are found in multiple countries. Finn will follow a similar modeling process as the one described for a traditional hierarchy, but instead will create forecasts at the below levels.

Grand Total: model “Target” across the sum of “Country”, “Segment”, and “Product”
Country: model “Target” across the sum of “Country”. Creating forecasts for the grand total of “United States” and “Mexico”
Segment: model “Target” across the sum of “Segment”. Creating forecasts for the grand total of “Enterprise” and “Public Sector”
Product: model “Target” across the sum of “Product”. Creating forecasts for the grand total of “Coffee” and “Tea”.

Reviewed Custom Rules

Standalone hierarchy runs can apply a reviewed custom rule independently to every prepared node, including aggregates, then use FinnTS’s regular hts reconciliation to publish the lowest level. Review must explicitly permit this adjustment: reconciled values are not guaranteed to equal the rule’s original base predictions.

An HTS forecast using custom models requires a shared, explicitly enrolled mixed custom/FinnTS candidate pool. Custom-only requests must use bottoms_up with a compatible bottom-up definition. Finn does not silently add competitors or switch approaches. Eligibility depends on requested candidates, not winners; custom models may win everywhere. Existing built-in-only hierarchy behavior is unchanged.

custom <- create_custom_model(
  instructions = paste(
    "For each hierarchy node, forecast the mean of all historical targets.",
    "Allow normal FinnTS reconciliation to adjust base forecasts."
  ),
  llm = llm, input_data = data_tbl,
  combo_variables = c("Region", "Product"), target_variable = "value",
  date_type = "month", forecast_horizon = 3,
  model_type = "local", name = "business_mean",
  control = custom_model_control(forecast_approach = "standard_hierarchy")
)

run_info <- set_run_info(data_output = "csv", object_output = "rds")
forecast_time_series(
  run_info, data_tbl, c("Region", "Product"), "value", "month", 3,
  forecast_approach = "standard_hierarchy",
  custom_models = list(business_mean = custom),
  models_to_run = c("business_mean", "meanf"),
  recipes_to_run = c("R1", "C1"), stationary = FALSE,
  clean_missing_values = FALSE, clean_outliers = FALSE,
  run_local_models = TRUE, run_global_models = FALSE,
  run_ensemble_models = FALSE, average_models = TRUE,
  feature_selection = FALSE, inner_parallel = FALSE,
  negative_forecast = TRUE, weekly_to_daily = FALSE
)

The input must contain a complete finite leaf panel over the requested historical window. This custom profile does not silently zero-fill missing observations. Raw authoring samples up to three deterministic leaf series and keeps all their derived nodes; prepared authoring reads the whole existing hierarchy without modifying it. Both enforce a 10000 historical-node-row bound and validate every captured node. Manual and automatic approval use the same gates. Individual authoring validation does not enroll a custom-only HTS run.

Old bottom-up definitions and hashes remain valid but do not acquire hierarchy permission. New hierarchy definitions bind one approach, per-node application and permission for reconciliation. The approved envelope and registry format are unchanged. Run manifests additionally bind topology and input artifacts. Changed context or missing base candidates cannot be treated as a completed restart.

Custom hierarchy requires average_models = FALSE, no external regressors, no target transformations or cleaning, and no pooled/global fitting. Weekly output stays weekly. Every enrolled candidate must have complete finite node/fold/date coverage before reconciliation; missing nodes are not filled from snaive and failed candidates are not silently skipped. Normal weights and selection policy remain unchanged. Per-model results retain their readable aliases and full version IDs; the selected mixture is published as Best-Model.

The following matrix distinguishes implementation from qualification evidence. It does not imply every cross-product of formats, cadences and workers was tested.

Capability	Current Evidence or Boundary
Standard/grouped, local R1, monthly CSV	Implemented-tested: real mixed forecasts, strict coverage and exact-version restarts
Grouped monthly CSV, two local-machine workers	Implemented-tested: sequential parity, two worker PIDs and exact installed namespace
Year/quarter/month/week/day at native cadence	Implemented-tested: independent reconciliation projection and date-key checks; complete nonmonthly pipeline qualification remains open
RDS and Parquet data, RDS objects	Implemented-tested: preparation, context capture and exact-path reload; full forecasting/worker combinations remain unverified
Raw/prepared hierarchy review	Implemented-tested: real installed TEST/APPROVE children, complete-node evidence and authored mixed forecast
Other installed compatible local R1 FinnTS competitors	Inert workflow/dependency checks implemented; individual model-family qualification remains open
Custom-only HTS	Disallowed by explainability policy; use compatible bottom-up definitions
Foundation-model competitors	Not qualified for this custom hierarchy profile; rejected rather than counted as token competitors
Drivers, pooling, R2, transformed targets, hierarchy averages, weekly expansion	Remaining work; rejected by the current profile
Agent hierarchy and hierarchical new-data updates	Remaining work; current custom Agent behavior remains bottom-up only
Remote/Spark, other platforms, minimum dependency versions and live providers	Not qualified here; separate approval and validation required

The per-series routing interface is deferred. Future user-specific model restrictions will require bottoms_up; ordinary node-level winner differences and internal worker partitions are not routing restrictions. Users requiring exact single-rule bottom-up output must also explicitly disable averaging. Trusted source and RDS are not sandboxed or authenticated approval. Offline provider fixtures verify control flow, not live generation quality.

External Regressors

The aggregation process is automatically calculated for each external regressor, depending on how the regressor maps to the combo variables. If a regressor is unique for each time series, then the standard aggregation process is implemented (same as the process for the target variable). If the regressor repeats the same values across all time series, then only one global copy is retained and applied to all new aggregated time series. If the regressor specifically maps to one or more combo variables, then the relationship is respected while still being able to aggregate to the total level. Aggregations of drivers are always summed.

Intermediate standard-hierarchy nodes use their actual bottom-series membership, even when hts assigns synthetic node names. Each node sums only its own source entities, rather than inheriting a grand-total driver value.

Explore the final results of the aggregations by seeing the end result using get_prepped_data().

Spark Parallel Processing

There is also a small limitation when doing hierarchical forecasting using spark as the parallel computing back-end. The hts package finnts uses cannot handle spark data frames. So behind the scenes finnts has to bring all of the data into memory on the spark cluster driver node in order to create the hierarchies. Please keep that in mind. If you have a lot of historical data that is greater than the RAM on the spark driver node, please consider using a different VM size with more memory. The same issue holds true when reconciling the hierarchical forecast, where all of the forecasts across all time series for a specific model is loaded into the same executor node to be reconciled. Because of these limitations, we are now exploring other options outside of the hts package, including the building of our own hierarchical forecasting package. Stay tuned.

Corner Cases in Forecast Reconciliation

Some forecasts can be widely off from the target value. This can cause issues when finnts tries to use historical back testing residuals as the initial weights in hts to reconcile the forecast at various aggregations of your data down to the lowest level. To prevent reconciliation issues, finnts will shrink these residuals to more appropriate numbers that still convey they are considerably off from the initial target values, but not astronomical. For example a residual value of 1,000,000,000,000 compared to a target value of 100 might get shrunk down to 900, since keeping the initial residual will throw the weighted reconciliation process out of whack and cause issues. Most forecasts in finnts will not have this problem, but if your data is very noisy then this fix will help prevent any issues when trying to run hierarchical forecasts in finnts.