Best Model Selection

The Best_Model flag identifies the selected forecast for each series. Finn first checks candidate validity, then balances backtest weighted MAPE with future-forecast plausibility. Individual models, learned ensembles, and simple averages are evaluated on the same series and expected backtest dates.

Process at a glance

Model selection is several connected decisions, not one minimum-error calculation:

Stage	Question answered	Result
Candidate validity	Does each model have complete, finite predictions and usable backtest accuracy?	Hard-invalid candidates cannot win or contribute to a simple average.
Within-run selection	Among sufficiently accurate candidates, which future path ranks best by risk, concern count, supported seasonal fidelity, WMAPE, and stable ID?	One selected model or exact average for each series. Soft concerns can remain when no better eligible alternative exists.
Agent iteration context	Which completed iteration’s settings should guide further optimization?	A near-best iteration can be preferred when its individual model-pool accuracy improves. This does not automatically replace a better saved local forecast.
Saved forecast promotion	Should the current iteration replace the saved result?	Local winners stay protected; globally selected series reference one winning global iteration.
Update acceptance	Is the newly refitted version of the saved choice still acceptable?	Keep that choice or attempt the existing default-local replacement workflow before reconciliation.

Seasonal fidelity breaks a risk-and-concern tie only when every candidate in that tied group has a finite assessed score; otherwise WMAPE and stable ID decide the tie.

For a standard run, follow candidate checks, the accuracy allowance, and selection and saved averages. The Agent and update sections describe the additional decisions. Reconciliation makes the selected hierarchy coherent; it does not select the models again.

Two uses of “average” are important: a simple-average forecast combines predictions from selected components, while average model accuracy summarizes individual model WMAPEs to guide Agent search. Improving the latter can be useful even when the winning model has not changed.

Experimental custom models

prep_models() and forecast_time_series() accept a named custom_models list. With models_to_run = NULL, every built-in model and every supplied custom model runs. An explicit models_to_run must name every supplied custom model; an unlisted custom model is an error rather than being silently dropped. An explicit pool can contain custom models only or a mixture of custom and built-in models. Custom hierarchy execution also requires separately reviewed hierarchy capabilities.

create_custom_model() is the single natural-language authoring entrypoint. It accepts raw input_data with Finn metadata or an existing prepared run_info. Humans and coding agents use the same function in one call. Advanced settings belong in custom_model_control(); approval stays visible at the top level and defaults to manual. Manual approval needs an interactive R session; scripts, schedulers and coding agents use approval = "automatic".

Instructions may contain up to 1 MiB (1048576 UTF-8 bytes), including accumulated review edits. This permits thousands of words and hundreds of code lines; text is never silently truncated. The selected LLM’s context window must also accommodate Finn’s schema, examples and conversation history. This input limit does not enlarge the separate generated-response or reviewed-source limits.

Declared string drivers retain their type when C1 CSV data is read for workflow preparation and training. Numeric-looking category labels such as "100" and leading-zero labels remain strings rather than being inferred as numeric values. This does not coerce ordinary numeric drivers or change the stored CSV format.

Custom-driver importance.

Newly generated Agent model summaries include custom-driver importance without requiring vip. Finn holds out the latest complete historical forecast horizon, refits the exact approved rule on earlier history once per fitted workflow, and compares baseline holdout RMSE with ten deterministic cyclic permutations of each eligible driver. The score is the mean signed increase in holdout RMSE, in target units. Zero and negative scores are retained. Scores are conditional on the supplied assessment drivers, chosen holdout and correlations; they are not causal attribution, percentages, or a model-selection criterion.

Date/series identities and cohort keys never move. Global prediction requests retain their complete peers and horizons; permutations stay within declared groups. Global recipients share the diagnostic computation within one summary invocation. This executes additional trusted custom fitting and prediction code with the same permissions as forecasting; it adds runtime and is not sandboxed. It does not overwrite saved fits or forecasts. No-driver models need no extra diagnostic fitting.

Insufficient history or lags, incomplete historical holdouts, and constant or cohort-key drivers are explicitly reported as not estimable, rather than given invented zero scores. Unexpected source, identity, data or storage errors still fail. Summary artifacts retain their existing columns and add importance rows (name is the driver, value is its score), plus importance_metadata method/status rows. Consumers should accept these additional rows. Existing completed runs and saved summaries are not migrated.

New calls default to local models. Raw input must contain a column named Date and an explicit target and horizon. Grouping can be omitted only when dates are unique and no varying undeclared columns suggest multiple series. Cadence can be omitted only when at least three actual dates per series match exactly one complete day/week/month/quarter/year grid. Ambiguous or incomplete grids require explicit metadata. Prepared runs supply their saved settings, including forecast approach. Neither inference supplies fiscal calendars nor proves business close.

Raw driver-based models can be created and validated before the caller creates its forecast run or calls prep_data(). Authoring keeps original driver names, values and types alongside its privately prepared historical target data. Only observed historical driver rows enter validation; future values are not used, and missing/nonfinite driver history is rejected rather than filled. Check forecast-date coverage and supply the required future driver inputs before executing the resulting model. Private temporary preparation still occurs inside authoring; it does not create the caller’s forecast run.

Use hist_end_date as the last closed actual period, with the same date-label convention as the data. For monthly observations labelled on the first day, August 2026 is as.Date("2026-08-01"), not August 31. Without an explicit cutoff, every series must have the same last finite target date for inference. Finite values do not prove a business close: supply a cutoff when later target rows contain budgets, prior forecasts or partial actuals.

Before sampling or padding, raw input must cover an uninterrupted common historical window with finite targets for every series. Drivers must cover that window and every requested future period; future target values are ignored. Missing periods, uneven histories and future-only series now fail rather than being padded with zeros. Explicit common hist_start_date/hist_end_date bounds can select a complete overlap. Prepared driver artifacts are checked across all requested series before sampling, but their stored values cannot prove whether earlier preparation filled a raw gap.

The authoring cutoff is saved validation evidence, not a permanent model cutoff. Every newly approved model is reusable by default, without a special control option. Save the entire envelope with saveRDS() and reload it with readRDS() for later standard or Agent runs. Those runs may use different positive whole-number forecast horizons, history lengths, end dates, compatible driver values and global training populations. Each fit derives its own cutoff and per-series cutoffs; relative calculations must use those dates and the prediction request. Reusing an old fitted workflow without refitting does not advance its cutoff.

The approved formula, cadence, units, required driver types, local/global mode, cohort rules and business constants remain fixed. Declared minimum history is checked on each analysis fold; forecast-relative observed lags must exist for every requested future date. A longer horizon can therefore expose a genuine rule constraint and fail explicitly. Horizon-wide allocations can legitimately change earlier predictions when the horizon changes; they need not have an invariant prefix. Representative validation is evidence, not proof for all positive horizons.

Existing run IDs remain exact restart contracts. Use a fresh standard run or a new Agent version for changed data/settings. Older pre-release custom objects and saved custom runs may be incompatible with this contract and must be recreated; they are not migrated or rewritten automatically.

Driver-based holdout metrics are explicitly conditional on supplied assessment drivers. The current interface does not verify historical plan vintages or publication delays. For realistic business backtests, provide the plans actually available at each forecast origin rather than revised plans or realized drivers.

Enrollment accepts only finnts_custom_model envelopes containing an exact passive definition, version-bound technical check summaries, manual intent confirmation or explicit automatic trusted-code authorization. Raw definitions, registry references, fitted models and stale approval evidence are rejected. Attestations are provided by a trusted caller; they are not authenticated proof of human approval or a sandbox for untrusted R code. Tests use clearly synthetic evidence, not authenticated proof of approval or live-provider validation.

The default approval = "manual" asks unresolved business questions, prepares a fixed interpretation and numerical examples, and generates a candidate. One compact review then asks whether to create that exact model. A yes both authorizes same-session execution and conditionally approves the model if validation passes. There is no second approval prompt after success. A repair cannot change the expected values or reuse another version’s consent: revised code always receives a fresh review. Holdout accuracy is reported, not used to rewrite the human rule.

If validation fails and another candidate is proposed, manual review now explains why approval is needed again. It displays the prior failure phase and safe reason, previous/current version IDs, and the names of changed source functions. The rule and worked examples remain unchanged; the proposed repair is not yet validated. An identical resubmission is labelled as the same failed candidate, not a fix. details shows full identities and the current source without executing it. Compact per-function hashes support the comparison without retaining another full source snapshot. Older diagnostics can lack those hashes, in which case comparison detail is explicitly unavailable. Existing saved reviews remain valid; exact-candidate consent, replay handling and retry limits do not change.

llm <- ellmer::chat_openai()
custom_model_examples_template(forecast_horizon = 3, date_type = "month")
dates <- seq(as.Date("2020-01-01"), by = "month", length.out = 5)
examples <- data.frame(
  example = rep(c("mean_case", "edge_zero_history"), each = 5),
  Date = rep(dates, 2),
  Target = c(10, 20, NA, NA, NA, 0, 0, NA, NA, NA),
  expected = c(NA, NA, 15, 15, 15, NA, NA, 0, 0, 0),
  how = rep(c("Mean of 10 and 20.", "All-zero history gives zero."), each = 5)
)
custom <- create_custom_model(
  instructions = "Use the mean of all historical targets independently for each series.",
  llm = llm,
  input_data = data_tbl,
  combo_variables = "id",
  target_variable = "value",
  date_type = "month",
  forecast_horizon = 3,
  name = "business_mean",
  validation_examples = examples
)

validation_examples is one plain table, easy for a person or coding agent to write. Each example value is one worked case. History rows carry the observed Target and leave expected missing; future rows leave Target missing and carry the independently calculated expected forecast. There is no recycling or inferred answer. The target column may also use the target_variable name. Add series columns named like combo_variables for multi-series cases: one series validates local logic and two or more validate global logic. Driver columns named like external_regressors are filled on every row. Examples named edge_... or flagged with edge_case = TRUE are edge cases. For a deliberate error case, set expected_error = "fit:zero_denominator" on its rows and leave expected missing. Optional how and tolerance (at most 0.01) columns document and loosen a case. Two to eight examples including an edge case are required; every supplied expectation stays fixed during repair.

custom_model_control() has five settings: max_attempts = 3L, validation_budget = 60, package_policy = "installed_finnts", required_properties = character() and forecast_approach = NULL. Build a fresh control to change settings; explicit-setting markers distinguish supplied values from defaults. Supply approval and control by name. Unknown arguments fail. validation_budget is not a hard timeout. Custom-model creation supports one current implementation; obsolete development arguments and artifacts must be recreated, not migrated through compatibility bridges.

In an interactive R session, answer y/yes or n/no, without typing hashes. Case and surrounding whitespace are ignored; blank or EOF declines. details shows readable source and complete examples without executing or regenerating anything. edit collects a requested change and prepares a new review within the cumulative proposal limits. Existing data/capability constraints stay fixed. The default display omits instructions, schema/package inventories and source dumps; a short measured summary follows success. Outside an interactive session, approval = "manual" fails before contacting the LLM; scripts, schedulers and coding agents use approval = "automatic". See ?create_custom_model for the complete contract.

Deterministic Authoring Mechanics

The current authoring contract uses typed structured responses. The default control$package_policy = "installed_finnts" freezes installed direct FinnTS Depends, Imports and Suggests, plus the existing R base/recommended allowance. It does not admit arbitrary installed or transitive-only packages. Use custom_model_control(package_policy = "base_only") to restrict generated code to base R.

FinnTS parses source to derive only the packages actually referenced through pkg::function; the LLM no longer supplies a dependency list. The resulting packages and complete assembled source are hashed before candidate review and authorization. Nothing is installed or attached. Namespace availability and public exports are checked after authorization, before candidate bodies execute. Namespace loading can itself have side effects; it is not a sandbox.

FinnTS also supplies fit/predict signatures, prediction row IDs, and generated example dates, keys, declared-mode coverage and tolerance. The LLM returns calculation bodies, optional helpers, synthetic value arrays, independent expected answers and business rationale. Prediction bodies return finite numeric vectors in request order; the fixed wrapper rejects wrong lengths/types rather than recycling, clipping or imputing. The full reusable model remains ordinary R.

Authoring metadata lists only the workflow’s actual input columns: Date, Combo, Target and declared external regressors. Prediction excludes Target. Internal prepared features such as Date_index.num are not advertised as implicit inputs; an explicitly declared driver with that name remains valid. For a six-observation trend, generated code can construct positions 1:6 and predict at 6 + context$forecast_step, without assuming a prepared feature exists. Only the outgoing metadata copy is filtered. Captured data, saved metadata, accepted examples, approved model identities and older authoring protocols stay unchanged.

For new definitions, FinnTS supplies fit and prediction rows sorted by series and Date, with integer context$forecast_step values derived from each series’ current training cutoff. It restores the caller’s original output order. Each included series must supply its complete forecast horizon; partial-date requests fail before prediction source executes. The runtime does not synthesize missing drivers or infer fiscal calendars. Daily through yearly cadences use their saved period labels. Separate series batches are allowed when the business rule permits them, not as a guarantee that pooled rules are independent. Later refits derive new cutoffs without changing model identity. Saved older definitions keep their original ordering and partial-request behavior. Internal resampling may have shorter scoring windows. FinnTS supplies existing predictor rows through the full horizon, then retains only the original scoring rows. It does not extend training history, expose assessment targets to generated prediction code, fill missing drivers, or change forecast-selection policy.

Body fields prefer calculation statements, but new proposals also accept one complete function with the exact signature function(data, context, parameters) for fit or function(object, new_data, context) for prediction. No defaults, ellipsis, reordered names or extra arguments are allowed in this alternate form. FinnTS extracts that function body once using parsed syntax, before hashing, review and authorization. It never calls the function to discover its result. Equivalent complete-function and body inputs yield the same canonical candidate.

Ambiguous signatures, extra top-level expressions alongside a function, and direct nested function returns produce source_body_format feedback before execution. Local assigned helper functions and callbacks remain valid. Fitted state must still be portable data and prediction must still return a numeric vector. Saved-source replay does not normalize or silently repair old models; their exact source, identity and authorization remain unchanged. This support does not introduce another authoring protocol or any new public argument.

Generated examples are native nested objects and arrays in the structured response, not JSON embedded inside a text field. series remains an array even for one series. FinnTS validates required fields, lengths and captured numeric, character/factor or logical predictor types locally as well. The missing-data policy is fixed to rejection of required missing inputs; the LLM cannot return a blank replacement. This adds neither cleaning nor imputation.

The arbitrary passive fixed-parameter map uses parameters_json to support structured parameter values. Invalid JSON identifies that field explicitly; malformed strings are not repaired by guessing braces or stripping text. Expected answers remain independent of candidate execution. Structural validation does not independently prove their arithmetic. New calls treat LLM-generated numerical expectations as diagnostics: mismatches produce a user warning, not rejection or a request to rewrite the formula. Measured errors remain in the validation evidence, and the warning is retained in model$validation$checks$llm_example_warning. Matching incorrect code and answers can still pass finite comparisons; a warning alone cannot identify which side is wrong.

Bring your own validation examples. Calculate expected outputs with business logic reviewed independently of the generated model, then pass the materialized cases as one table through validation_examples. This is an examples table, not a validation-function callback. Caller-provided output mismatches remain mandatory and can trigger bounded source repair. These exact examples are included in the initial contract request, so the provider can align history requirements and preserve caller error codes/phases before generating source. It must not invent extra business error declarations without matching caller error cases. Include at least two cases, every declared mode, and meaningful edges such as zeros, negative values or calendar boundaries. For example, these independently specified checks describe a three-point mean:

dates <- as.Date(c("2020-01-01", "2020-02-01", "2020-03-01", "2020-04-01", "2020-05-01"))
validation_examples <- data.frame(
  example = rep(c("mean", "edge_zero"), each = 5),
  Date = rep(dates, 2),
  Target = c(10, 20, 30, NA, NA, 0, 0, 0, NA, NA),
  expected = c(NA, NA, NA, 20, 20, NA, NA, NA, 0, 0),
  how = rep(c("(10 + 20 + 30) / 3 = 20", "The mean of three zeros is zero."), each = 5)
)

Use those cases only for that rule and matching cadence/horizon, not for arbitrary custom logic. Caller examples are also provider-visible; keep sensitive real observations out of examples you do not want sent to the provider. Technical failures such as broken R, invalid output shape, nonportable state, required properties and coded-error mismatches still fail and may use bounded repair. Only caller-supplied numerical expectations can request an output-matching rewrite. LLM-generated numerical differences cannot.

Parameters have explicit types and units. Declared numeric weight or decay vectors become atomic R vectors; objects and arrays of records retain list structure. Synthetic numeric history and future columns share double types, including whole-valued historical discounts followed by fractional forecasts. Blank nonnumeric units normalize to not_applicable. Numeric units remain required; no business units are inferred. Malformed declarations receive bounded field-specific diagnostics before contract acceptance.

Required business errors are separate outcomes, not placeholder zero forecasts. An error_policy binds the code and fit/predict stage, and each mode must include valid numerical controls plus tests for its declared errors. Generated error cases use empty expected arrays. Caller error cases set the table’s expected_error column to "predict:zero_denominator" on that example’s rows and leave expected missing. Candidate code raises the portable sibling helper finntsRuleError(code). An unrelated exception or a zero-returning fallback does not pass that test. Additional named error scenarios remain within the existing eight-case limit.

Row-order invariance and fitted-state serialization are mandatory. Other properties are tested when declared: constant forecasts, target-scale equivariance, independent series, fixed history windows, and relative calendars. The caller control$required_properties are frozen hard requirements. Other LLM suggestions are diagnostic only: their failed checks are reported without rejecting valid calculations. This does not weaken caller numerical cases, coded business errors, row identity, finite outputs, portability or exact-version approval. For example, use custom_model_control(required_properties = "constant_forecast") only when constant output is a genuine business requirement, not for every custom model. Calendar tests translate inputs by four years; fixed-calendar rules should not declare that property. Reports retain measured counts and expectation provenance, explicitly labelled caller-supplied or LLM-proposed and unverified. These are behavioral checks, not a general rule compiler or independent arithmetic verifier.

Stateless future-driver formulas may declare zero formula-history need. Their workflow examples still need a historical row to establish the schema and cutoff; FinnTS records that workflow minimum separately. Preprocessing casting failures stop rather than requesting different source arithmetic. Repeating the same source version and failure stops the repair loop without changing fixed answers. Contradictory contracts require a new authoring version and authorization. New source responses may explicitly report contract_conflict; this stops before execution without rewriting the accepted contract or answers. Under the new comparison policy, disagreement with an LLM-generated numerical answer is not a reason to declare a conflict: source should implement the business rule and let validation record any numerical warning. Tagged protocol questions receive package-owned facts within the existing retry budget, while unresolved business questions still require clarification. Automatic defaults cannot override explicitly supplied settings. Invalid defaults declarations are corrected as contract defects, not by changing the caller’s settings. Current-format approved models are not silently revalidated or repaired.

Generated examples use one basic and one edge scenario per declared mode, with a fixed cadence-aligned synthetic cutoff and enough provider-supplied historical values for the rule. Caller-provided examples retain their original tables, values and tolerances. No candidate execution supplies its own expected answers. Those answers and business interpretation remain fallible LLM work: this is not an independent general formula evaluator. New automatic calls supply a percentage-growth convention through the current defaults catalog, but that policy does not prove arbitrary provider prose or code follows it.

The contract requests explicit history_requirements: a nonnegative integer minimum_rows_per_series and unique positive required_lag_periods, bounded by the existing example limits. Offsets use the captured cadence. Rules without exact observed-history lookups may declare an empty lag array. FinnTS validates each example series against the minimum count and every required lag date before accepting the contract. For monthly same-month values and three annual differences, offsets 12, 24, 36 and 48 require four annual observations; seven monthly rows cannot meet that requirement.

Fit-time windows use the minimum count, not forecast-relative lag offsets. For example, “use the median of the last six actual months for every forecast month” needs minimum_rows_per_series = 6 and required_lag_periods = []. The model computes that median at fit time and repeats it across the horizon. Declaring lags 1 through 6 instead asks for observations before each future forecast date. With three forecast months, offsets 1 and 2 eventually point into the unobserved forecast window. More old observations cannot fix this.

Repair feedback explicitly identifies lag_reference = "forecast_date" and the conflicting unobserved_lag_periods. Exhaustion errors include the same bounded explanation. A provider must correct an unaccepted declaration that misrepresents the rule; FinnTS never silently drops its lags or changes expected answers. Seasonal forecast-date lookups retain their existing strict checks.

Missing coverage is rejected during contract review, before source generation. Authoring can retry an unaccepted history declaration that conflicts with structurally valid caller examples. For example, a fit-time growth window must not be described as observed lags reaching into the forecast window. The next attempt receives numerical-case row counts and impossible-lag feedback; intentional error cases are excluded from those counts. The declaration must still express the actual business rule, not weakened history needs merely to pass. Exhaustion returns an error under the existing contract-attempt limit. Only unaccepted generated examples may be replaced. Caller examples, accepted answers and the source-repair oracle never change, and no history is invented, imputed or silently extended. The declaration is testable metadata, not a proof that the LLM has interpreted every possible business rule correctly.

New source preflight also rejects obvious uninitialized protocol-state references, such as object$history <- data in fit when object was never created. Fit receives data, context and parameters; prediction receives object, new_data and context. The check never inserts an object or rewrites a formula. It inspects ordinary evaluated syntax and lexical bindings without execution, while leaving quoted/formula and unknown nonstandard-evaluation syntax alone. Uncertain control flow still requires runtime validation. Bounded source_binding evidence names the function and symbol for source repair, without arbitrary exception text.

Approved models never gain new permissions on forecast reuse. Unknown protocols, altered scaffolds and changed runtime/package metadata are rejected.

Automatic Authorization

Set approval = "automatic" to authorize generated R and bounded repairs without manual review. It never calls readline() or requests clarification: it returns a validated model or a typed actionable error. This deliberately gives up source review, not numerical validation. The code runs in the current R session with your permissions, without a sandbox or reliable interruption.

custom <- create_custom_model(
  instructions = "Use the mean of historical targets independently for each series.",
  llm = llm, input_data = data_tbl,
  combo_variables = "id", target_variable = "value",
  date_type = "month", forecast_horizon = 3,
  approval = "automatic"
)

Automatic calls use one closed, package-owned defaults catalog. Explicit supported instructions and supplied metadata take precedence; incompatible or contradictory requirements fail instead of silently widening capabilities.

Omitted Choice	Default or Required Action
Target and horizon	Required for raw input; inherited from compatible prepared metadata. Never guessed.
Series keys and cadence	Only unambiguous single-series grouping and complete regular grids can be inferred; otherwise supply them explicitly.
Forecast approach	bottoms_up for raw input; inherited for prepared runs. Hierarchy retains capability restrictions and a mixed execution pool.
Model mode	local; global/pooling requires explicit model_type and compatible metadata.
Lookback	All training history within supplied historical bounds and the current fold cutoff. Explicit supported lookback wins.
Ordinary arithmetic-average weights	Equal weights only when unspecified. A requested weighted rule with missing weights fails.
Growth basis	Percentage change, newer / older - 1, when unspecified; apply the rate multiplicatively. Explicit absolute, percentage-point or CAGR rules override this fallback.
Drivers	None unless declared by arguments or prepared metadata. No automatic feature selection or derived columns.
Required history/predictors	Reject empty, missing or nonfinite values and duplicate series/date keys; no dropping or imputation. Unsupported cleaning fails.
Zero/negative values	Valid observations; no implicit clipping or rounding.
Undefined division	Error, with no invented epsilon, cap or rate. Zero-denominator holdout WMAPE remains unavailable.
Trend, seasonality, transforms, tuning	None implicitly added; custom models train on the original-scale C1 recipe.
Dependencies	Freeze installed FinnTS eligibility, or explicit base_only; derive actual dependencies from syntax. No installation.
Worked examples	New calls scaffold basic/edge examples per mode; the provider supplies independent values/answers. Generated tolerance is 1e-8.
Caller-provided examples	Preserve expected values and tolerances; never relax them during repair.
Attempt limit	Three attempts total per contract/source phase; control$max_attempts may be 1-3.
Validation budget	60 seconds after evaluation returns; positive control$validation_budget may be at most 120. Not a hard interrupt.
Sampling	Up to three deterministic series/leaves and 10000 historical node rows. Raw hierarchy retains derived nodes; prepared hierarchy is read completely.
Name	Generate a safe descriptive name; preserve an explicit name.

Never invent business offsets, multipliers, numeric growth rates, allocation percentages, caps, commercial targets, required weights/drivers/formula terms, unspecified fiscal calendars or aggregation semantics. If essential context remains unresolved, start a new call with explicit instructions and compatible metadata. The catalog does not infer these choices from observed values or improve the requested rule.

For a prior-year seasonal value plus average growth over three years, four annual observations of 100, 120, 150 and 180 give percentage changes of 20%, 25% and 20%. Their equally weighted arithmetic mean applied to 180 gives 219. This is not CAGR, an absolute increment or extra compounding across forecast steps. The requested lookback and seasonal/calendar alignment still govern which observations are used. Undefined division still errors; no epsilon, clipping or fallback rate is introduced. Existing absolute-growth worked examples and caller-supplied answers are never rewritten to fit this convention.

The fixed contract records catalog_version, effective settings and actually applied defaults. The approved model retains their JSON in custom$validation$checks$automatic_defaults; success prints the applied choices and measured results. Semantic defaults-used declarations come from the provider and are restricted to catalog keys/values. Neither those declarations nor finite examples prove arbitrary prose/code semantics for every future input. growth_basis is recorded only when the provider declares that the percentage fallback was applied, not for explicit growth bases or non-growth rules.

Every source version receives a separate automatic authorization before execution. The record has intent_confirmed = FALSE, allow_code = TRUE and mode = "automatic", bound to its version_id; it never claims human review. Manual envelopes record the three explicit confirmation fields. Definition hashes and Model_ID do not change solely because approval policy differs, while run and Agent provenance distinguish the policies. Forecast reuse does not call the authoring LLM or regenerate source.

The only public approval values are "manual" and "automatic". Explicit NULL, "interactive" and "deferred" fail. Manual approval outside an interactive R session fails; use "automatic" there.

Example Dates and Retries

Every worked example must forecast exactly the next forecast_horizon consecutive cadence dates immediately after its last historical Date. With monthly horizon 3 and history ending December 2025, those dates are January-March 2026. A prior-year seasonal rule uses January-March 2025 values from history; it does not move the forecast window another year forward. History must contain the observations the rule needs. For a fixed 11% increase, prior-year values 10, 20 and 30 produce 11.1, 22.2 and 33.3. Zero and negative values remain valid unless explicitly excluded by the supported rule.

Examples must cover every declared model_type and no undeclared modes. A local-only contract needs only local examples, with exactly one series per example; it must not add a global example merely because FinnTS supports global models. A global-only contract needs only global examples, each with at least two series. Either single-mode contract still needs two to eight total examples, including an edge case. Mixed contracts require coverage of both declared modes.

An undeclared example mode produces specific mode feedback during contract retries. FinnTS does not silently remove or relabel the example or enable another mode to make it pass. The provider may regenerate the unaccepted proposal within the existing attempt limit; caller-provided examples must instead be corrected by the caller. Valid tolerance values are not reported as the cause of a well-formed example’s undeclared mode.

When a generated contract is rejected, the next bounded contract attempt receives safe package-owned feedback explaining the rejection category and correction. The final error includes the last safe rejection if attempts are exhausted. Raw exception messages, private values, paths and provider output are not copied into that feedback.

Feedback also names the rejected field, such as parameters_json or examples.series. Unexpected validator failures stop rather than consume more provider retries. An optional diagnostics$rejected_contract snapshot retains only known provider-visible response fields, their hash, the field/code and an omission marker. It shares the existing one-MiB diagnostic cap; oversized content is omitted without truncating an accepted oracle. Raw exceptions, Chat objects and actual training data are excluded. This snapshot is diagnostic evidence, not an accepted contract or authorization. No extra automatic file is written.

Automatic business_clarification errors retain the unanswered questions at diagnostics$rejected_contract$proposal$questions, subject to the same size limit. The normal error message also displays the questions, labelled as provider clarification rather than independently verified input errors. Each question is limited to 500 displayed characters with a truncation marker, and control characters are removed; the original diagnostic text is unchanged. No debugger-frame search is needed to see ordinary questions. Supplied metadata and catalog defaults should resolve known choices; genuinely missing business terms still require clarification rather than guessed answers.

For insufficient declared history, retries receive feedback$history with package-computed minimum rows, the supplied row counts per scenario/series, declared lags and whether all lookups can remain historical through the horizon. Generated cases use scaffold identities; caller cases use their already provider-visible names and series keys. Only counts and identifiers are included in this feedback, not observed training values. No rows or answers are filled or rewritten. New examples should use simple, independently checkable arithmetic; source repairs cannot replace a bad accepted oracle merely to pass validation.

The provider may replace invalid generated examples only before the contract is accepted. Caller-provided examples are never replaced: structurally invalid supplied examples fail when checked, without spending further provider attempts on them. A conflicting unaccepted history declaration is retried instead of blaming or changing a valid caller example; earlier protocols retain their existing history-failure behavior. After contract acceptance, code repairs must preserve its examples and tolerances. The existing date checks, numerical checks, three-attempt limits and manual or automatic authorization policy remain unchanged. Corrective feedback improves repair information; it does not guarantee successful live generation.

Source Repair Diagnostics

New source requests contain fit_body, predict_body and optional named helper functions. Body fields may use the exact complete-function alternate form above. FinnTS assembles the fixed interfaces and row-identity wrapper from parsed expressions, without evaluating them. A passive preflight checks syntax, permitted namespaces and resolvable base/lexical function calls, without guessing packages for unresolved calls. Units come from the accepted interpretation and examples: a lag of twelve monthly observations means one year, not twelve years. Parameters needed by predict() belong in fitted state.

Dependency repairs also receive bounded export-location hints from already-loaded eligible namespaces. FinnTS neither loads additional namespaces for this lookup nor substitutes functions automatically; identical exported names may have different semantics. Date lookup guidance preserves matching key classes: vapply() can drop a Date result’s class even with a Date template. These aids reduce mechanical errors, but do not prove arbitrary generated logic correct.

Every custom model uses one current calendar-helper bundle. The reviewed helper source includes:

• finntsShiftDate(dates, periods, date_type, invalid_day = "error") returns Date values in the supplied order for integral day, week, month, quarter or year offsets. Negative offsets look back; offsets must be scalar or match the dates’ length. Nonexistent month-end or leap-day dates error by default. invalid_day = "last_day" is available only when the business rule explicitly permits clamping. The helper does not infer fiscal calendars.
• finntsMatchDate(dates, history_dates, combo = NULL, history_combo = NULL) returns historical row indices, preserving request order. Pooled histories must supply both character series-key vectors. Date vectors must retain their class, and historical keys must be unique within each series. Missing keys return NA rather than filling history or changing the model’s error policy.
• finntsRowsDate(dates, history_dates, history_combo) returns a list of all matching historical row indices for each requested date. Repeated dates across different series are valid; duplicate series/date keys still error. Missing dates return integer(0). The LLM implements the actual aggregation, weighting, grouping and business missing-history behavior.

These helpers handle calendar mechanics, not the business calculation. Their source and the reserved finntsDateError helper are frozen into model identity, repair snapshots and saved approvals. They require no new package dependency or provider call during prediction and leave room for 24 provider helpers within the unchanged source-size limit.

New global contracts distinguish shared fitting from prediction scope. prediction_cohort declares series, groups with explicit stable group columns, or all fitted series. A shared regression can predict complete series batches independently; portfolio allocations generally need the full cohort. FinnTS checks required peers and group membership before generated prediction code runs. Resampling expands execution to required peers and dates while retaining the original analysis rows and scoring only original assessment rows. Local models use series scope; global models declare the scope their rule needs.

Explicit global authoring checks every supplied series, then validates generated code on a deterministic representative sample of at most 100 series within 10000 historical rows, which keeps authoring preparation fast on large cohorts. The sample always includes edge series (largest and smallest scale, most zeros, negatives, flattest, shortest), covers multi-variable combo levels when they fit, and fills the rest in hash order. Its counts, never series labels, are disclosed to the provider; forecasting later fits the approved model on every series. Regression guidance asks generated code to retain feature order, category levels, contrasts, scaling and time origin in portable fitted state. These mechanics do not choose formulas: the LLM still writes the actual model, transformations and allocation.

A genuine fixed-contract contradiction stops authoring. An explicitly reported implementation difficulty can use only remaining source attempts; it cannot change examples, extend the retry budget or bypass exact-version approval. Provider failures are separate operational errors with bounded category, HTTP status when known, phase, prompt bytes and elapsed time. Unknown causes remain unknown; raw provider messages, response bodies and headers are not retained.

The shared source guard catches explicit Date-template vapply() and plainly Date-returning sapply() simplification before execution. It preserves numeric iteration, quoted code, local shadows and explicit Date/numeric conversions; unknown expressions are not treated as proven type errors. Runtime helper conditions distinguish date_key_type, date_key_duplicate, and date_shift_invalid from genuine missing-history business errors. Together with date_class_loss, these produce bounded technical repair feedback without exposing lookup values, rewriting examples, increasing retries or bypassing approval. They are not a general proof of generated-code correctness.

Unsupported development formats require recreation, not compatibility bridges. Approved models never acquire helpers silently. Approved helper source remains unchanged during later-cutoff refits, serialization and worker execution, just like the model’s calculation source.

Repairs receive the previous source and a version-bound failure record identifying the phase, reason and, where applicable, worked-example index and mode. Fitting, prediction, arithmetic mismatch, nonportable state, roundtrip disagreement, holdout failure and elapsed-budget failure remain distinct. Up to four keyed expected/actual comparisons may be provided for fixed worked examples. Real holdout values, driver values and business-series identifiers are never added.

Recognized missing-object runtime errors now use reason = "unbound_symbol" with a separate failure$unbound_symbol value, such as history. That value must be an ASCII identifier of at most 120 bytes referenced in parsed candidate code. Names found only in strings, member names, assignment bindings or quoted/formula expressions do not qualify, and sensitive-looking identifiers are withheld. Unrecognized or localized error wording remains generic. No raw exception text, stack trace or runtime value is included in this diagnostic.

The source request instructs prediction code to bind fitted-state aliases explicitly. For example, history <- object$history makes history$Date valid; with(object$history, ...) alone exposes the table’s columns but does not create the history alias. Valid data masks and nested functions remain supported. FinnTS reports runtime evidence rather than pretending to infer every R scope, and never injects an alias or changes the generated calculation automatically.

Manual repair reviews and exhausted-attempt messages show the bounded missing identifier. Repairs still preserve the accepted contract, caller examples and attempt budget; manually repaired source still requires fresh version-specific consent. Saved failures without the new field and existing approved models remain readable without changing their source or triggering new authoring.

Messages are package-owned by default. An exact constant literal stop() message already present in source can be returned only after restrictive length/content checks. Dynamic error values, arbitrary stack traces and credential/path-like content are withheld. Preparation, missing dependencies and RDS I/O remain hard operational failures instead of being disguised as repairable business logic.

New-protocol dependency failures carry up to eight static references and 32 eligible package names within eight KiB, with explicit omission counts. They distinguish undeclared/unavailable packages, unexported/private members, forbidden operations and unresolved functions. For example, an eligible package can still have a requested function that is not exported. Repairs receive that exact reference, not advice to install the package or change the accepted formula. Call arguments and arbitrary namespace-load errors are never included.

Capture an authoring error to inspect its retained diagnostics without entering the debugger:

result <- tryCatch(
  create_custom_model(
    instructions = "Use the same calendar month in the prior year.",
    llm = llm, input_data = data_tbl,
    combo_variables = "id", target_variable = "value",
    date_type = "month", forecast_horizon = 3,
    approval = "automatic"
  ),
  error = function(error) error
)
feedback <- custom_model_feedback(result)
print(feedback[c("status", "stage", "code", "next_action", "attempts", "questions")])
if (identical(feedback$status, "approved")) custom <- result

Feedback schema 1 covers approved, review_required, needs_input, and failed. It includes available candidate/contract/request identities, resolved inputs and bounded safe details without dumping source, training rows or raw exception text. Questions are labelled provider proposals. NULL counts mean unavailable evidence, not unlimited retries; approved models do not retain attempt histories. needs_user_input is true for questions and manual review. Preflight failures without a model have unknown approval policy (NULL); known error stages are retained. Errors still raise normally and are never usable models. Full diagnostic history remains separately available in result$diagnostics when an error is captured. Both reports can contain confidential column names, provider-visible questions or explicitly chosen artifact paths.

Coding agents should relay needs_input questions and any review request to the user, then call create_custom_model() again with refined instructions or approval = "automatic". Feedback does not authorize execution, restart an interrupted action or increase budgets. LLM configuration remains per call; no global provider configuration or hard provider/execution timeout is introduced.

The bundle retains at most 12 compact attempts and one candidate/accepted-contract snapshot under a one-MiB JSON limit. Oversized snapshots have an explicit omission flag and retained hashes; validation examples themselves are never truncated. The condition carries diagnostics in memory; no data persistence or cloud upload occurs. Treat source, instructions and provider-visible examples as confidential. The snapshot excludes real training rows, fitted objects and Chat.

Source repair cannot change the rule, units, parameters, expected answers or tolerances, and each candidate still needs exact manual or automatic authorization. The three-attempt limits remain unchanged. Tests cover last-value, mean/offset, rolling and weighted means, seasonal lookup/growth, explicitly defined mean YoY growth, driver ratios and conditional rules, plus quarterly and pooled controls. These are examples of supported ordinary-R logic, not a closed model menu or a promise that every arbitrary or ambiguous request succeeds. Live-provider reliability requires separate evaluation; no sandbox or hard timeout is added.

Manual Review

Manual approval runs only in an interactive R session, using the console replies described in Experimental custom models. Business questions are asked in the same console. Changed source always requires a fresh review; expected examples stay fixed during code repairs.

Scripts, scheduled jobs and coding agents run noninteractively. Manual approval there fails before contacting the LLM; use approval = "automatic" instead. Coding agents should show the user the instructions and example table they plan to submit, and relay any authoring error back to the user.

Authoring validation uses the currently loaded FinnTS functions in the current R session, including devtools::load_all(). It starts no child process and installs no package snapshot. Load the intended development code and revalidate after changing it; FinnTS does not compare loaded functions against on-disk source. Local authoring prepares at most three deterministic series; explicit global authoring prepares a representative sample of at most 100 series. Both enforce at most 10000 total historical rows, using a separate temporary root for raw data. Existing prepared runs are read without writes. Explicit hierarchy authoring samples up to three raw leaf series and retains every derived node; a prepared hierarchy is read completely. The 10000-row limit applies to all historical node rows, and larger hierarchies are rejected rather than truncated. The configured provider receives instructions, column types and fixed examples during generation, not automatically raw Target/regressor values or business series labels. The return is passive and serializable; it contains measured summaries but no fitted state, raw data, callbacks or Chat.

Generated R runs in your current session with access to session/environment values and your account’s filesystem/network permissions. There is no sandbox, crash isolation or reliable automatic interruption; stuck code may require restarting R. control$validation_budget is an elapsed acceptance budget checked after evaluation returns, not an interrupting timeout. Ordinary options, RNG, working directory and library paths are restored on normal return/error, but arbitrary global, environment, filesystem or native-code side effects cannot be rolled back. Manual mode requires review before consenting; automatic mode explicitly waives that review. Sample validation is evidence, not a proof of all future behavior. No packages are installed and authoring does not register the result or publish a forecast. Select its alias explicitly through models_to_run when supplying custom_models = list(business_mean = custom) to standard preparation.

Custom models train on a dedicated raw recipe, C1. Add it to recipes_to_run (for example c("R1", "C1"), or "C1" for a custom-only run). prep_data() writes the C1 files alongside any R1/R2 files, so built-in and custom models can share one run without sharing a representation:

Setting	Effect on custom models (C1)	Effect on built-in models (R1/R2)
recipes_to_run	Must include "C1", otherwise preparation errors	Need at least one R recipe; built-ins on a C1-only run error
external_regressors	Does not restrict C1 availability; each custom model uses its declared drivers from eligible original input columns	Selects drivers for R1/R2
stationary, box_cox, multistep_horizon, feature_selection	Not applied; C1 stays on the original target scale	Applied as usual
clean_missing_values	Cleans historical original-scale values; missing required future drivers still error	Applied as usual
clean_outliers	Not applied; C1 keeps observed outliers	Applied as usual
average_models, run_ensemble_models	Custom forecasts can join simple averages and ensembles when they have enough back tests	Unchanged
negative_forecast	FALSE floors finished custom forecasts at zero, as for built-ins; authoring validation checks unclipped output	Unchanged
parallel_processing, inner_parallel	Sequential, "local_machine" and "spark" are supported, with or without inner_parallel	Unchanged
Storage and formats	Local/mounted paths or supported Blob/M365 storage_object; CSV/RDS/Parquet data and RDS model objects	Unchanged

A prepared C1 recipe with no selected custom model is ignored with a warning. forecast_time_series() exposes stationary, retaining its existing TRUE default for ordinary calls.

Custom workflows preserve Date and Combo plus explicitly declared predictors. They do not acquire implicit PCA, feature selection, target transforms or hidden features. Missing required data is an error. Local/global declarations constrain the enabled run modes; a definition with no compatible enabled mode fails rather than enabling a different mode. Custom fits use chronological analysis folds and fixed parameters, without tuning, source regeneration or package installation. Custom source or incomplete-prediction failures cannot be silently dropped to make a mixed run appear successful.

C1 retains eligible original numeric, character, factor, logical and Date columns; reserved outcome/assessment columns and unsupported types are excluded. Supply each declared driver’s future values through the forecast horizon. Finn neither forecasts those future drivers nor replaces missing values with zero. This remains true when built-ins use other xregs, the Agent’s first iteration uses none, or update recovery uses simple defaults. C1 source identity is pinned before recipe writes, including partial preparation: changing eligible source values requires a new standard run or Agent version, not a restart.

A manifest attached to the normal -model_workflows.rds artifact pins the effective pool, exact source content, attestations and preparation context. Standard preparation needs no separate custom registry or pool-manifest file. Reordered equivalent selections can reuse it; changed definitions, approvals or context require a new run. Custom Model_Name values keep their readable aliases, including underscores. Custom Model_ID values use custom-<full content hash>--<local|global>--C1, avoiding ambiguity with existing average-component separators. Saved workflows, fits, predictions and averages are checked before completion shortcuts; stale definitions and foreign components raise errors instead of being silently repaired.

The ordinary accuracy and plausibility policy still selects the best eligible candidate. With average_models = TRUE, custom forecasts can be averaged with other selected models, and learned ensembles can use custom back-test forecasts. Enrollment does not guarantee selection. Exact single-rule workflows must explicitly disable averaging; bottoms_up alone does not do that. Supported storage backends retain their ordinary access and package requirements.

Reviewed Hierarchy Rules

Use create_custom_model(control = custom_model_control(forecast_approach = "standard_hierarchy"), ...) or "grouped_hierarchy" to review a new local-per-node definition. Its contract explicitly permits normal reconciliation to adjust the rule’s base predictions before publishing the bottom level. An old bottom-up approval cannot be silently upgraded; changing hierarchy semantics requires a new version and human approval.

Forecast enrollment must include both compatible custom rules and eligible FinnTS models in one shared models_to_run pool. A custom-only hierarchy request errors without adding a model or switching the approach. Individual rule validation during authoring does not require a competitor and does not enroll a forecast. Different winners at different nodes, including custom winners everywhere, are valid outcomes of an eligible mixed pool.

This profile is local and univariate; custom nodes train on original-scale C1, while built-ins keep their usual settings. Simple averages and learned ensembles are allowed. Every candidate must cover every required node and forecast key. Missing coverage cannot be replaced by snaive, zero-filled or silently omitted. Existing hts mathematics and quality selection are reused. Per-series routing, pooled/global hierarchy and Agent hierarchy remain later work. See the hierarchy walkthrough for explicit controls and the compatibility evidence matrix.

Custom models in Agent iterations and updates

Supply approved objects with set_agent_info(custom_models = list(business_mean = custom)). The setup models_to_run argument is removed: built-in candidate search remains the Agent’s responsibility, while every mode-compatible enrolled custom model is mandatory from the first iteration onward and executes before built-ins. The LLM cannot omit these models or alter their approved logic. Required drivers and raw C1 are retained; ordinary built-in transformations, ranking and average-model accuracy signals remain intact. Custom-only forecasting without built-in search belongs in a standard run.

Each version’s existing CSV log always contains one custom_models names column, using NA when none are enrolled. Setup saves one complete, immutable enrollment bundle at prep_models/<project-hash>-<hash-of-agent-run-id>-custom-models.rds. The returned Agent object and dispatched worker metadata contain names/references, not untrained custom workflows or source/approval payloads. Workers load the exact version-owned bundle after dispatch. Sequential, local-machine and Spark dispatch are supported with CSV data, RDS model objects and no inner parallelism for this Agent profile. Spark still requires its ordinary mounted-storage and worker-package setup; permitting dispatch is not evidence of a live-cluster test.

Compatible updates require an identically enrolled new parent, unchanged cadence, regressors and enabled modes, and explicit allow_iterate_forecast = FALSE. The positive whole-number forecast horizon may change, subject to the approved rule’s history and driver requirements. Successful series replay their saved winners and component subsets. New series and recoverable built-in fitting or forecast-quality failures use Finn’s existing default-local fallback, with all compatible current-version custom models added first, plus enrolled foundation models whose credentials are available. The coordinator checks foundation availability once and passes it to workers for consistent default-model validation. Changed global cohorts are refitted using compatible current peers rather than reusing a pooled fit from different membership. Saved global winners are published only for surviving series; new peers still receive default-local forecasts. Required custom drivers remain mandatory. Custom rules are never retuned or regenerated; mixed retuning reuses their fixed results.

Storage/authentication failures, missing or malformed required artifacts, enrollment/identity mismatches and custom-source failures remain hard errors. Fallback cannot silently drop a custom model or fabricate successful output. Accepted defaults retain provenance and completed restarts do not refit them. Update replay provenance uses a separate logs/...-agent-custom-update.rds record: it identifies the predecessor and saved choices, not another copy of the untrained enrollment bundle. Older unreleased custom artifact layouts must be recreated with the current setup/preparation functions; they are not migrated.

Future-forecast checks

These checks apply to candidate base forecasts before reconciliation, including averages. Candidates with incomplete or non-finite predictions are ineligible. Backtests and future predictions without a supported trend are rejected when their absolute magnitude exceeds 100 times a positive robust historical scale. That scale uses the 95th percentile of absolute levels and robust level/change variation, not just the last observation. A supported trend supplies a pointwise future bound of 100 * max(robust_scale, abs(projected_reference)), so sustained growth is not rejected merely for accumulating over a long horizon. The projection comes only from history, never from the candidate being screened. Existing finite-negative handling still follows negative_forecast, but missing or infinite predictions are no longer replaced with zero.

Level and trend deviations beyond six robust reference scales create soft concerns. The reference uses the latest available max(12, 3 * seasonal_period, 2 * forecast_horizon) periods before the historical cutoff. Without supported drift, its path is seasonal naive when a complete seasonal cycle is available, otherwise a recent median level. The robust scale is the largest of the 95th percentile of absolute levels, level MAD, and one-step-change MAD. In that fallback, level-reference width is the larger of the relevant change MAD and 5% of that scale, multiplied by 6 * sqrt(horizon_step) for the comparison. Supported drift instead supplies a projected center and a width reflecting residual noise and slope variation, described below. Trend checks compare median changes over matched historical and forecast spans, with a robust scale floor; proportional references perform those comparisons on log changes.

With at least two complete finite historical cycles, a detrended seasonal profile can support additional checks. Seasonal strength must be at least 0.6 and the period at least three. For forecasts covering at least one cycle, negative profile correlation or amplitude outside one-third to three times historical amplitude creates a concern. Supported proportional trends use log-scale historical and forecast profiles, so constant proportional seasonality is not confused with growing absolute amplitude. Additive and fallback paths retain the original seasonal calculations. Unsupported checks remain unassessed. These are guardrails, not calibrated prediction intervals or guarantees of future accuracy.

For a shorter horizon with at least three future points, Finn compares only the upcoming phases. Their historical range must exceed both three times the historical phase-residual MAD and 5% of the full historical seasonal amplitude. Finn removes the robust historical deseasonalized trend from the future path, rather than fitting a free trend through those few points. Negative correlation creates a phase concern only when future variation also exceeds that noise threshold. Full-cycle amplitude concern bounds are not applied to a partial cycle.

One or two seasonal cycles remain usable. One cycle supplies a cautious reference, while repeated cycles support stronger comparisons. Weak seasonality, fewer than three future points, low-variation partial phases, and unavailable checks remain unassessed instead of making the series fail. Flat forecasts are not rejected simply because observations are noisy.

The table describes strict screening. The least-issues fallback below may relax catastrophic magnitude, but not structural failures, and assesses applicable soft checks on those usable extreme paths.

Check	Trigger	Effect	When Unassessed
Coverage	Missing, duplicate, or unexpected expected scenario/date keys in backtests or future predictions	Hard rejection	Never for a candidate being selected
Finite predictions	Any required NA, NaN, Inf, or -Inf prediction	Hard rejection	Never
Catastrophic magnitude	Absolute backtest or unsupported-trend future prediction exceeds 100 * robust_scale; supported future step exceeds 100 * max(robust_scale, abs(projected_reference))	Hard rejection	When historical scale is zero
Accuracy availability	No finite weighted MAPE from usable actuals	Hard rejection	Never
Level	Forecast departs from the supported trend reference, or seasonal-naive/recent-median fallback, beyond its horizon-adjusted width	Soft concern	Skipped after a hard failure
Zero-history level	A nonzero future forecast when the reference history is entirely zero	Soft unsupported_level concern	Zero forecasts pass; this is not a ratio-based hard rejection
Trend	Median forecast change differs from historical matched-span changes beyond six robust slope scales, using log changes for a supported proportional reference	Soft concern	Fewer than four matched periods, non-finite reference values, incompatible log-domain forecasts, or a hard failure
Seasonal amplitude	Future detrended amplitude is below one-third or above three times the historical profile	Soft concern	Insufficient cycles, strength below 0.6, period below three, short forecast, or zero historical profile amplitude
Seasonal phase	Future and historical profiles have negative correlation after the appropriate trend adjustment	Soft concern	Insufficient historical cycles or strength, period below three, undefined correlation, or insufficient points or signal for a partial horizon
Seasonal amplitude preference	Amplitude difference exceeds the tolerance learned from historical cycle variation	Tie-break only; no additional concern or rejection	Insufficient historical evidence, zero reference amplitude, or insufficient partial-horizon phase signal

For these checks the default seasonal period is daily 7, weekly 52, monthly 12, quarterly 4, and yearly 1. If multiple seasonal periods are configured, the smallest valid integer greater than one is used. The checks do not independently validate every seasonal pattern a model can learn. The trend scale is the larger of historical matched-span slope MAD and 0.05 * working_scale / matched_span. The working scale is the original normalized robust scale for additive or fallback paths, and the analogous robust scale of normalized log values for proportional paths.

Each assessed soft check has a normalized excess score. Candidate risk is the maximum of the level, trend, and seasonality scores, not their average. The concern count counts the triggered reasons; seasonal phase and amplitude can contribute two reasons. Unassessed checks do not count as concerns.

The separate in-memory Seasonal_Fidelity score measures amplitude distortion beyond historical variation. For the phases being assessed, let reference_amplitude be the reference profile range, future_amplitude the comparable future range, and cycle_amplitudes the ranges of centered historical cycle profiles. The tolerance is max(3 * MAD(cycle_amplitudes), 0.05 * reference_amplitude). Fidelity is the finite nonnegative score max(0, (abs(future_amplitude - reference_amplitude) - tolerance) / reference_amplitude). Insufficient evidence gives NA, not an artificial zero. A partial-horizon flat average can have an assessed amplitude preference even when its phase correlation is undefined. This score does not change risk or the concern count, and by itself cannot reject reuse, prevent accuracy-goal stopping, or trigger another fit.

Supported historical growth

A trend reference requires at least max(12, 3 * seasonal_period) finite, regularly spaced prepared-history values in the existing reference window. Missing dates are not bridged, and missing values are not dropped to manufacture regular history. If explicit observation flags supplied to the evaluator identify missing or unobserved fitting values, trend support is declined. Existing prepared artifacts do not establish such provenance automatically; retained imputation is still part of their evidence.

Finn compares at most two robust reference forms, without training any additional forecasting model:

• Additive drift uses normalized target values. Drift is the median of same-phase changes divided by the seasonal period, or adjacent changes for nonseasonal yearly data. Phase intercepts are medians after removing that drift, rather than an extrapolation anchored to one last-point spike.
• Proportional drift applies the same method to log(target) - log(normalization). It is considered only when all historical values are strictly positive and exceed sqrt(.Machine$double.eps) * robust_scale. Signed, zero, and relatively near-zero history is not shifted, clipped, or transformed with log1p to make this form fit.

With period p, two chronological prefixes end at n - 2q and n - q, where q = max(1, floor(p / 2)). Each predicts the next q historical values and must contain at least max(8, 2p) training values. Drift must be nonzero, have the same sign in both prefixes and the full window, and exceed twice the MAD of its per-step slope estimates, with a numerical-precision floor. Each validation block must improve MAE by at least 20% over the no-drift reference on the same dates. Comparisons occur on the original target scale, normalized by a common factor for numerical stability. Proportional drift displaces an accepted additive reference only if it improves additive MAE by at least 20% in both blocks. Zero-error and negligible comparisons retain the simpler reference. These fixed support thresholds are engineering choices, not statistical confidence statements.

For a supported reference, let sigma_error be the larger of its residual MAD and a noise floor, and sigma_slope the MAD of its per-step drift estimates. The additive floor is 5% of the existing normalized robust scale; the log-space floor is log1p(0.05). At step h, the level-comparison width is:

$$6\sqrt{h\,\sigma_{\mathrm{error}}^2+h^2\,\sigma_{\mathrm{slope}}^2}.$$

Overlapping slope estimates are not treated as independent observations to shrink uncertainty. The projected center follows the supported drift and upcoming seasonal phase; it does not widen itself in response to candidate forecasts. Reference estimation is shared across candidates through the existing evaluation cache. Nonrepresentable reference projections fall back to the original checks instead of clipping forecasts or removing the magnitude bound.

For example, a positive series with well-supported 2% monthly growth can have a reference near 126.8 after 12 months and 160.8 after 24 months when its current fitted level is 100 and it has no seasonal effect. Continuing that growth need not produce a level or absolute-slope penalty merely because the horizon is longer. A path that accelerates beyond the supported trajectory can still receive a concern or fail its pointwise magnitude bound. This reference is not substituted for the selected model’s actual forecast.

A finite zero or negative forecast under a supported proportional reference receives a level_deviation soft concern, while log-only slope and seasonal comparisons are unassessed. The forecast is not changed, and this does not introduce a new hard sign veto. Short, irregular, missing, or unstable histories and all-zero series retain the earlier fallback behavior. New scoring applies to newly evaluated predictions; a complete saved winner is still reused without retrospective reassessment. Unexpected regimes and mixed seasonal mechanisms can remain uncertain, and an eligible singleton with soft concerns can still win under the ordinary best-available policy.

Accuracy allowance

After hard failures are excluded, let best_wmape be the smallest eligible weighted MAPE as a fraction. Candidates within best_wmape + max(0.005, 0.05 * best_wmape) form the shortlist. Selection prefers lower forecast risk, then fewer concerns. Within each risk-and-concern tie, smaller seasonal amplitude distortion precedes WMAPE only when every tied candidate has a finite assessed fidelity score. Otherwise that tie retains the existing WMAPE ordering. A stable candidate identifier breaks the final tie. Missing seasonal evidence is not rewarded as zero distortion.

For example, with a best eligible WMAPE of 8.0%, the ceiling is 8.5%. A safer 8.3% candidate can beat a questionable 8.0% candidate, but an 8.8% candidate is outside the allowance. At 20% the ceiling is 21%; at 2% it is 2.5%. The absolute allowance is relatively generous for very accurate series and is an initial engineering default, not a calibrated statistical bound.

For ordinary and iterative forecasts, when every shortlisted eligible candidate has soft concerns, Finn retains the existing best-available selection. When normal selection has no winner solely because of plausibility failures, it may select a usable fallback as described below. Ordinary update acceptance is unchanged. No policy can guarantee that a good forecast exists for every input.

Least-issues fallback

Standard and iterative forecasts, and fresh default forecasts for both new series and failed updates, can select a fallback from existing complete, finite predictions with usable backtest accuracy. Missing, extra or duplicate required keys, nonfinite predictions, unavailable accuracy, and artifact/provider failures cannot be waived. If no usable candidate exists, the run still fails.

Fallback ordering is the number of distinct failed plausibility checks, maximum normalized failure severity (including magnitude beyond the existing bound), backtest WMAPE, then stable model ID. Unlike normal selection, this last-resort ranking is not restricted to the accuracy shortlist. Applicable soft checks are assessed even on usable extreme forecasts, so a previously skipped zero risk cannot make an extreme forecast look safe. Normal successful selections and their seasonal-fidelity preference remain unchanged.

The selected candidate receives the existing Best_Model = "Yes" flag. A catchable finnts_forecast_selection_fallback warning names the series, model, and concerns; parallel workers relay this warning to the caller. No extra models are fitted, no predictions are fabricated, and strictly rejected components are not introduced into averages or learned ensembles. Fallback evidence stays in memory: there are no new saved columns, status values, attributes, or tracking files. Existing forecast, accuracy and best-run values reflect the actual selected result.

Standard forecasts and saved averages

forecast_time_series() finishes through final_models(). A manual standard workflow uses the same process when final_models() is called:

1. Load prepared original-scale actuals and expected backtest/future dates. Future target values never become historical evidence.
2. For normal selection and averaging, exclude candidates with missing or duplicate required keys, non-finite predictions, catastrophic magnitude, or unavailable finite WMAPE. A model with incomplete backtest coverage cannot win or enter a simple average. Learned-ensemble inputs are screened before fitting their existing ensemble specifications.
3. Form the requested simple averages from eligible existing predictions, up to max_model_average. A non-finite component is not concealed by averaging with na.rm = TRUE.
4. Evaluate the individual, learned-ensemble, and simple-average candidates on the same dates and actuals. Apply the accuracy allowance, then rank by risk, number of concerns, the supported seasonal-amplitude preference within ties, WMAPE, and stable model identifier.
5. Mark the delivered winner Best_Model = "Yes". If it is an average, save that exact combination in the existing average-model artifact. Otherwise rank just the eligible averages by the same policy and save the winner of that subset with Best_Model = "No". This is not necessarily the lowest-WMAPE average. If no eligible average can be formed, there is no required average artifact.
6. For hierarchical forecasts, pass the selected models at every prepared hierarchy node to the existing hts solver and publish its bottom-level result. Do not run another future-quality selection after reconciliation. Format intervals and any weekly-to-daily allocation using the existing output workflow.

Individual outputs remain available with their best-model flags, but only the selected simple average is retained, not every computed combination. Standard forecasting selects among models already run; a quality failure does not fit an additional model automatically.

On retry, Finn validates the combined saved outputs before reusing a winner. A Best_Model column, an average filename, or a finite completion log alone is insufficient. All individual models marked "No" can be correct when a complete saved average is marked "Yes". Missing, incomplete, or ambiguous selection flags cause averages and selection to be rebuilt from existing predictions, without training again. Saved weekly outputs are restored to native cadence before recomputation. Complete winners are not re-evaluated for future quality, and a multi-series retry retains both previously completed and newly finalized results. A saved average whose original component outputs are unavailable requires restoration of those artifacts; it is not silently reconstructed from a different subset.

Reconciliation

Quality-aware selection happens before reconciliation. Different hierarchy nodes can choose different models or averages. Finn reconciles that selected mixture with the existing hts::combinef() nodes/groups, residual weights, and nonnegative settings, then publishes the resulting bottom-level forecasts.

There is no runtime plausibility scoring of reconciled future paths, no switch of the whole output to a uniform reconciled model family, and no replacement of individual bottom rows after the solve. Standard runs may still save per-model reconciled outputs for inspection, but those outputs are not automatically promoted by a post-reconciliation quality ranker. An outer Agent reconciliation solves its selected mixture directly.

Reconciled backtests provide normal bottom-level WMAPE reporting and run comparison. The solver consumes selected forecast values, hierarchy structure, and residual-based weights, not quality rankings. A resumed run does not require every source node’s prior ranking or repeat its future-quality assessment. Actual complete selected source forecasts are still required when a solve must run. Weekly source forecasts are evaluated at weekly cadence during selection, and a non-finite value on any daily-expanded forecast row remains non-finite when its weekly key is restored.

Reconciliation can dampen, spread, or amplify a problem. Better base forecasts can improve bottom-level output, but passing the source checks is not a guarantee that every reconciled path is plausible. Solver and ordinary artifact errors remain errors. Test-only audits can measure the resulting forecasts without changing production selection.

Iterative Agent forecasts

iterate_forecast() runs final_models() within each submitted forecast run. Iteration ranking starts with the earliest minimum-WMAPE result in the current Agent version. Later eligible iterations within 10% relative of that WMAPE can become the preferred search context when their model_avg_wmape is strictly lower; the lowest such average wins, with earlier ties retained. Missing or non-finite averages do not create an improvement. This is separate from the within-iteration 0.5-percentage-point/5% allowance: risk, concern count, and seasonal fidelity are not reassessed to rank past iterations.

For local iterations, model_avg_wmape, model_median_wmape, and model_std_wmape summarize the individual model candidates’ backtest WMAPEs, excluding simple-average forecasts. For example, ARIMA may remain best after adding an external regressor while the multivariate models improve. A lower average preserves that promising search direction because further iterations may make those models the winners; it is not a guarantee of future accuracy. Global summary fields retain their original meanings: overall run WMAPE for mean and median, and zero spread. Statistics use already-loaded current backtests; history comparisons reuse recorded metrics without reloading past forecasts. Unavailable local statistics are not fabricated from the selected model’s score.

Choosing an iteration’s settings for further optimization is distinct from replacing a saved forecast: a better saved local forecast remains protected. Global promotion is decided once for the complete iteration, then all saved global winners move together even if one series individually worsens. All globally selected series must reference one best_run_name; they may still use different models or averages from that iteration. A partial global evaluation cannot promote only its successful series. Reload, finalization, and update reject mixed global iteration metadata, including inconsistent state left by interrupted writes, rather than silently using multiple global iterations. These per-series writes are not a storage transaction. Agent comparisons, saved WMAPE, and accuracy-goal decisions retain four-decimal precision.

Normal accuracy-goal stopping and local optimization routing use completeness and WMAPE, without another soft-quality veto. Consequently, an iteration winner with soft concerns can beat an earlier winner or meet the accuracy goal; hard-invalid and incomplete results still cannot claim successful completion. Rejected runs consume iteration budget. At the limit, an eligible best-available result may be retained with concerns; if none exists, Finn fails or uses an already enabled local phase for unresolved global series. Final outer reconciliation does not initiate another quality-selection or refitting loop. Avoided evaluator calls and artifact reads are covered by tests; no fixed runtime reduction is promised.

For ordinary Agent iterations, the goal and saved-forecast comparisons use selected completed-output backtests, including daily-expanded rows when weekly forecasts are delivered daily. Within-run candidate selection stays at native cadence. Those scores are not interchangeable: the zero-actual convention and rounding can produce different WMAPEs after daily allocation. Future target placeholders never contribute to either accuracy calculation.

Updated Agent forecasts

update_forecast() follows a different fitting path but uses the same evaluator:

1. Recover the previously selected single model or average for each series from its saved source forecasts. All globally selected series must reference one winning global iteration; mixed iteration metadata fails before any global refit. Refit only the union of required components from that run with their saved settings; do not average all models listed in the earlier request. Different series or hierarchy source nodes can retain different subsets within that one iteration. Missing or ambiguous selected identities or saved fits normally require restoring the original artifacts, except for the explicit legacy compatibility path below. Reused fits do not call final_models() to search over a new candidate pool.
2. Assemble predictions before reconciliation. Preserve a saved choice when its components are hard-eligible and its selected path has no soft concerns. Local and nonhierarchical updates retain this strict reuse gate. For an inner hierarchical global update, recover a rejected source from the already-refitted component union and eligible-component averages using the ordinary accuracy shortlist and risk ranking. Average sizes respect the configured limit or the saved combination size. Soft concerns rank these replacements rather than vetoing the entire global group; hard-invalid components cannot be hidden inside an average. Repeat assessment after any existing WMAPE-triggered retuning.
3. Deduplicate quality-rejected current series into the existing new-series/default-local forecast path. Healthy siblings are retained and removed series are excluded. Quality-only rejections do not consume the ordinary execution-failure cap of max(10, ceiling(0.20 * existing_series)).
4. Run final_models() for each fresh default, including new series and failed-update replacements. Prefer a normal acceptable winner; otherwise select a usable least-issues fallback with a warning. Short-history checks that cannot be assessed do not count as failures. Structural validity and usable accuracy remain mandatory. This exception does not apply to ordinary update assessment or retuning.
5. An inner hierarchical global update needs a hard-valid selection for every current source node. If recovery cannot provide one or the hierarchy is incomplete, do not solve a partial hierarchy; report a global quality rejection and send the covered current series through the existing default-local path. Accepted hierarchies reconcile only selected rows, not both components and their average together. Existing-format source files retain the actual selected component identities for subsequent updates. Recovery counts remain in memory, and diagnostics report hard-rejection reasons. No post-reconciliation candidate switch occurs.
6. If outer reconciliation is required, reconcile the selected forecasts once and publish its result without a post-quality switch or late default reforecast. A structurally unusable default raises an explicit quality error and cannot trigger repeated fitting while its saved forecasts establish the rejection. Stable default run identities and existing forecast artifacts preserve that boundary without an acceptance/rejection column.

The optional accuracy-degradation allow_iterate_forecast workflow remains separate. Checks after refit or retune assess newly generated predictions, not old iteration winners. A published default is identified by its deterministic run name and reused only after its saved models and forecasts pass integrity checks. When submission resumes without a usable parent completion record, quality is recomputed from existing predictions. Older plausibility-only rejections, including legacy rejection metadata, can be reselected and published from saved usable predictions and fits without training again. Structurally unusable rejections retain the typed error. All-No winner flags alone do not establish rejection; incomplete or damaged results retain the appropriate repair path. No replacement status is stored in another field or artifact. Accuracy is recomputed honestly when selection changes. Quality recovery itself does not ask the LLM to waive a check. Default refitting can increase compute or configured-provider cost, and it is a bounded recovery attempt rather than a promise of a satisfactory forecast.

With parallel_processing = "spark", local-model updates and default forecasts each receive up to three Spark attempts. After three execution failures, the failing step retries once with parallel_processing = "local_machine" and inner_parallel = FALSE, preserving num_cores and all other inputs. This fallback applies independently to each step; a local-update fallback does not change the default-forecast step’s initial backend. A successful Spark attempt stops retries, and an unsuccessful local fallback raises an error. Explicit forecast-quality rejections are not backend failures and bypass these retries. Non-Spark default forecasts retain their existing three-attempt budget.

Update accuracy keeps the existing native-cadence aggregate used for refit/retune and aggregate logging. Completed-output backtests supply per-series saved-winner accuracy and local model-pool statistics. This distinction preserves the update workflow’s metric meanings when weekly output is expanded to daily rows.

Updating an earlier update

Some older hierarchical global updates saved fitted models and reconciled forecasts, but no per-series source forecasts. When all expected source artifacts are genuinely absent, Finn can reuse that run’s actual saved global model/recipe components. One component is used directly; multiple components are averaged equally at every source node. A warning identifies this policy once per global group. Requested models without saved fits are not added, and recipe identities are preserved rather than replaced with R1.

This compatibility path requires valid reconciled output covering the global series being updated and consistent saved model settings and fits. Each requested series must have unique, complete future and backtest keys matching the saved preparation calendar and train/test splits, including every day when weekly forecasts were expanded to daily output. Missing or unreadable preparation is a hard artifact error. Unrelated local winners’ rows are not required. It cannot recover heterogeneous historical choices from reconciled Best-Model rows. It does not search older Agent versions. Partial source sets, existing empty or corrupt source content, ambiguous winners, and storage/access failures cannot be treated as legacy absence. A missing optional unselected average remains valid; a schema-correct empty average represents an intentionally cleared average, not proof that the entire source set is absent.

Accepted new hierarchical updates save their pre-reconciliation components and selected averages in the existing per-series format. Unused average files are overwritten with empty tables so an interrupted refit cannot leave a second winner behind. Publication verifies readable source coverage, selected identities, fitted components, prediction keys and average arithmetic. Restart checks compare each node’s intended selection, not just the shared model pool, unless hierarchical recovery is enabled. A changed choice must then match recovery recomputed in memory from the saved source predictions, prepared history, and predecessor selections, without fitting or reconciliation. Existing source artifacts record the per-node mapping for subsequent updates. No additional tracking columns or values are stored in run logs, parent best-run records, or final run metadata. Source files are authoritative: missing or ambiguous winners and invalid components are rejected, but there is no separate provenance record to authenticate an internally consistent flag edit. The withdrawn global-update tracking columns and default_reforecast_status, if encountered in older metadata, are omitted when that metadata is next written; existing files are not migrated in place. If the raw hierarchy table omits a historical target, hierarchical global updates use the corresponding original-scale prepared source history already used for selection. Observed targets stay unchanged, and future targets remain unknown. This keeps delivered accuracy and completion logging usable for sparse series without inventing actuals or bypassing the accuracy gate. The updater requires the logger to confirm every requested global series before reporting completion.

Changing hierarchy membership

Selection recovery uses the predecessor’s identities, while reconciliation requires the current hierarchy. Removed bottom series and orphaned aggregates are excluded; input ordering or an unchanged node count does not establish identity. Generated HTS aggregate labels can be reassigned when membership or hierarchy ordering changes. Finn compares their membership on surviving bottom series: a reassigned label is treated as a new node, not as the old aggregate. Added or reintroduced nodes have no selection in the immediate predecessor and follow the default-local workflow. Their saved local result is the predecessor for the next update. This includes first/last group members and overlapping grouped aggregates, not only bottom-level additions.

For an inner global hierarchy, new source nodes inherit the predecessor’s selection when every previous source used the same single model or average. This includes the uniform selection recovered from older reconciled-only runs. Finn retains all existing choices and saves the new nodes’ component and selected average forecasts, allowing the existing global series to remain global through later updates. New source predictions must pass the same completeness, component and quality checks; reassigned-label safeguards remain in force.

This source-node coverage does not promote a newly added bottom series to a global winner: its default-local delivery routing remains unchanged. When previous source selections differ, new inner sources are evaluated from the already-refitted pool rather than inheriting another node’s identity. The whole inner hierarchy is rejected only when it cannot supply a complete hard-valid source set or a surviving generated label was reassigned. Pre-expanded hierarchy inputs retain their local/global per-node routing, followed by one outer reconciliation. The existing new-series limits, quality gates, retuning rules and coherent-global-run policy remain unchanged.

Evidence and limitations

Developer tests compare two paths on the same fake candidate outputs: accuracy-only selection followed by real hts, and quality-aware selection/averaging followed by real hts. Held-out future truth is used only by the tests, never supplied to the selector. They measure bottom-level error and shape, including effects on uninjected siblings. Small regressions run with package tests; broader stress and full-catalogue averaging timings are separate developer experiments.

A separate bounded, in-memory corner-test matrix checks exact prediction keys, signed/zero/missing-actual accuracy, threshold boundaries, seasonal alignment, optional-score ties, weekly group isolation, and replacement-selector contracts. Fixed-seed cases also check independent accuracy and ranking expectations, row-order invariance, and the inability of a hard-invalid candidate to displace a valid winner. These tests do not fit forecast models or run additional reconciliation, and passing them is not proof that every possible input or future regime change is handled correctly.

Earlier experiments exposed half-amplitude averages and short-horizon phase reversals that the original concern checks did not distinguish. Focused regressions now prefer an intact available seasonal candidate within the accuracy allowance and assess informative partial horizons using repeated historical evidence. Controls retain naturally varying seasonal amplitudes within the history-adaptive tolerance. These results do not calibrate a guarantee: one-cycle histories, fewer than three future points, or low-signal partial phases can still retain reversed timing because assessment is unsupported. A plausible alternative outside the WMAPE allowance cannot win a soft-risk or fidelity comparison. Unexpected regime changes are not known in advance, and the existing nonnegative solver floor can produce small positive values for all-zero histories. None of these limits invokes a post-reconciliation replacement model.

The number of simple averages grows as choose(N, 2) + choose(N, 3) when max_model_average = 3. The normal smoke test uses four mixed model/recipe candidates and independently checks all ten pair/triple averages. The separate developer timing workload uses every supported model/recipe output and reports averaging and evaluation separately, with fresh artifacts for every trial. Earlier timing and broad stress measurements predate the refined seasonal ranking; they are not new measurements of this revision. No new elapsed-time assertion or production timeout is added. Runtime depends on candidate count, horizon, backtests, storage, and the machine; measurements should precede any future regression limit.

Historical actuals

Checks use the existing prepared data for each series, preferring R1 when prepared and otherwise normalizing R2’s Horizon == 1 rows. Differencing and Box-Cox are reversed using existing metadata. Target_Original is used when available so outlier cleaning does not change the actuals used for evaluation; otherwise Target is used. Prior missing-value imputation is retained. Recipe and metadata files are read by exact path, without directory listing, and history is reused in memory across candidates.

Selection requires at least one candidate and at least one finite actual inside the recent reference window. An empty candidate pool or a reference window containing only missing/non-finite actuals raises a clear input error. Finite observations farther back, or future target values, cannot silently supply that window’s evidence. Partly missing windows remain usable when a finite value, including zero, is present; unavailable seasonal checks remain unassessed, and missing backtest actuals continue to receive no accuracy weight.

Original and cleaned targets have separate inverse-differencing starting values. Older second-order original-target artifacts missing those values must be regenerated from original input; already saved forecast outputs remain readable.

Rankings and explanations are held in memory. No additional diagnostic or reference files are written. The selector has an internal replaceable function contract; a public per-run custom-evaluator API is not included in this release.

Backtest weighted MAPE

A pointwise absolute percentage error is produced first, rounded to four decimal places, then weighted by the absolute actual value across expected backtest rows. The existing convention replaces zero actuals with 0.1 for this calculation. Original actuals are matched by date; missing/non-finite actuals do not contribute weight, and a candidate without a finite accuracy score is ineligible. Overlapping backtests keep their separate scenario/horizon observations. The positive-target example below illustrates the weighting.

#> Simple Back Test Results
#> # A tibble: 10 × 8
#>    Combo     Date       Model  FCST Target   MAPE Target_Total Percent_Total
#>    <chr>     <date>     <chr> <dbl>  <dbl>  <dbl>        <dbl>         <dbl>
#>  1 Country_1 2020-01-01 arima     9     10 0.1             150        0.0667
#>  2 Country_1 2020-02-01 arima    23     20 0.15            150        0.133 
#>  3 Country_1 2020-03-01 arima    35     30 0.167           150        0.2   
#>  4 Country_1 2020-04-01 arima    41     40 0.025           150        0.267 
#>  5 Country_1 2020-05-01 arima    48     50 0.04            150        0.333 
#>  6 Country_1 2020-01-01 ets       7     10 0.3             150        0.0667
#>  7 Country_1 2020-02-01 ets      22     20 0.1             150        0.133 
#>  8 Country_1 2020-03-01 ets      29     30 0.0333          150        0.2   
#>  9 Country_1 2020-04-01 ets      42     40 0.05            150        0.267 
#> 10 Country_1 2020-05-01 ets      53     50 0.06            150        0.333
#> 
#> Overall Model Accuracy by Combo
#> # A tibble: 2 × 4
#>   Combo     Model   MAPE Weighted_MAPE
#>   <chr>     <chr>  <dbl>         <dbl>
#> 1 Country_1 arima 0.0963        0.0800
#> 2 Country_1 ets   0.109         0.0733

During the simple back test process above, arima seems to be the better model from a pure MAPE perspective, but ETS ends up being the winner when using weighted MAPE. The benefits of weighted MAPE allow finnts to find the optimal model that performs the best on the biggest components of a forecast, which comes with the added benefit of putting more weight on more recent observations since those are more likely to have larger target values then ones further into the past. Another way of putting more weight on more recent observations is how Finn overlaps its back testing scenarios. This means the most recent observations are tested for accuracy in different forecast horizons (H=1, H=2, etc). More info on this in the back testing vignette.

Users can evaluate the retained model outputs with their own metrics and choose another model. The internal selector is replaceable, but this release does not expose a public per-run evaluator or persist custom selection functions.