Back Testing and Hyperparameter Tuning

Most models automatically included in finnts, including all multivariate models, have various hyperparameters with values that need to be chosen before a model is trained. Finn solves this by leveraging the tune package within the tidymodels ecosystem.

When prep_models() is ran, hyperparameters and back test splits are calculated and written to disk. You can get the results by calling get_prepped_models().

#> Loading required package: modeltime
#> Finn Submission Info
#> • Experiment Name: finnts_fcst
#> • Run Name: get_prepped_models-20240315T171257Z
#> 
#> ℹ Prepping Data
#> ✔ Prepping Data [952ms]
#> 
#> ℹ Creating Model Workflows
#> ✔ Creating Model Workflows [108ms]
#> 
#> ℹ Creating Model Hyperparameters
#> ✔ Creating Model Hyperparameters [136ms]
#> 
#> ℹ Creating Train Test Splits
#> ✔ Creating Train Test Splits [348ms]
#> 
#> # A tibble: 31 × 4
#>    Run_Type        Train_Test_ID Train_End  Test_End  
#>    <chr>                   <dbl> <date>     <date>    
#>  1 Future_Forecast             1 2015-06-01 2015-12-01
#>  2 Back_Test                   2 2015-05-01 2015-06-01
#>  3 Back_Test                   3 2015-04-01 2015-06-01
#>  4 Back_Test                   4 2015-03-01 2015-06-01
#>  5 Back_Test                   5 2015-02-01 2015-06-01
#>  6 Back_Test                   6 2015-01-01 2015-06-01
#>  7 Back_Test                   7 2014-12-01 2015-06-01
#>  8 Back_Test                   8 2014-11-01 2015-05-01
#>  9 Back_Test                   9 2014-10-01 2015-04-01
#> 10 Validation                 10 2014-09-01 2014-10-01
#> # ℹ 21 more rows
#> [1] "Future_Forecast" "Back_Test"       "Validation"      "Ensemble"
#> # A tibble: 3 × 3
#>   Model_Name Model_Recipe Model_Workflow
#>   <chr>      <chr>        <list>        
#> 1 arima      R1           <workflow>    
#> 2 ets        R1           <workflow>    
#> 3 xgboost    R1           <workflow>
#> # A tibble: 12 × 4
#>    Model   Recipe Hyperparameter_Combo Hyperparameters 
#>    <chr>   <chr>                 <dbl> <list>          
#>  1 arima   R1                        1 <tibble [0 × 0]>
#>  2 ets     R1                        1 <tibble [0 × 0]>
#>  3 xgboost R1                        1 <tibble [1 × 4]>
#>  4 xgboost R1                        2 <tibble [1 × 4]>
#>  5 xgboost R1                        3 <tibble [1 × 4]>
#>  6 xgboost R1                        4 <tibble [1 × 4]>
#>  7 xgboost R1                        5 <tibble [1 × 4]>
#>  8 xgboost R1                        6 <tibble [1 × 4]>
#>  9 xgboost R1                        7 <tibble [1 × 4]>
#> 10 xgboost R1                        8 <tibble [1 × 4]>
#> 11 xgboost R1                        9 <tibble [1 × 4]>
#> 12 xgboost R1                       10 <tibble [1 × 4]>

The above outputs allow a Finn user to understand what hyperparameters are chosen for tuning and how the model refitting process will work. When tuning hyperparameters, Finn uses the “Validation” train/test splits, with the final parameters chosen using RMSE. For some models like ARIMA that don’t follow a traditional hyperparameter tuning process, the model is fit from scratch across all train/test splits. After hyperparameters are chosen, the model is refit across the “Back_Test” and “Future_Forecast” splits. The “Back_Test” splits are the true testing data that will be used when selecting the final “Best-Model”. “Ensemble” splits are also created as ensemble training data if ensemble models are chosen to run. Ensemble models follow a similar tuning process.