02. Training & Evaluation (VAR).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author: Blai Ras\n",
    "# Vector Autoregressive with exogenous variables (VARX)\n",
    "\n",
    "### From the master thesis \n",
    "\n",
    "### <center>  _'Accuracy comparison between Sparse Autoregressive and XGBoost models for high-dimensional product sales forecasting'_ </center>\n",
    "\n",
    "In this notebook we present the building, training and evaluation of all the segments using the VAR model. All the needed documentation and detailed description can be found in the thesis paper.\n",
    "\n",
    "## Needed libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:14.864531Z",
     "start_time": "2021-08-31T14:28:14.493Z"
    }
   },
   "outputs": [],
   "source": [
    "library(bigtime)\n",
    "library(Metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support functions\n",
    "\n",
    "This first function receives a dataframe and returns the index of the first exogenous variable. Therefore, since the endogenous sales variables are ordered, we can perform the splitting between sales and promotion variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:14.962820Z",
     "start_time": "2021-08-31T14:28:14.921Z"
    }
   },
   "outputs": [],
   "source": [
    "find.endogenous.index <- function(data) {\n",
    "    # Finding the number of endogenous variables (only the ones who are sales)\n",
    "    logical = grepl(\"Sales\" , names(data))\n",
    "    num.endogenous <- length(logical[logical == TRUE])\n",
    "    num.endogenous <- num.endogenous+1\n",
    "    return(num.endogenous)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scaler function receives a dataframe and returns 3 arguments: the same dataframe standardized (zero mean and unit standard deviation) and the original standard deviation and mean, so we can recover the standardization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:15.266117Z",
     "start_time": "2021-08-31T14:28:15.244Z"
    }
   },
   "outputs": [],
   "source": [
    "scaler <- function(data) {\n",
    "    scaled <- scale(data)\n",
    "    return(list(\"data\"=scaled,\n",
    "               \"std\"=attr(scaled,'scaled:scale'),\n",
    "               \"mean\"=attr(scaled, 'scaled:center')))\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function performs the model building step. It receives the endogenous variable matrix (Y), the exogenous variable matrix (X) and a horizon forecast (h=4). Then, iteratively, builds a model with the specified configuration for each forecast horizon, saving them in an array which we return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:17.226450Z",
     "start_time": "2021-08-31T14:28:17.208Z"
    }
   },
   "outputs": [],
   "source": [
    "model.building <- function(Y,X,h){\n",
    "    models <-list()\n",
    "    for (i in 1:h) {\n",
    "        cat(paste0(\"\\nIteration \",i))\n",
    "        flush.console()\n",
    "        VARXfit_cv <- sparseVARX(Y=Y,\n",
    "                                 X=X,\n",
    "                                 selection = \"cv\",\n",
    "                                 h=i,\n",
    "                                 check_std = FALSE)\n",
    "        models[[i]] <- VARXfit_cv\n",
    "    }\n",
    "    return(models)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function performs the prediction step. It receives the list of models and a forecast horizon, h. Then, for each forecast horizon we call its model pair, and save the predictors in a matrix, which we return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:19.373285Z",
     "start_time": "2021-08-31T14:28:19.355Z"
    }
   },
   "outputs": [],
   "source": [
    "predict.all <- function(models,h) {\n",
    "    preds <- directforecast(models[[1]])\n",
    "    for (i in 2:h) {\n",
    "        preds <- cbind(preds,directforecast(models[[i]],h=i))\n",
    "    }\n",
    "    return(as.matrix(t(preds)))\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This two functions are modified evaluation methods. As we can see, the modification is done in both of their denominators, so when an actual value (y_true) is exactly zero, we don't return a NaN. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:19.699018Z",
     "start_time": "2021-08-31T14:28:19.679Z"
    }
   },
   "outputs": [],
   "source": [
    "MAAPE <- function(y_true, y_pred) {\n",
    "    return(mean(atan(abs((y_true-y_pred)/(y_true+0.000000000000001))))*100)\n",
    "}\n",
    "\n",
    "MAPE.divide.zero <- function(y_true, y_pred) {\n",
    "  MAPE <- mean(abs((y_true - y_pred) / (y_true+0.000000000000001)))*100\n",
    "  return(MAPE)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This following function performs the evaluation part. It receives the segment identifier (an integer), the prediction matrix, the test/actual values, the time needed for the model training and a boolean indicating if we need to evaluate standardized or not predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:20.107033Z",
     "start_time": "2021-08-31T14:28:20.088Z"
    }
   },
   "outputs": [],
   "source": [
    "evaluate <- function(segment,preds,test,time,is.scaled) {\n",
    "    \n",
    "    #These are the monthly forecast and actual values.\n",
    "    tSum <- colSums(test)\n",
    "    pSum <- colSums(preds)\n",
    "    \n",
    "    #The results vector with the variables that we will output on the .csv file: the segment identifier,  weekly RMSE, \n",
    "    #MAAPE, MASE, MAPE and MAE, and then repeated again for the monthly errors. Finally, the time and the number of \n",
    "    #variables of the segment.\n",
    "    results <- c(segment,\n",
    "                 round(rmse(test,preds),2),round(MAAPE(test,preds),2),round(mase(test,preds),2),\n",
    "                 round(MAPE.divide.zero(test,preds),2),round(mae(test,preds),2),\n",
    "                 round(rmse(tSum,pSum),2),round(MAAPE(tSum,pSum),2),round(mase(tSum,pSum),2),\n",
    "                 round(MAPE.divide.zero(tSum,pSum),2),round(mae(tSum,pSum),2),\n",
    "                 ncol(preds),\n",
    "                 time)\n",
    "    \n",
    "    #If we performed the evaluation with the predictions standardized, we save it to a file named just 'evaluation'.\n",
    "    #If not, the file is 'Raw Evaluation'\n",
    "    if (is.scaled) {\n",
    "        write.table(t(results), file = 'evaluation.csv', append = TRUE, row.names = FALSE, \n",
    "                    col.names = FALSE, sep = \",\")\n",
    "    } else {\n",
    "        write.table(t(results), file = 'Raw Evaluation.csv', append = TRUE, row.names = FALSE, \n",
    "                    col.names = FALSE, sep = \",\")\n",
    "    }\n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function controls the overall process of data preprocessing, model building & training and evaluation.\n",
    "\n",
    "First, we list all the training files with extension .csv in the indicated folder. Due that all these files have their SKU id on their name, we extract it using regex.\n",
    "\n",
    "For each SKU we have train and test files. That's why we iterate 2 by 2. So, for each SKU, we read their test pre-processed file, their original test file and their train file, unless the model already exists, which we check by looking at the Models folder.\n",
    "\n",
    "If not, then we get its number of endogenous and exogenous variables using the function ``find.endogenous.index``. Thanks to this index, we perform the split of endogenous and exogenous variables. We scale each matrices with our function ``scaler``. With the generated standard deviation and mean, we also scale the test values.\n",
    "\n",
    "We continue by creating and training the model, using the function ``model.building``. After the training step is done, we save the model using the .RDS file extension. \n",
    "\n",
    "Before proceeding with the evalatuion, we undo the first-differentiation to test our model re-escalating the predictions. In order to do so, we recover the last sales of our train set and use them with the function ``diffinv``. Now we can evaluate with standardization and without."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:21.663911Z",
     "start_time": "2021-08-31T14:28:21.645Z"
    }
   },
   "outputs": [],
   "source": [
    "train.all <- function() {\n",
    "    \n",
    "    #Load all files\n",
    "    file.list.processed <- list.files(path=\"Data/Processed Train & Test Sets\",pattern=\"*.csv\",full.names=TRUE)\n",
    "\n",
    "    #Write the header of the evaluation results files\n",
    "    Metrics <- c(\"Segment\",\"Weekly RMSE\",\"Weekly MAAPE\",\"Weekly MASE\",\"Weekly MAPE\",\"Weekly MAE\",\n",
    "                \"Monthly RMSE\", \"Monthly MAAPE\", \"Monthly MASE\", \"Monthly MAPE\", \"Monthly MAE\", \"Variables\",\"Time\")\n",
    "    write.table(t(Metrics), file = 'evaluation.csv', append = TRUE, row.names = FALSE, col.names = FALSE, sep = \",\")\n",
    "    write.table(t(Metrics), file = 'Raw Evaluation.csv', append = TRUE, row.names = FALSE, col.names = FALSE, sep = \",\")\n",
    "    \n",
    "    for(i in seq(from=1, to=length(file.list.processed), by=2)) {\n",
    "        \n",
    "        segment = as.numeric(gsub(\"[^\\\\d]+\", \"\", file.list.processed[[i]], perl=TRUE))\n",
    "        if (file.exists(paste0(\"Models/Processed/\",segment,\".rds\"))) {\n",
    "            print(paste0(\"Segment \",segment,\" already trained, skipping.\"))\n",
    "            next\n",
    "        }\n",
    "        cat(paste0('\\nStart of Segment:',segment),'\\n')\n",
    "        \n",
    "        #Read the data\n",
    "        test <- read.csv(paste0(\"Data/Processed Train & Test Sets/\",segment,\"Test.csv\"))\n",
    "        train <- read.csv(paste0(\"Data/Processed Train & Test Sets/\",segment,\"Train.csv\"))\n",
    "        test.raw <- read.csv(paste0(\"Data/Raw Train & Test Sets/\",segment,\"Test.csv\"))\n",
    "        \n",
    "        print(paste0(\"Dimensions:\",dim(train)))\n",
    "        \n",
    "        #Get the number of variables of the problem\n",
    "        endogenous.index <- find.endogenous.index(train)\n",
    "              \n",
    "        print(paste0(\"Num. of endogenous variables:\",endogenous.index))\n",
    "        \n",
    "        #Split the train sets by endogenous and exogenous variables\n",
    "        train.endo <- train[,2:endogenous.index]\n",
    "        train.exo <- train[,(endogenous.index+1):ncol(train)]\n",
    "        \n",
    "        #Data Scaling\n",
    "        #Train sets\n",
    "        scaled.endo <- scaler(train.endo)\n",
    "        scaled.exo <- scaler(train.exo)\n",
    "        \n",
    "        #Just in case there's a column with all of its values to zero (no discount, no duration)\n",
    "        scaled.exo$data[is.nan(scaled.exo$data)] = 0\n",
    "        \n",
    "        #Test set\n",
    "        scaled.test <- (test[,2:endogenous.index] - scaled.endo$mean) / scaled.endo$std\n",
    "        \n",
    "        ptm <- proc.time()\n",
    "        \n",
    "        #Build the model\n",
    "        models <- model.building(Y=scaled.endo$data,\n",
    "                                 X=scaled.exo$data,\n",
    "                                 h=4)\n",
    "        \n",
    "        time <- proc.time() - ptm\n",
    "        print(time)\n",
    "        \n",
    "        #Save the model\n",
    "        saveRDS(models, paste0(\"Models/Processed/\",segment,\".rds\"))\n",
    "        \n",
    "        #Forecast without scaling\n",
    "        preds <- predict.all(models,h=4)\n",
    "        \n",
    "        #Evaluate\n",
    "        evaluate(segment,preds,as.matrix(scaled.test),time[[3]],is.scaled=TRUE)\n",
    "        \n",
    "        # Undo Scaling\n",
    "        unscaled.preds <- (preds * scaled.endo$std) + scaled.endo$mean\n",
    "\n",
    "        # Undo First Differenciation\n",
    "        last.sales <- read.csv(paste0(\"Data/Raw Train & Test Sets/\",segment,\"Train.csv\"))[nrow(train)+1,2:endogenous.index]\n",
    "        original.preds <- diffinv(unscaled.preds,xi=last.sales)[-1,]\n",
    "\n",
    "        #Evaluate without scaling\n",
    "        evaluate(segment,original.preds,as.matrix(test.raw[,2:endogenous.index]),time[[3]],is.scaled=FALSE)\n",
    "        \n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-08T14:44:38.476690Z",
     "start_time": "2021-08-08T14:44:36.916Z"
    }
   },
   "outputs": [],
   "source": [
    "train.all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model evaluation standalone, without training\n",
    "\n",
    "This next functions perform only the model evaluation, once each model is built. They were built for test purposes, since we don't need to re-train again all the models.\n",
    "\n",
    "This first function is pretty similar to the previous evaluation one. It has been adapted to save the results in another temporal files and to perform the evaluation with or without rounding, for test purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:32.238967Z",
     "start_time": "2021-08-31T14:28:32.221Z"
    }
   },
   "outputs": [],
   "source": [
    "evaluate.one <- function(segment,preds,test,is.scaled,rounding=FALSE) {\n",
    "    \n",
    "    if (rounding) {\n",
    "        test <- round(test)\n",
    "        preds <- round(preds)\n",
    "    }\n",
    "    \n",
    "    tSum <- colSums(test)\n",
    "    pSum <- colSums(preds)\n",
    "    #Metrics <- c(\"Segment\",\"Weekly RMSE\",\"Weekly MAAPE\",\"Weekly MASE\",\"Weekly MAPE\",\"Weekly MAE\",\n",
    "    #             \"Monthly RMSE\", \"Monthly MAAPE\", \"Monthly MASE\", \"Monthly MAPE\", \"Monthly MAE\", \"Variables\",\"Time\")\n",
    "    results <- c(segment,\n",
    "                 round(rmse(test,preds),2),round(MAAPE(test,preds),2),round(mase(test,preds),2),\n",
    "                 round(MAPE.divide.zero(test,preds),2),round(mae(test,preds),2),\n",
    "                 \n",
    "                 round(rmse(tSum,pSum),2),round(MAAPE(tSum,pSum),2),round(mase(tSum,pSum),2),\n",
    "                 round(MAPE.divide.zero(tSum,pSum),2),round(mae(tSum,pSum),2),\n",
    "                 ncol(preds))\n",
    "\n",
    "    if (is.scaled) {\n",
    "        write.table(t(results), file = 'temp.csv', append = TRUE, row.names = FALSE, \n",
    "                    col.names = FALSE, sep = \",\")\n",
    "    } else {\n",
    "        write.table(t(results), file = 'Raw temp.csv', append = TRUE, row.names = FALSE, \n",
    "                    col.names = FALSE, sep = \",\")\n",
    "    }\n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this other function is quite similar to the ``train.all()``, but without the model building & training part. Therefore, it just reads the training, test and model files for performing the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:38.927783Z",
     "start_time": "2021-08-31T14:28:38.911Z"
    }
   },
   "outputs": [],
   "source": [
    "evaluate.models <- function(rounding) {\n",
    "    \n",
    "    #Load all files\n",
    "    file.list.processed <- list.files(path=\"Data/Processed Train & Test Sets\",pattern=\"*.csv\",full.names=TRUE)\n",
    "\n",
    "    #Write the header of the evaluation results files\n",
    "    Metrics <- c(\"Segment\",\"Weekly RMSE\",\"Weekly MAAPE\",\"Weekly MASE\",\"Weekly MAPE\",\"Weekly MAE\",\n",
    "                 \"Monthly RMSE\", \"Monthly MAAPE\", \"Monthly MASE\", \"Monthly MAPE\", \"Monthly MAE\", \"Variables\",\"Time\")\n",
    "    write.table(t(Metrics), file = 'temp.csv', append = TRUE, row.names = FALSE, col.names = FALSE, sep = \",\")\n",
    "    write.table(t(Metrics), file = 'Raw temp.csv', append = TRUE, row.names = FALSE, col.names = FALSE, sep = \",\")\n",
    "    \n",
    "    for(i in seq(from=1, to=length(file.list.processed), by=2)) {\n",
    "        \n",
    "        segment = as.numeric(gsub(\"[^\\\\d]+\", \"\", file.list.processed[[i]], perl=TRUE))\n",
    "        \n",
    "        cat(paste0('\\nStart of Segment:',segment),'\\n')\n",
    "        \n",
    "        #Read the data\n",
    "        test <- read.csv(paste0(\"Data/Processed Train & Test Sets/\",segment,\"Test.csv\"))\n",
    "        train <- read.csv(paste0(\"Data/Processed Train & Test Sets/\",segment,\"Train.csv\"))\n",
    "        test.raw <- read.csv(paste0(\"Data/Raw Train & Test Sets/\",segment,\"Test.csv\"))\n",
    "        \n",
    "        print(paste0(\"Dimensions:\",dim(train)))\n",
    "        \n",
    "        #Get the number of variables of the problem\n",
    "        endogenous.index <- find.endogenous.index(train)\n",
    "              \n",
    "        print(paste0(\"Num. of endogenous variables:\",endogenous.index))\n",
    "        \n",
    "        #Split the train sets by endogenous and exogenous variables\n",
    "        train.endo <- train[,2:endogenous.index]\n",
    "        train.exo <- train[,(endogenous.index+1):ncol(train)]\n",
    "        \n",
    "        #Data Scaling\n",
    "        #Train sets\n",
    "        scaled.endo <- scaler(train.endo)\n",
    "        scaled.exo <- scaler(train.exo)\n",
    "        \n",
    "        #Just in case there's a column with all of its values to zero (no discount, no duration)\n",
    "        scaled.exo$data[is.nan(scaled.exo$data)] = 0\n",
    "        \n",
    "        #Test set\n",
    "        scaled.test <- (test[,2:endogenous.index] - scaled.endo$mean) / scaled.endo$std\n",
    "        \n",
    "        #Model recover\n",
    "        models <- readRDS(paste0(\"Models/Processed/\",segment,\".RDS\"))\n",
    "        \n",
    "        #Forecast without scaling\n",
    "        preds <- predict.all(models,h=4)\n",
    "        \n",
    "        #Evaluate\n",
    "        evaluate.one(segment,preds,as.matrix(scaled.test),is.scaled=TRUE,rounding=rounding)\n",
    "        \n",
    "        # Undo Scaling\n",
    "        unscaled.preds <- (preds * scaled.endo$std) + scaled.endo$mean\n",
    "\n",
    "        # Undo First Differenciation\n",
    "        last.sales <- read.csv(paste0(\"Data/Raw Train & Test Sets/\",segment,\"Train.csv\"))[nrow(train)+1,2:endogenous.index]\n",
    "        original.preds <- diffinv(unscaled.preds,xi=last.sales)[-1,]\n",
    "\n",
    "        #Evaluate without scaling\n",
    "        evaluate.one(segment,original.preds,as.matrix(test.raw[,2:endogenous.index]),is.scaled=FALSE,rounding=rounding)\n",
    "        \n",
    "    }\n",
    "    \n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-08-31T14:28:52.375022Z",
     "start_time": "2021-08-31T14:28:42.648Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Start of Segment:10120 \n",
      "[1] \"Dimensions:150\"  \"Dimensions:1204\"\n",
      "[1] \"Num. of endogenous variables:402\"\n",
      "\n",
      "Start of Segment:10140 \n",
      "[1] \"Dimensions:150\" \"Dimensions:49\" \n",
      "[1] \"Num. of endogenous variables:17\"\n",
      "\n",
      "Start of Segment:10150 \n",
      "[1] \"Dimensions:150\"  \"Dimensions:1534\"\n",
      "[1] \"Num. of endogenous variables:512\"\n",
      "\n",
      "Start of Segment:10160 \n",
      "[1] \"Dimensions:150\" \"Dimensions:133\"\n",
      "[1] \"Num. of endogenous variables:45\"\n",
      "\n",
      "Start of Segment:10170 \n",
      "[1] \"Dimensions:150\" \"Dimensions:952\"\n",
      "[1] \"Num. of endogenous variables:318\"\n",
      "\n",
      "Start of Segment:10180 \n",
      "[1] \"Dimensions:150\" \"Dimensions:343\"\n",
      "[1] \"Num. of endogenous variables:115\"\n",
      "\n",
      "Start of Segment:11902 \n",
      "[1] \"Dimensions:150\" \"Dimensions:46\" \n",
      "[1] \"Num. of endogenous variables:16\"\n",
      "\n",
      "Start of Segment:11903 \n",
      "[1] \"Dimensions:150\" \"Dimensions:523\"\n",
      "[1] \"Num. of endogenous variables:175\"\n",
      "\n",
      "Start of Segment:11908 \n",
      "[1] \"Dimensions:150\"  \"Dimensions:1078\"\n",
      "[1] \"Num. of endogenous variables:360\"\n",
      "\n",
      "Start of Segment:11920 \n",
      "[1] \"Dimensions:150\" \"Dimensions:466\"\n",
      "[1] \"Num. of endogenous variables:156\"\n",
      "\n",
      "Start of Segment:11924 \n",
      "[1] \"Dimensions:150\" \"Dimensions:100\"\n",
      "[1] \"Num. of endogenous variables:34\"\n",
      "\n",
      "Start of Segment:11926 \n",
      "[1] \"Dimensions:150\" \"Dimensions:271\"\n",
      "[1] \"Num. of endogenous variables:91\"\n",
      "\n",
      "Start of Segment:11928 \n",
      "[1] \"Dimensions:150\" \"Dimensions:244\"\n",
      "[1] \"Num. of endogenous variables:82\"\n",
      "\n",
      "Start of Segment:11934 \n",
      "[1] \"Dimensions:150\" \"Dimensions:34\" \n",
      "[1] \"Num. of endogenous variables:12\"\n",
      "\n",
      "Start of Segment:11936 \n",
      "[1] \"Dimensions:150\" \"Dimensions:34\" \n",
      "[1] \"Num. of endogenous variables:12\"\n",
      "\n",
      "Start of Segment:11940 \n",
      "[1] \"Dimensions:150\" \"Dimensions:115\"\n",
      "[1] \"Num. of endogenous variables:39\"\n",
      "\n",
      "Start of Segment:11944 \n",
      "[1] \"Dimensions:150\" \"Dimensions:76\" \n",
      "[1] \"Num. of endogenous variables:26\"\n",
      "\n",
      "Start of Segment:11950 \n",
      "[1] \"Dimensions:150\" \"Dimensions:145\"\n",
      "[1] \"Num. of endogenous variables:49\"\n",
      "\n",
      "Start of Segment:11954 \n",
      "[1] \"Dimensions:150\" \"Dimensions:256\"\n",
      "[1] \"Num. of endogenous variables:86\"\n",
      "\n",
      "Start of Segment:11956 \n",
      "[1] \"Dimensions:150\" \"Dimensions:34\" \n",
      "[1] \"Num. of endogenous variables:12\"\n",
      "\n",
      "Start of Segment:11960 \n",
      "[1] \"Dimensions:150\" \"Dimensions:70\" \n",
      "[1] \"Num. of endogenous variables:24\"\n",
      "\n",
      "Start of Segment:11970 \n",
      "[1] \"Dimensions:150\" \"Dimensions:70\" \n",
      "[1] \"Num. of endogenous variables:24\"\n",
      "\n",
      "Start of Segment:11986 \n",
      "[1] \"Dimensions:150\" \"Dimensions:94\" \n",
      "[1] \"Num. of endogenous variables:32\"\n"
     ]
    }
   ],
   "source": [
    "evaluate.models(rounding=FALSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.0.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}