Verify Predictive Ability Robustness of a regARIMA Model

This example shows how to forecast a regression model with ARIMA errors, and how to check the model predictability robustness.

Load the Credit Defaults data set, assign the response (IGD) to y and the predictors AGE, CPF, and SPR to X. For illustration, specify that the response series is a regression model with AR(1) errors. To avoid distraction from the purpose of this example, assume that all predictor series are stationary.

load Data_CreditDefaults
y = Data(:,5);
X = Data(:,[1 3:4]);
T = size(X,1); % Sample size
Mdl = regARIMA(1,0,0);

Vary the validation sample size (m), and forecast responses from Mdl recursively. That is, for each validation sample size:

1. Fit the model to the data (EstMdlY).

2. Forecast responses from the estimated model (yF).

3. Compute the two performance statistics, root mean square error (RMSE) and root prediction mean square error (RPMSE).

m = 4:10;  % Validation sample lengths
rPMSE = m; % Preallocate rPMSE
rMSE = m;  % Preallocate rMSE

for k = 1:numel(m);
    yEst = y(1:(T-m(k)));    % Response data for estimation
    yVal = y((T-m(k)+1):T);  % Validation sample
    EstMdlY = estimate(Mdl,yEst,'X',X,'display','off');
    yHat = EstMdlY.Intercept + X(1:(T-m(k)),:)*EstMdlY.Beta';...
        % Estimation sample predicted values
    [e0,u0] = infer(EstMdlY,yEst,'X',X);
    yF = forecast(EstMdlY,m(k),'Y0',yEst,...
			'X0',X(1:T-m(k),:),'XF',X((T-m(k)+1):T,:));...
            % Validation sample predicted values
    rMSE(k) = sqrt(mean((yEst - yHat).^2));
    rPMSE(k) = sqrt(mean((yF - yVal).^2));
end

rMSE and rPMSE are vectors that contain the RMSE and RPMSE, respectively, for each validation sample.

Display the performance measures.

fprintf('\n m |  rMSE  |  rPMSE\n')

 m |  rMSE  |  rPMSE

fprintf('====================\n')

====================

for k = 1:length(m)
    fprintf('%2d | %0.4f | %0.4f\n',m(k),rMSE(k),rPMSE(k))
end

 4 | 0.0947 | 0.2274
 5 | 0.0808 | 0.1902
 6 | 0.0810 | 0.2036
 7 | 0.0714 | 0.1924
 8 | 0.0809 | 0.1532
 9 | 0.0720 | 0.1557
10 | 0.0899 | 0.1300

The predictive ability of this model is fairly robust because rPMSE changes slightly for increasing m. However, rMSE is less than rPMSE for all m. This signifies poor predictive ability.

Search for a better model by specifying, e.g., more AR or MA lags in the error model, and compare the PMSEs over these models. Choose the model with the lowest PMSE for a given validation sample size.