How do i quantitatively evaluate Prediction Interval accuracy?

I am using a time series data using GPR Model https://au.mathworks.com/help/stats/gaussian-process-regression-models.html and then need to quantitatively evaluated Prediction Intervals accuracy with PICP (PI coverage probability) and PINAW (PI normalized average width) for assessing the accuracy of the PIs. I need help with the writing matlab code for PICP and PINAW. TIA

Answer 1

[Kartik]

Hi,

To evaluate the accuracy of Prediction Intervals (PIs) using PICP and PINAW in MATLAB, you can follow these steps:

1. Load or generate your time series data and split it into training and testing sets.
2. Train your Gaussian Process Regression (GPR) model on the training data using the fitrgp function.
3. Use the predict function to generate mean predictions and standard deviations for the testing data.
4. Compute the lower and upper bounds of the PIs for each prediction by adding and subtracting a multiple of the standard deviation from the mean prediction.
5. Calculate the PICP by dividing the number of actual observations within the PIs by the total number of observations. The PICP should ideally be close to the specified confidence level of the PIs (e.g., 95%).
6. Compute the PINAW by averaging the widths of the PIs and dividing by the range of the data. The PINAW should ideally be small, indicating narrow and precise PIs.

Refer to the following MathWorks documentation for more info:

https://www.mathworks.com/help/stats/gaussian-process-regression-models.html#mw_e2ac9ebf-6dc7-4b5c-947e-2c146fb2423d