plot_calibration#

plot_calibration(y_true, y_pred, ax=None)[source]#

Plot the calibration curve for a sample of quantile predictions.

Visualizes calibration of the quantile predictions.

Computes the following calibration plot:

Let \(p_1, \dots, p_k\) be the quantile points at which predictions in y_pred were queried, e.g., via alpha in predict_quantiles.

Let \(y_1, \dots, y_N\) be the actual values in y_true, and let \(\widehat{y}_{i,j}\), for \(i = 1, \dots, N, j = 1, \dots, k\) be quantile predictions at quantile point \(p_j\), of the conditional distribution of \(y_i\), as contained in y_pred.

We compute the calibration indicators \(c_{i, j},\) as \(c_{i, j} = 1, \text{ if } y_i \le \widehat{y}_{i,j} \text{ and } 0, \text{otherwise},\) and calibration fractions as

\[\widehat{p}_j = \frac{1}{N} \sum_{i = 1}^N c_{i, j}.\]

If the quantile predictions are well-calibrated, we expect \(\widehat{p}_j\) to be close to \(p_j\).

x-axis: interval from 0 to 1, quantile points

y-axis: interval from 0 to 1, calibration fractions

plot elements: calibration curve of the quantile predictions (blue) and the ideal calibration curve (orange), the curve with equation y = x.

Calibration curve are points \((p_i, \widehat{p}_i), i = 1 \dots, k\);

Ideal curve is the curve with equation y = x, containing points \((p_i, p_i)\).

Parameters:
y_truepd.Series, single columned pd.DataFrame, or single columned np.array.

The actual values

y_predpd.DataFrame

The quantile predictions, formatted as returned by BaseDistribution.quantile, or predict_quantiles

axmatplotlib.axes.Axes, optional (default=None)

Axes on which to plot. If None, axes will be created and returned.

Returns:
figmatplotlib.figure.Figure, returned only if ax is None

matplotlib figure object

axmatplotlib.axes.Axes

matplotlib axes object with the figure

Examples

>>> import numpy as np
>>> from sktime.datasets import load_airline
>>> from sktime.forecasting.naive import NaiveForecaster
>>> from sktime.forecasting.base import ForecastingHorizon
>>> from sktime.utils.plotting import plot_calibration

>>> y_train = load_airline()[0:24]  # train on 24 months, 1949 and 1950
>>> y_test = load_airline()[24:36]  # ground truth for 12 months in 1951

>>> # try to forecast 12 months ahead, from y_train
>>> fh = ForecastingHorizon(y_test.index, is_relative=False)

>>> forecaster = NaiveForecaster(strategy="last")
>>> forecaster.fit(y_train)  

>>> pred_quantiles = forecaster.predict_quantiles(fh=fh, alpha=[0.1, 0.25, 0.5, 0.75, 0.9])  
>>> plot_calibration(y_true=y_test.loc[pred_quantiles.index], y_pred=pred_quantiles)
On this page