20
3.5 True versus predicted error in the mBEEF
functional
As previously shown in the Figure 3.1 in most of the cases the experimental
values lie within the predicted uncertainties by the mBEEF functional with
slight overestimation (large errorbars) of the predicted errors. However, the
size of uncertainties decreased significantly with the FERE. Therefore, in order
to understand the distribution of error before and after the fitting a histogram
of the true error (HmBEEF - HExpt. and HFERE
mBEEF - HExpt.) divided
by the predicted error (BEE and FERE
BEE ) is plotted in the Figure 3.2. The
histogram is a running average calculated as 38:
P(12
xi + xi+J )
J
N(xi+J − xi) , (3.3)
with xi as the statistical quantity plotted in the histogram and an intermediate
value 20 for the parameter J has been chosen.
If the predicted error matches exactly the true error then one would expect
that the distribution would be a Gaussian of unit width (shown in green in
the figure). However, in the Figure 3.2 this is not the case. As also noticed
before, the tendency of the mBEEF to overestimate the errors in manifested
in the large peak around zero in (a) which renders the mBEEF to have most
of the experimental values lie within the uncertainties.
However, with the FERE the distribution flattens out and becomes closer
to the unit Gaussian implying that the real and the predicted error are close.
The tail in the histogram indicates those cases where the predicted error is
smaller than the actual error. This is a fairly common feature of the ensemble
approach 39.
3.6 Cross validation
As pointed out before, the fitting model should be such that the data is neither
overfitted nor underfitted. Therefore, it is of utmost importance that the
quality of the fit is tested on a dataset (also called as test set) which is not
included in the fitting dataset (also called as training set). A good quality
fit should provide a reasonable prediction on a new dataset. A point worth
noticing in the current fitting scheme is that only binary compounds have
been used in fitting dataset, therefore good predictions are expected for the
new binary compounds. Additionally, reasonable predictions can be expected