The ``comp`` and ``tutsum`` Subscripts
========================================

In order to quantify *how* well ``fit`` has estimated the parameters,
the ``comp`` script recomputes OBJV for the true parameters and the 
final estimated parameters. The ``tutsum`` script then creates an HTML
report containing absolute and relative errors as part of a summary of the
whole experiment.

.. _fit_vs_true_fx_builtin:

Fitted vs True f[X] values
------------------------------

If the :ref:`comp_script` has been run, 
the :ref:`tutsum_script` output also contains a convenient table 
that compares the initial, fitted and true |fx| values 
(:numref:`table_true_vs_fit_fx_main_values_builtin`).

.. _table_true_vs_fit_fx_main_values_builtin:

.. csv-table:: Comparison of main initial, fitted and true |fx| values
    :file: ../../case-studies/quick_start/builtin_tut_example/fx_comp_main.csv
    :header-rows: 1

:numref:`table_true_vs_fit_fx_main_values_builtin` shows that
the :pyml:`f[KA], f[CL], f[V1], f[Q]` parameters are recovered reasonably well,
in the sense that the fitted values are much closer to the true values, 
compared to the initial starting values. 
However the :pyml:`f[V2]` fitted value is quite different from the true value.

You can also compare the proportional noise estimate:

.. _table_true_vs_fit_fx_noise_values_builtin:

.. csv-table:: Comparison of fitted and true proportional noise |fx| values
    :file: ../../case-studies/quick_start/builtin_tut_example/fx_comp_noise.csv
    :header-rows: 1
    
where :pyml:`f[PNOISE]` started at 0.1 and is estimated at 0.141, 
which can be compared with the true value 0.15. 
Generally the proportional noise is identifiable in a |pkpd| model because
every row of the data set and the current :pyml:`f[PNOISE]` parameter 
has an influence on the overall likelihood.

The comparison of the covariance matrix estimates is quite long,
as it displays a row for each of the 5*5 |fx| comparisons:

.. _table_true_vs_fit_fx_isv_values_builtin:

.. csv-table:: Comparison of fitted and true isv variance |fx| values
    :file: ../../case-studies/quick_start/builtin_tut_example/fx_comp_variance.csv
    :header-rows: 1
    
which suggests that:

* the inter-subject variances of the clearances, 
  :pyml:`f[CL_isv]` and :pyml:`f[Q_isv]`, 
  are estimated reasonably well
* the :pyml:`f[KA_isv]` estimate is far too high
  (possibly due to the relative lack of data 
  before the |cmax| peak of the |pk| curve, 
  there being only 5 data points sampled per individual)
* the :pyml:`f[V1_isv]` and :pyml:`f[V2_isv]` estimates 
  are badly estimated (which is not that surprising, 
  since |vols| are harder to estimate in |pk| models
  compared to |clears| which are rates.

There can be multiple reasons for the fitted values not agreeing well with 
true parameters, including:

* Too few observations in the data set
* Too few individuals in the data set
* Too much noise added to the measurement data
* False minima on the likelihood surface
* A fundamental difficulty in identifying some |pk| parameters, 
  such as if the model is over-parameterised 
  relative to the data set or even unidentifiable

Given the relatively small amount of data generated (250 observation points), 
our results (:numref:`table_true_vs_fit_fx_main_values_builtin`) are adequate. 
As shown in the next section the Objective Function 
for the fitted |fx| solution is actually lower than for the true |fx| solution.


.. _fit_vs_true_obj_builtin:

Fitted vs True Objective value
------------------------------

It can be difficult to know by just comparing the fitted |fx| and true |fx| values
(:ref:`fit_vs_true_fx_builtin`) whether the fitting method has done well or badly. 
We can run a form of sanity check by computing the objective function 
of the true |fx| and comparing this with the objective function of the fitted |fx|. 
This is done by optimising the |rx| for both solutions 
and using the synthetic data file to compute the likelihood.

The rationale is that the fitted |fx| objective value 
should always be **lower** than the true |fx| objective value, 
because the fitted model estimates can take advantage of 
correlations in the random measurement noise to get a better fit 
to the synthetic data. 
If the fitted objective function is *higher* 
then the |popy| fitting method has ended up in a suboptimal local minimum
because the known true values are a better minimum.
        
In this example, the true model objective function is

.. code-block:: pyml

    -881.0027

which can be compared with the fitted model objective function,
    
.. code-block:: pyml

    -896.8752
      
This indicates that the fitted |fx| pass the sanity check and 
perhaps justifies the lack of agreement with parameters such as :pyml:`f[V2]`. 
It does |not|, however, say whether the global optimum solution was found,
|ie| whether or not |fx| is a true global minimum of the likelihood surface.

The difference between the two objective values above is partially dependent
on the amount of noise applied to the measurements (|ie| :pyml:`f[PNOISE]`), 
the number of individuals simulated, 
the number of observations per individual 
and the number of parameters in the model. 
More random noise in the `synthetic data` creates more likelihood minima 
that are away from the true |fx| solution.

If the model is practically identifiable 
then increasing the number of data points and individuals 
should lead to a convergence between the true and fitted |fx| 
and the objective functions above.
