The comp and tutsum Subscripts

In order to quantify how well fit has estimated the parameters, the comp script recomputes OBJV for the true parameters and the final estimated parameters. The tutsum script then creates an HTML report containing absolute and relative errors as part of a summary of the whole experiment.

Fitted vs True f[X] values

If the Comp Script has been run, the TutSum Script output also contains a convenient table that compares the initial, fitted and true f[X] values (Table 5).

Table 5 Comparison of main initial, fitted and true f[X] values

Name	Initial	Fitted	True	Abs. Error	Prop. Error
f[KA]	1	0.215	0.2	1.48e-02	7.41%
f[CL]	1	1.79	2	2.10e-01	10.52%
f[V1]	20	55.5	50	5.46e+00	10.92%
f[Q]	0.5	1.06	1	5.64e-02	5.64%
f[V2]	100	487	80	4.07e+02	508.21%

Table 5 shows that the f[KA], f[CL], f[V1], f[Q] parameters are recovered reasonably well, in the sense that the fitted values are much closer to the true values, compared to the initial starting values. However the f[V2] fitted value is quite different from the true value.

You can also compare the proportional noise estimate:

Table 6 Comparison of fitted and true proportional noise f[X] values

Name	Initial	Fitted	True	Abs. Error	Prop. Error
f[PNOISE]	0.1	0.142	0.15	8.50e-03	5.67%

where f[PNOISE] started at 0.1 and is estimated at 0.141, which can be compared with the true value 0.15. Generally the proportional noise is identifiable in a PK/PD model because every row of the data set and the current f[PNOISE] parameter has an influence on the overall likelihood.

The comparison of the covariance matrix estimates is quite long, as it displays a row for each of the 5*5 f[X] comparisons:

Table 7 Comparison of fitted and true isv variance f[X] values

Name	Initial	Fitted	True	Abs. Error	Prop. Error
f[KA_isv]	0.05	0.139	0.1	3.93e-02	39.25%
f[KA_isv;CL_isv]	0.01	-0.0553	0.01	6.53e-02	653.38%
f[KA_isv;V1_isv]	0.01	0.0414	0.01	3.14e-02	314.14%
f[KA_isv;Q_isv]	0.01	-0.0256	0.01	3.56e-02	356.13%
f[KA_isv;V2_isv]	0.01	0.122	0.01	1.12e-01	1116.86%
f[CL_isv;KA_isv]	0.01	-0.0553	0.01	6.53e-02	653.38%
f[CL_isv]	0.05	0.0808	0.03	5.08e-02	169.21%
f[CL_isv;V1_isv]	0.01	-0.00702	-0.01	2.98e-03	29.83%
f[CL_isv;Q_isv]	0.01	-0.00606	0.02	2.61e-02	130.30%
f[CL_isv;V2_isv]	0.01	-0.178	0.02	1.98e-01	992.15%
f[V1_isv;KA_isv]	0.01	0.0414	0.01	3.14e-02	314.14%
f[V1_isv;CL_isv]	0.01	-0.00702	-0.01	2.98e-03	29.83%
f[V1_isv]	0.05	0.11	0.09	1.98e-02	21.97%
f[V1_isv;Q_isv]	0.01	-0.0827	0.01	9.27e-02	927.02%
f[V1_isv;V2_isv]	0.01	0.214	0.01	2.04e-01	2039.38%
f[Q_isv;KA_isv]	0.01	-0.0256	0.01	3.56e-02	356.13%
f[Q_isv;CL_isv]	0.01	-0.00606	0.02	2.61e-02	130.30%
f[Q_isv;V1_isv]	0.01	-0.0827	0.01	9.27e-02	927.02%
f[Q_isv]	0.05	0.317	0.07	2.47e-01	353.28%
f[Q_isv;V2_isv]	0.01	-0.324	0.01	3.34e-01	3339.14%
f[V2_isv;KA_isv]	0.01	0.122	0.01	1.12e-01	1116.86%
f[V2_isv;CL_isv]	0.01	-0.178	0.02	1.98e-01	992.15%
f[V2_isv;V1_isv]	0.01	0.214	0.01	2.04e-01	2039.38%
f[V2_isv;Q_isv]	0.01	-0.324	0.01	3.34e-01	3339.14%
f[V2_isv]	0.05	0.944	0.05	8.94e-01	1788.59%

which suggests that:

• the inter-subject variances of the clearances, f[CL_isv] and f[Q_isv], are estimated reasonably well

• the f[KA_isv] estimate is far too high (possibly due to the relative lack of data before the cmax peak of the PK curve, there being only 5 data points sampled per individual)

• the f[V1_isv] and f[V2_isv] estimates are badly estimated (which is not that surprising, since volumes of distribution are harder to estimate in PK models compared to clearances which are rates.

There can be multiple reasons for the fitted values not agreeing well with true parameters, including:

• Too few observations in the data set

• Too few individuals in the data set

• Too much noise added to the measurement data

• False minima on the likelihood surface

• A fundamental difficulty in identifying some PK parameters, such as if the model is over-parameterised relative to the data set or even unidentifiable

Given the relatively small amount of data generated (250 observation points), our results (Table 5) are adequate. As shown in the next section the Objective Function for the fitted f[X] solution is actually lower than for the true f[X] solution.

Fitted vs True Objective value

It can be difficult to know by just comparing the fitted f[X] and true f[X] values (Fitted vs True f[X] values) whether the fitting method has done well or badly. We can run a form of sanity check by computing the objective function of the true f[X] and comparing this with the objective function of the fitted f[X]. This is done by optimising the r[X] for both solutions and using the synthetic data file to compute the likelihood.

The rationale is that the fitted f[X] objective value should always be lower than the true f[X] objective value, because the fitted model estimates can take advantage of correlations in the random measurement noise to get a better fit to the synthetic data. If the fitted objective function is higher then the PoPy fitting method has ended up in a suboptimal local minimum because the known true values are a better minimum.

In this example, the true model objective function is

-881.0027

which can be compared with the fitted model objective function,

-896.8752

This indicates that the fitted f[X] pass the sanity check and perhaps justifies the lack of agreement with parameters such as f[V2]. It does not, however, say whether the global optimum solution was found, i.e. whether or not f[X] is a true global minimum of the likelihood surface.

The difference between the two objective values above is partially dependent on the amount of noise applied to the measurements (i.e. f[PNOISE]), the number of individuals simulated, the number of observations per individual and the number of parameters in the model. More random noise in the synthetic data creates more likelihood minima that are away from the true f[X] solution.

If the model is practically identifiable then increasing the number of data points and individuals should lead to a convergence between the true and fitted f[X] and the objective functions above.