.. _input_data_format:

|popy| Data Format
####################

Many |popy| actions output several tables with one row per event (typically
a dose or observation) over a population of individuals.

The columns or fields in the data file are split into four main types:

* Required fields must be present in every data file; they identify rows
  in the dataset
* Dose event parameters define the administration of drug to the body
* Observation field parameters define the measurements 
  that should be sampled (during simulation) or 
  that contribute to the error function (during model fitting)
* All other columns represent user-defined inputs to or outputs of the
  mathematical model (|eg| measured covariates)
      
The data file values for each field can be accessed using the |cx| 
(or similar) notation in the |popy| |script_file|.

Required Fields
================

A |popy| data set requires the following fields:-

* :ref:`TYPE` - type of row
* :ref:`ID` - identity 
* :ref:`TIME` - time field

Note the names ``TYPE``, ``ID`` and ``TIME`` are the default names
of these three required fields. (These three fields are the minimum required
to uniquely define a row in the dataset, though other fields maybe be required
to define rows when there are other levels in the ``EFFECTS`` hierarchy, |eg|
when using between-occasion variability.)

If you are using pre-existing data that does not use these field headings,
you can map the names you have to ``TYPE``, ``ID`` and ``TIME``
in the |script_file| |data_fields| section.

.. _type:

TYPE
------

The ``TYPE`` field specifies the event that is happening 
in each row of the data file, namely one of:

* obs: Measurements that contribute to the log likelihood 
  as defined in the |predictions| section
* dose: Creates a dose according to the dosing functions in the
  |derivatives| section
* pred: Extra prediction data points. |popy| will output extra |px| data 
  at these time points, but they do |not| contribute to the likelihood.
* reset: Set the |sx| compartment states back to the initial values 
  (usually zero)
* reset+dose: A 'reset' immediately followed by a 'dose' event 
  at the same ``TIME``

Drug trial data typically consists of 'obs' and 'dose' rows 
with a few 'reset' rows per subject.
 
.. _id:

ID
------

The ``ID`` field value defines the individual for a given row. 
As |popy| is a |poppkpd| system, the ``ID`` field is required
because the data is split over multiple individuals to form a population.

Note that non-population analysis can be performed in |popy| 
by assigning all rows the same ``ID`` value.

.. _time:

TIME
------

The ``TIME`` field defines the time stamp for each row
and is required to be monotonically increasing
unless a |TYPE| = 'reset' or 'reset+dose' row is reached. 

.. note::

  Note that when the :ref:`ID` identifier changes between rows, 
  then an implicit 'reset' occurs.

For an example of a valid combination of TYPE/ID/TIME data see 
:numref:`table_popy_time`.

.. _table_popy_time:

.. list-table:: |popy| time reset example 
    :header-rows: 1

    * - |type|
      - |id|
      - |time|
      - comment
      
    * - obs
      - Bob
      - 0.0
      - observation at time zero
    
    * - dose
      - Bob
      - 4.0
      - dose for bob at time 4.0
      
    * - obs
      - Bob
      - 4.0
      - observation for bob at time 4.0
         
    * - obs
      - Bob
      - 8.0
      - later observation
          
    * - obs
      - Ruth
      - 0.0
      - time goes back, ok cos new ID

    * - dose
      - Ruth
      - 10.0
      - dose for Ruth at time 10.0
      
    * - obs
      - Ruth
      - 20.0
      - later observation
      
    * - reset
      - Ruth
      - 30.0
      - |sx| reset at time 30.0
      
    * - obs
      - Ruth
      - 1.0
      - observation following reset

In :numref:`table_popy_time` the time always increases 
or stays the same in consecutive rows, 
but time is allowed to go backwards after a new ID or a reset.


.. _dosing_fields:

Dosing Fields
===============

Dosing events are indicated in the data file 
by a |type| field that starts with ``dose``.

The amount of drug administered is usually specified in an |amt| field.

Note in |popy| ``AMT`` is |not| a keyword; it is just the conventional name 
(and our default label) for the dose amount used by any dosing function.

.. _table_popy_single_doses:

.. list-table:: |popy| single dose type example 
    :header-rows: 1

    * - |type|
      - |time|
      - |amt|
      - comment
      
    * - dose
      - 1.0
      - 100
      - dose of 100 at time 1.0
      
    * - dose
      - 2.0
      - 200
      - dose of 200 at time 2.0
      
    * - dose
      - 3.0
      - 100
      - dose of 100 at time 3.0

As an example, we can create a data file (:numref:`table_popy_single_doses`) 
that specifies three dose events at times [1.0, 2.0, 3.0] respectively, 
whose parameters will be 'injected' into the dosing function specified 
in the |derivatives| section of the script.

.. _obs_fields:

Observation Fields
=====================

Observation rows are indicated by a |type| field that starts with ``obs`` 
and are used by the |predictions| section of the |popy| |script_file| 
either to sample an observation (during simulation) 
or evaluate the likelihood of an observation given an estimated probability
distribution (during model fitting).

.. _table_single_obs:

.. list-table:: |popy| single observed field example 
    :header-rows: 1

    * - |type|
      - DRUG_CONC
      
    * - obs
      - 10.5
      
    * - obs
      - 20.0

    * - obs
      - 15.5
      
As an example, we can create a data file (:numref:`table_single_obs`) 
that specifies three drug concentration observations of 10.5, 20.0 and 15.5
units respectively.
The amount or concentration of drug or biomarker will be simulated
at these time points by solving the differential equations given in 
|derivatives|, and passed to the |predictions| block to define the 
observation's estimated probability distribution for sampling or evaluation.

Although the ``TYPE`` field disambiguates observation events from other
events, we may also wish to disambiguate valid observations 
that should contribute to likelihood values 
from invalid ones that should not (|eg| where data are missing).
For this purpose, |popy| data files come with a ``FLAG`` field 
that uses a value of zero to filter out invalid observations.

.. _table_single_obs_missing:

.. list-table:: |popy| single observed field missing data example 
    :header-rows: 1

    * - |type|
      - DRUG_CONC
      - DRUG_CONC_FLAG
      - comment
      
    * - obs
      - 10.5
      - 1
      - Valid observation
      
    * - obs
      - 20.0
      - 1
      - Valid observation
      
    * - obs
      - 15.5
      - 1
      - Valid observation
     
    * - obs
      - -2.0
      - 0
      - Observation out of valid range
      
As an example, we can extend our earlier data file (:numref:`table_single_obs`) 
with an invalid observation of -2.0 units that falls outside of the valid range
(:numref:`table_single_obs_missing`).
This invalid observation is identified by setting the ``DRUG_CONC_FLAG`` value
to zero. Any likelihood computations will ignore this observation.

Missing ``FLAG`` fields automatically default to a value of 1 such that all
observation rows contribute to the likelihood of the data given the model.


User-Defined Fields
=====================

The other columns of the |popy| data file are available to use in the 
:term:`verbatim` sections 
|model_params|, |states|, |derivatives| and |predictions| 
that define the mathematical model of the process.

For example see below for a simple example of 
:ref:`covariate modelling <covariates>` using the |model_params|:

.. code-block:: pyml

    MODEL_PARAMS: |
        m[X] = f[X] + f[X_Y_EFFECT]*c[Y]

Here the |mx| parameter is modelled as having a linear relationship with the 
:pyml:`c[Y]` covariate from the data file.

It is also possible to use |cx| variables in the other sections. 
One use case is when you already have |pk| parameters estimated
(from a previous study) and wish to use these |cx| variables
in the |derivatives| section
instead of estimating |mx| parameters for each individual.

Having covered the basics of how |popy| is used, the functions it provides, 
and the outputs it generates, we can now look at how to build (or grow)
a mathematical model of the process under consideration.