Concepts¶
About this documentation
This documentation aims to be comprehensive, but be aware that there is also rich information available in docstrings.
These can be accessed at the interactive prompt with the help
function;
they are also reproduced in API reference.
Experiment structure¶
In experimentator, experiments (represented by an Experiment
instance) are organized as hierarchical tree structures.
Each section of the experiment (represented by an ExperimentSection
instance)
is a node in this tree, and its children are the sections it contains.
Levels on the tree are named;
common level names in behavioral research are 'participant'
, 'session'
, 'block'
, and 'trial'
.
For example, an experiment with two participants with two blocks of three trials each would have a tree that looks like this:
'_base' ______________1______________
/ \
'participant' ______1______ ______2______
/ \ / \
'block' ___1___ ___2___ ___1___ ___2___
/ | \ / | \ / | \ / | \
'trial' 1 2 3 1 2 3 1 2 3 1 2 3
The top level is always called '_base'
;
the leading underscore indicates that you should not have to refer to this level directly.
All other level names are arbitrary and are specified when the experiment is created.
An important principle of experimentator is that each section only handles its children,
the sections immediately below it.
In a structure with levels 'participant'
, 'block'
, and 'trial'
,
every block section knows how to create and order trials (e.g., by crossing independent variables),
but knows nothing of participants.
Likewise, every participant section organizes the blocks under it,
but lets each block figure out its constitutent trials.
The only exception to this rule is in the case of non-atomic orderings.
Note
For simplicity, this documentation uses the term trial to mean the lowest level of an experiment, even though experimentator will let you use whatever string you want to name this level.
Design¶
In experimentator, every section has a design, represented by a Design
object
(usually, these will be created for you).
Most of the time, all sections at the same level have the same design
(but see Heterogeneous experiment structures).
The design is a high-level description of one level of an experiment.
It includes everything experimentator needs to know to create the children of a section.
This consists of two things:
independent variables and an ordering method.
An experiment requires multiple Design
instances in a certain relationship to each other.
Such a collection is modeled with DesignTree
objects.
Again, you usually will not manually create these.
Independent variables¶
A central concept in experimentator (and in experimental design more generally)
is that of independent variables, or IVs.
An IV is a variable that you are explicitly varying in order to test its effects.
The easiest way to represent IVs in experimentator is using a dictionary.
Each key is a string, the name of an IV.
Each value is either a list, representing the possible values the IV can take,
or None
if the IV takes continuous values (continuous values are only possible with a design matrix).
For example:
>>> independent_variables = {
... 'congruent': [True, False],
... 'distractor': [None, 'left', 'right'],
... }
Note
In Python, dictionaries have no order.
In most cases, the order of IVs is not important and so representing IVs as dictionaries will work fine.
However, there are times when the order you specify the IVs is important.
This is the case, for example, when using a design matrix, because each column of the design matrix refers to one IV.
You will need to rely on the order of IVs in order to know which column controls which IV.
In these cases you should use one of two alternative ways of representing IVs:
using a collections.OrderedDict
, or a list of 2-tuples.
Here is an example of the latter method (equivalent to the previous example):
>>> independent_variables = [
... ('congruent', [True, False]),
... ('distractor', [None, 'left', 'right']),
... ]
When you specify your IVs, you will specify them separately for every level of the experiment.
That is, every IV is associated with a level of the experimental hierarchy.
This determines how often the IV value changes.
For example, a within-subjects experiment will probably have IVs at the 'trial'
level,
a between-subjects experiment will have IVs at the 'participant'
level,
and a mixed-design experiment will have both.
An IV at the 'participant'
level will always take the same value within each participant.
Similarly, a blocked experiment may have IVs at the 'block'
level;
these IVs will only take on a new value when a new block is reached.
IV values are ultimately passed to your run callback as a condition. A condition is a combination of specific IV values. Although you don’t need to create conditions yourself, you can think of them as dictionaries mapping IV names to values. For example, the six conditions generated by a full factorial cross of the IVs above are:
[{'congruent': True, 'distractor': None},
{'congruent': True, 'distractor': 'left'},
{'congruent': True, 'distractor': 'right'},
{'congruent': False, 'distractor': None},
{'congruent': False, 'distractor': 'left'},
{'congruent': False, 'distractor': 'right'}]
Just like IVs, different conditions apply at different levels of the experimental hierarchy.
These conditions propagate down the tree.
For example, imagine a trial has one of the conditions in the list above,
{'congruent': True, 'distractor': None}
.
The block that the trial is part of may have an additional condition, like {'practice': False}
.
When the trial is run, these conditions are effectively merged.
Note
This merging is implemented with the standard-library object collections.ChainMap
.
A ChainMap
can be accessed just like a dictionary;
this is the sense in which it is correct to say that the conditions are merged.
To continue the example, one can access the IV values without worrying about what level each IV came from:
>>> condition['congruent']
True
>>> condition['practice']
False
However, it is possible to differentiate the conditions if needed,
using the maps
attribute.
See the ChainMap
docs for details.
You might see something like this:
>>> condition.maps[0]
{'trial': 1,
'congruent': True,
'distractor': None}
>>> condition.maps[1]
{'block': 2,
'practice': False}
>>> condition.maps[2]
{'participant': 1}
Orderings¶
The second element of a design is an ordering method. The ordering method determines how children of a section wll be ordered (and possibly repeated). For example, an experiment may shuffle trials within each block, counter-balance blocks within each session, and put all sessions within each participant in the same order.
Each ordering method is a class in the experimentator.order
module.
Currently, experimentator includes
Ordering
(the base class, resulting in a deterministic order),
Shuffle
,
CompleteCounterbalance
,
Sorted
, and
LatinSquare
.
Shuffle
is usually the default, except if you’re using a design matrix,
in which case experimentator assumes you want a deterministic order and makes Ordering
the default.
Each ordering method class has different parameters, so see the specific API reference for details.
Commonly, the first argument is number
, which specifies the number of times each condition will be repeated.
For example, with the ordering method Shuffle(3)
,
each unique condition will be repeated three times, and then the order will be randomized.
Non-atomic orderings¶
The included ordering classes can be divided into two categories: atomic and non-atomic. If every ordering of sections is independent of all other orderings, then the ordering method is atomic. For example, if trials within a block are shuffled, then the ordering of trials within each block will be independent. Each block can shuffle its trials without needing to know the order of trials within the other blocks.
However, this is not the case for non-atomic orderings.
The ordering of sections using non-atomic orderings are dependent on each other.
For example, if blocks within a session are counterbalanced using CompleteCounterbalance
,
then each session cannot, on its own, determine the order of blocks within it.
Non-atomic orderings are implemented by automatically creating a new independent variable.
For example, if the 'block'
level has three conditions (e.g., one IV with three possible values)
and a CompleteCounterbalance
ordering (with number=1
),
then there are six possible orderings of blocks.
A new IV called 'counterbalance_order'
will be automatically created one level up (e.g., at the 'session'
level),
with six possible values (the integers 0-5).
Don’t forget to take this automatically-created IV into account when designing your experiment.
In the above example, if there are no other IVs at the 'session'
level, and number=1
for the 'session'
ordering,
there will still be six sessions per participant due to the six conditions defined by the 'counterbalance_order'
IV.
Only Ordering
and Shuffle
are atomic; the other ordering methods provided in experimentator are non-atomic
(the Sorted
ordering method straddles the line; it may or may not be atomic, depending on the parameter order
.
If order='ascending'
or order='descending'
,
then the ordering method is atomic as it is sorted the same way at every section.
However, if order='both'
, then it is non-atomic and a new IV {'order': ['ascending', 'descending']}
will be created).
Why use levels?¶
You may be wondering how many levels to use, or why to use them at all
(after all, flat is better than nested).
That decision must be made on a case-by-case basis.
For example, imagine your experiment has sessions of 20 trials, divided into two blocks.
As long as the order of conditions within each session is correctly specified
(for example, by using a design matrix),
using an explicit 'block'
level may not be necessary.
Alternatively, you could define a 'block'
level but not a 'trial'
level
and stick a trial loop inside the block.
However, using levels makes it possible to...
associate an IV with a level, facilitating the creation and ordering of conditions.
run code before and/or after every section at a particular level, using section context managers. For example, offer participants a break between blocks.
run experiment sections by level (using the command-line interface). For example, using blocks you could do
exp run my_exp.exp participant 1 block 2
rather than the more awkward
exp run my_exp.exp participant 1 --from 11
index the data by level, after running the experiment, using hierarchical indexing. For example, to get the third trial of the first participant’s second block you could do
experiment.dataframe.loc[(1, 2, 3), :]
or to get the first trial of the second block of every participant,
data.xs((2, 1), level=('block', 'trial'))
Heterogeneous experiment structures¶
A final concept to explain is the difference between homogeneous and heterogeneous experiment structures. In a homogeneous experiment, every section at the same level has the same design. For example, if the first block contains ten trials and the second block contains twenty, the experiment structure is heterogeneous. If the order of blocks within the first session is random but the order of blocks within the second session is counterbalanced, the experiment structure is heterogeneous. Even different possible IV values across sections is enough to break homogeneity.
Heterogeneous experiments are a little trickier to set up, but they are fully supported by experimentator. See Constructing heterogeneous experiments.