July 11, 2017 at 14:41 UTC #88
I was wondering if anyone could advise me on whether it’s appropriate to analyse data from a Latin square design with mixed effects ANOVA
Essentially I have a 4 x 4 Latin square design in which my treatment factor (Correction) is combined with the column blocking factor (Narrative). There are four combinations of Correction and Narrative which correspond to the row blocking factor.
My original plan was to analyse the data using mixed effects ANOVA, but I spoke to a statistician who told me that this wouldn’t be a sensible solution given that I do not have a crossed or nested design. I am uncertain about few things:
1) Should I use mixed effects ANOVA in the first place,
2) If mixed effects is appropriate, what’s the best way to specify the random effects term, and
3) What would be the most sensible way to construct the model,
Some additional complications are that I did not expect there to be an interaction between Correction and Narrative but there does seem to be one. Further, my N is quite small (N = 37) and – due to some issues with the randomisation algorithm – there are more participants for some combinations than for others.
I guess one option would be to remove the intercorrelations among treatments by subtracting the participant mean from each of their scores and then running this as a normal 4 x 4 ANOVA.
Any thoughts or advice would be greatly appreciated.
July 13, 2017 at 16:39 UTC #90
I am not sure I fully understand the nature of your latin square design without actually seeing the design figure/plot (i.e., I do not understand the meaning of treatment factor, column blocking factor, and row blocking factor). It may help to provide it, especially if there are replications (i.e., given that 4 x 4 = 16 and you have 37 participants, that is quite likely). Example here.
As long as a single unit of observation (i.e., probably participant) provides data for more than one condition, setting up a mixed model is probably doable. For this question it is important to know if one of the factors (
Narrative) is a within-subjects (i.e., repeated-measures) factor or not.
Even if none of your factors is a repeated-measures factor, setting up a latin square design with mixed models is possible. An example is given here.
Note that in any case, the presence of interactions makes an analysis using a latin square approach difficult as it assumes no interaction. An interaction in the data thus invalidates the results and suggests the necessity of collecting new data allowing for an interaction, potentially with a reduced design. But I think it would be helpful to first described exactly what each participant saw to evaluate whether or not a simple mixed model analysis is appropriate and one can have a look at the interaction.
July 14, 2017 at 10:05 UTC #93
So there were two repeated measures factors – Correction and Narrative. Order (the row factor – which corresponds to the specific pairing of correction and narrative that participants saw) was a between ss factor. Although participants were randomly assigned to the different orders there were, unfortunately, a different number of participants per cell.
Here is a screenshot of the design and how many participants there were per cell. (I hope you can see this!)
The presence of an interaction is a little troublesome I will probably need to run a new study with a different design.
July 14, 2017 at 10:35 UTC #98
Correctionin your design? Are these the levels within
Order? Does this entail that you had only four specific pairings of
To know how to set up a mixed model it is important to know if you have replicates for each cell of the design and participant. In other words, did each participant only provide one response per
Narrativelevel or several. For mixed models one usually wants/needs several responses per unit of observation and cell of the design (i.e., replicates) to estimate the full random effects structure.
I think to get more help the best would be to either post a mockup of your data (as in this example) or your data.
July 14, 2017 at 11:22 UTC #99
Correction is in each cell. For example, in participant #1 would see Order 1 and see Fire (no correction), Van (correction), Head injury (explanatory correction #1), and missing person (exp correction #2). For Order 2, however, the participant would see a different pairing of Narrative and Correction. So all in all there 16 different combinations of Correction and Narrative but each participant saw four of these. I hope this makes sense. Here is a screenshot of the first few rows of my data frame.
So each participant provided 1 response per Narrative level and one response per Correction level. There is more than one response per cell but there are not an equal number of responses in each cell (e.g., cells 1-4 have 8 whereas cells 5-8 have 13).
Here is a mock-up of the what the first 4 rows of the data frame would look like or what participant #1 would see.
mat <- data.frame(participant=factor(rep(1)),
inference=c(4, 0, 0, 1),
narrative=factor(c(“F”, “V”, “H”, “M”)),
correction=factor(c(“NC”, “CO”, “CE”, “CL”)))
- This reply was modified 2 months, 1 week ago by henrik. Reason: corrected image link
July 15, 2017 at 09:00 UTC #103
I had to remove my previous response because most things I said there were kind of incorrect. But I think I this response should be correct now.
In your design you have realized all 16 combinations of the factors
Narrative. However, not within-subjects, but only across subjects. This means you can estimate fixed effects for their interaction, but no random slopes for it and no within-subjects ANOVA. In fact, the absence of any replicates for any factor on the individual level, precludes basically any calculation of random-slopes. However, following the logic that in your situation the random variability is confounded with the residual variance, the following mixed model should in principle be appropriate (along the lines discussed here):
mixed(dv ~ Correction*Narrative + Order + (1|participants), data)
To make sure that the model is in fact not grossly anti-conservative one could run a simulation study from generated data without effect to see if the alpha-levels are within nominal ranges, but this might be outside what could be expected from a masters thesis (but would be pretty cool, see the previous link for some hints how to do it).
Orderhad more than 4 levels, one could probably also treat it like a random-effects grouping factor (e.g., see here). But with 4 levels that seems not really advisable. You can only hope it does not really effect the results.
I hope that helps. I would probably go with it.
July 15, 2017 at 10:50 UTC #104
Thank you, this is very helpful. I have tried this approach and it does work. There was one thing I was unsure about. When I use the anova() function to obtain F ratios and p values for the fixed effects and interaction, there are values for everything except Order. Is there a reason that there are no values for this fixed effect?
This study will actually be included in my PhD thesis, so the simulation study you suggest would actually be very useful. When you say to run a simulation study from generated data without effect, which effect do you mean exactly?
Many thanks for all your help.
July 15, 2017 at 11:16 UTC #105
I am not sure why you need the
anova()function, as the return output of
mixed()should already give you the
pvalues (although you can use
anova()to get a specific ANOVA table). Are you sure that all the variables are of the correct type (i.e., factors are in fact factors, you can check this with
str())? Are there any warnings?
Perhaps it would be best to post the output from
Regarding the simulation, I think about one where all your fixed effects have no effect. You simulate data from the Null model. So you generate data that is structurally identical to your data in all regards (same factor levels and number of participants), but has no effect besides the random effects, and then fit it with the model you use here. You do this a lot of times (e.g., 1000) and see how often the p-value is significant. Should be 5% of the times. Similar as done here: singmann.org/mixed-models-for-anova-designs-with-one-observation-per-unit-of-observation-and-cell-of-the-design/
July 15, 2017 at 15:04 UTC #106
Yes, that works and I can obtain F and p values. Here is the output though.
As you can see, the output for
is NA. I have looked at the structure of the data and it is a factor. There are no warnings aside from the ones created by the output.
July 16, 2017 at 19:46 UTC #107
I cannot see the figure, you need to add it in another way.
Are you sure you are using the latest version of
R? Does it also occur with other methods (e.g.,
method = "S")? If yes to both, this is an error I haven’t seen so far and I will not be able to help you without reproducing it (i.e., I will need the data).
July 17, 2017 at 10:29 UTC #108
I believe I’m running the latest version of afex (0.18.0) and am running the latest version of R. It also occurs with methods.
This is the output that I get:
Mixed Model Anova Table (Type 3 tests, KR-method)
Model: Inference ~ Correction * Narrative + Order + (1 | participant)
Effect df F p.value
1 Correction 3, 99 34.38 *** <.0001
2 Narrative 3, 99 7.33 *** .0002
3 Order 3, NA NA <NA>
4 Correction:Narrative 6, 99 6.48 *** <.0001
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1
Would it be possible to send you the data? If so, what would be the best method of doing this?
You must be logged in to reply to this topic.