Mixed effects ANOVA for Latin square designs

[Saoirse Connor Desai]

Hello,

I was wondering if anyone could advise me on whether it’s appropriate to analyse data from a Latin square design with mixed effects ANOVA

Essentially I have a 4 x 4 Latin square design in which my treatment factor (Correction) is combined with the column blocking factor (Narrative). There are four combinations of Correction and Narrative which correspond to the row blocking factor.

My original plan was to analyse the data using mixed effects ANOVA, but I spoke to a statistician who told me that this wouldn’t be a sensible solution given that I do not have a crossed or nested design. I am uncertain about few things:

1) Should I use mixed effects ANOVA in the first place,
2) If mixed effects is appropriate, what’s the best way to specify the random effects term, and
3) What would be the most sensible way to construct the model,

Some additional complications are that I did not expect there to be an interaction between Correction and Narrative but there does seem to be one. Further, my N is quite small (N = 37) and – due to some issues with the randomisation algorithm – there are more participants for some combinations than for others.

I guess one option would be to remove the intercorrelations among treatments by subtracting the participant mean from each of their scores and then running this as a normal 4 x 4 ANOVA.

Any thoughts or advice would be greatly appreciated.

[henrik]

I am not sure I fully understand the nature of your latin square design without actually seeing the design figure/plot (i.e., I do not understand the meaning of treatment factor, column blocking factor, and row blocking factor). It may help to provide it, especially if there are replications (i.e., given that 4 x 4 = 16 and you have 37 participants, that is quite likely). Example here.

As long as a single unit of observation (i.e., probably participant) provides data for more than one condition, setting up a mixed model is probably doable. For this question it is important to know if one of the factors (Correction or Narrative) is a within-subjects (i.e., repeated-measures) factor or not.

Even if none of your factors is a repeated-measures factor, setting up a latin square design with mixed models is possible. An example is given here.

Note that in any case, the presence of interactions makes an analysis using a latin square approach difficult as it assumes no interaction. An interaction in the data thus invalidates the results and suggests the necessity of collecting new data allowing for an interaction, potentially with a reduced design. But I think it would be helpful to first described exactly what each participant saw to evaluate whether or not a simple mixed model analysis is appropriate and one can have a look at the interaction.

[Saoirse Connor Desai]

Hi Henrik,

So there were two repeated measures factors – Correction and Narrative. Order (the row factor – which corresponds to the specific pairing of correction and narrative that participants saw) was a between ss factor. Although participants were randomly assigned to the different orders there were, unfortunately, a different number of participants per cell.

Here is a screenshot of the design and how many participants there were per cell. (I hope you can see this!)

The presence of an interaction is a little troublesome I will probably need to run a new study with a different design.

Thank you!

• This reply was modified 6 years, 9 months ago by henrik. Reason: corrected image link
• This reply was modified 6 years, 9 months ago by henrik.

[henrik]

Where is Correction in your design? Are these the levels within Order? Does this entail that you had only four specific pairings of Correction and Narrative?

To know how to set up a mixed model it is important to know if you have replicates for each cell of the design and participant. In other words, did each participant only provide one response per Narrative level or several. For mixed models one usually wants/needs several responses per unit of observation and cell of the design (i.e., replicates) to estimate the full random effects structure.

I think to get more help the best would be to either post a mockup of your data (as in this example) or your data.

[Saoirse Connor Desai]

Correction is in each cell. For example, in participant #1 would see Order 1 and see Fire (no correction), Van (correction), Head injury (explanatory correction #1), and missing person (exp correction #2). For Order 2, however, the participant would see a different pairing of Narrative and Correction. So all in all there 16 different combinations of Correction and Narrative but each participant saw four of these. I hope this makes sense. Here is a screenshot of the first few rows of my data frame.

So each participant provided 1 response per Narrative level and one response per Correction level. There is more than one response per cell but there are not an equal number of responses in each cell (e.g., cells 1-4 have 8 whereas cells 5-8 have 13).

Here is a mock-up of the what the first 4 rows of the data frame would look like or what participant #1 would see.

mat <- data.frame(participant=factor(rep(1)),
inference=c(4, 0, 0, 1),
ord=factor(rep(1)),
narrative=factor(c(“F”, “V”, “H”, “M”)),
correction=factor(c(“NC”, “CO”, “CE”, “CL”)))

• This reply was modified 6 years, 9 months ago by henrik. Reason: corrected image link

[henrik]

I had to remove my previous response because most things I said there were kind of incorrect. But I think I this response should be correct now.

In your design you have realized all 16 combinations of the factors Correction and Narrative. However, not within-subjects, but only across subjects. This means you can estimate fixed effects for their interaction, but no random slopes for it and no within-subjects ANOVA. In fact, the absence of any replicates for any factor on the individual level, precludes basically any calculation of random-slopes. However, following the logic that in your situation the random variability is confounded with the residual variance, the following mixed model should in principle be appropriate (along the lines discussed here):
mixed(dv ~ Correction*Narrative + Order + (1|participants), data)

To make sure that the model is in fact not grossly anti-conservative one could run a simulation study from generated data without effect to see if the alpha-levels are within nominal ranges, but this might be outside what could be expected from a masters thesis (but would be pretty cool, see the previous link for some hints how to do it).

If Order had more than 4 levels, one could probably also treat it like a random-effects grouping factor (e.g., see here). But with 4 levels that seems not really advisable. You can only hope it does not really effect the results.

I hope that helps. I would probably go with it.

[Saoirse Connor Desai]

Hi,

Thank you, this is very helpful. I have tried this approach and it does work. There was one thing I was unsure about. When I use the anova() function to obtain F ratios and p values for the fixed effects and interaction, there are values for everything except Order. Is there a reason that there are no values for this fixed effect?

This study will actually be included in my PhD thesis, so the simulation study you suggest would actually be very useful. When you say to run a simulation study from generated data without effect, which effect do you mean exactly?

Many thanks for all your help.

[henrik]

I am not sure why you need the anova() function, as the return output of mixed() should already give you the Fand p values (although you can use anova() to get a specific ANOVA table). Are you sure that all the variables are of the correct type (i.e., factors are in fact factors, you can check this with str())? Are there any warnings?
Perhaps it would be best to post the output from mixed().

Regarding the simulation, I think about one where all your fixed effects have no effect. You simulate data from the Null model. So you generate data that is structurally identical to your data in all regards (same factor levels and number of participants), but has no effect besides the random effects, and then fit it with the model you use here. You do this a lot of times (e.g., 1000) and see how often the p-value is significant. Should be 5% of the times. Similar as done here: singmann.org/mixed-models-for-anova-designs-with-one-observation-per-unit-of-observation-and-cell-of-the-design/

[Saoirse Connor Desai]

Yes, that works and I can obtain F and p values. Here is the output though.

As you can see, the output for

Order

is NA. I have looked at the structure of the data and it is a factor. There are no warnings aside from the ones created by the output.

>

[henrik]

I cannot see the figure, you need to add it in another way.

Are you sure you are using the latest version of afex and R? Does it also occur with other methods (e.g., method = "S")? If yes to both, this is an error I haven’t seen so far and I will not be able to help you without reproducing it (i.e., I will need the data).

[Saoirse Connor Desai]

I believe I’m running the latest version of afex (0.18.0) and am running the latest version of R. It also occurs with methods.

This is the output that I get:

Mixed Model Anova Table (Type 3 tests, KR-method)

Model: Inference ~ Correction * Narrative + Order + (1 | participant)
Data: ldf3.2
Effect df F p.value
1 Correction 3, 99 34.38 *** <.0001
2 Narrative 3, 99 7.33 *** .0002
3 Order 3, NA NA <NA>
4 Correction:Narrative 6, 99 6.48 *** <.0001
—
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1

Would it be possible to send you the data? If so, what would be the best method of doing this?

[Saoirse Connor Desai]

I guess this could be due to the somewhat unorthodox design…

[Saoirse Connor Desai]

The output seems fine if specify all the factors as main effects rather than interactions.

• This reply was modified 6 years, 9 months ago by Saoirse Connor Desai.

[Author]

Posts