Mixed effects ANOVA for Latin square designs

Home forums Mixed Models Mixed effects ANOVA for Latin square designs

This topic contains 12 replies, has 2 voices, and was last updated by  Saoirse Connor Desai 4 months ago.

  • Author
    Posts
  • #88

    Hello,

    I was wondering if anyone could advise me on whether it’s appropriate to analyse data from a Latin square design with mixed effects ANOVA

    Essentially I have a 4 x 4 Latin square design in which my treatment factor (Correction) is combined with the column blocking factor (Narrative). There are four combinations of Correction and Narrative which correspond to the row blocking factor.

    My original plan was to analyse the data using mixed effects ANOVA, but I spoke to a statistician who told me that this wouldn’t be a sensible solution given that I do not have a crossed or nested design. I am uncertain about few things:

    1) Should I use mixed effects ANOVA in the first place,
    2) If mixed effects is appropriate, what’s the best way to specify the random effects term, and
    3) What would be the most sensible way to construct the model,

    Some additional complications are that I did not expect there to be an interaction between Correction and Narrative but there does seem to be one. Further, my N is quite small (N = 37) and – due to some issues with the randomisation algorithm – there are more participants for some combinations than for others.

    I guess one option would be to remove the intercorrelations among treatments by subtracting the participant mean from each of their scores and then running this as a normal 4 x 4 ANOVA.

    Any thoughts or advice would be greatly appreciated.

  • #90

    henrik
    Keymaster

    I am not sure I fully understand the nature of your latin square design without actually seeing the design figure/plot (i.e., I do not understand the meaning of treatment factor, column blocking factor, and row blocking factor). It may help to provide it, especially if there are replications (i.e., given that 4 x 4 = 16 and you have 37 participants, that is quite likely). Example here.

    As long as a single unit of observation (i.e., probably participant) provides data for more than one condition, setting up a mixed model is probably doable. For this question it is important to know if one of the factors (Correction or Narrative) is a within-subjects (i.e., repeated-measures) factor or not.

    Even if none of your factors is a repeated-measures factor, setting up a latin square design with mixed models is possible. An example is given here.

    Note that in any case, the presence of interactions makes an analysis using a latin square approach difficult as it assumes no interaction. An interaction in the data thus invalidates the results and suggests the necessity of collecting new data allowing for an interaction, potentially with a reduced design. But I think it would be helpful to first described exactly what each participant saw to evaluate whether or not a simple mixed model analysis is appropriate and one can have a look at the interaction.

    • #93

      Hi Henrik,

      So there were two repeated measures factors – Correction and Narrative. Order (the row factor – which corresponds to the specific pairing of correction and narrative that participants saw) was a between ss factor. Although participants were randomly assigned to the different orders there were, unfortunately, a different number of participants per cell.

      Here is a screenshot of the design and how many participants there were per cell. (I hope you can see this!)

      My latin square design

      The presence of an interaction is a little troublesome I will probably need to run a new study with a different design.

      Thank you!

      • This reply was modified 4 months, 1 week ago by  henrik. Reason: corrected image link
      • This reply was modified 4 months, 1 week ago by  henrik.
      • #98

        henrik
        Keymaster

        Where is Correction in your design? Are these the levels within Order? Does this entail that you had only four specific pairings of Correction and Narrative?

        To know how to set up a mixed model it is important to know if you have replicates for each cell of the design and participant. In other words, did each participant only provide one response per Narrative level or several. For mixed models one usually wants/needs several responses per unit of observation and cell of the design (i.e., replicates) to estimate the full random effects structure.

        I think to get more help the best would be to either post a mockup of your data (as in this example) or your data.

  • #99

    Correction is in each cell. For example, in participant #1 would see Order 1 and see Fire (no correction), Van (correction), Head injury (explanatory correction #1), and missing person (exp correction #2). For Order 2, however, the participant would see a different pairing of Narrative and Correction. So all in all there 16 different combinations of Correction and Narrative but each participant saw four of these. I hope this makes sense. Here is a screenshot of the first few rows of my data frame.

    So each participant provided 1 response per Narrative level and one response per Correction level. There is more than one response per cell but there are not an equal number of responses in each cell (e.g., cells 1-4 have 8 whereas cells 5-8 have 13).

    Data

    Here is a mock-up of the what the first 4 rows of the data frame would look like or what participant #1 would see.

    mat <- data.frame(participant=factor(rep(1)),
    inference=c(4, 0, 0, 1),
    ord=factor(rep(1)),
    narrative=factor(c(“F”, “V”, “H”, “M”)),
    correction=factor(c(“NC”, “CO”, “CE”, “CL”)))

    • This reply was modified 4 months, 1 week ago by  henrik. Reason: corrected image link
  • #103

    henrik
    Keymaster

    I had to remove my previous response because most things I said there were kind of incorrect. But I think I this response should be correct now.

    In your design you have realized all 16 combinations of the factors Correction and Narrative. However, not within-subjects, but only across subjects. This means you can estimate fixed effects for their interaction, but no random slopes for it and no within-subjects ANOVA. In fact, the absence of any replicates for any factor on the individual level, precludes basically any calculation of random-slopes. However, following the logic that in your situation the random variability is confounded with the residual variance, the following mixed model should in principle be appropriate (along the lines discussed here):
    mixed(dv ~ Correction*Narrative + Order + (1|participants), data)

    To make sure that the model is in fact not grossly anti-conservative one could run a simulation study from generated data without effect to see if the alpha-levels are within nominal ranges, but this might be outside what could be expected from a masters thesis (but would be pretty cool, see the previous link for some hints how to do it).

    If Order had more than 4 levels, one could probably also treat it like a random-effects grouping factor (e.g., see here). But with 4 levels that seems not really advisable. You can only hope it does not really effect the results.

    I hope that helps. I would probably go with it.

  • #104

    Hi,

    Thank you, this is very helpful. I have tried this approach and it does work. There was one thing I was unsure about. When I use the anova() function to obtain F ratios and p values for the fixed effects and interaction, there are values for everything except Order. Is there a reason that there are no values for this fixed effect?

    This study will actually be included in my PhD thesis, so the simulation study you suggest would actually be very useful. When you say to run a simulation study from generated data without effect, which effect do you mean exactly?

    Many thanks for all your help.

  • #105

    henrik
    Keymaster

    I am not sure why you need the anova() function, as the return output of mixed() should already give you the Fand p values (although you can use anova() to get a specific ANOVA table). Are you sure that all the variables are of the correct type (i.e., factors are in fact factors, you can check this with str())? Are there any warnings?
    Perhaps it would be best to post the output from mixed().

    Regarding the simulation, I think about one where all your fixed effects have no effect. You simulate data from the Null model. So you generate data that is structurally identical to your data in all regards (same factor levels and number of participants), but has no effect besides the random effects, and then fit it with the model you use here. You do this a lot of times (e.g., 1000) and see how often the p-value is significant. Should be 5% of the times. Similar as done here: singmann.org/mixed-models-for-anova-designs-with-one-observation-per-unit-of-observation-and-cell-of-the-design/

    • #106

      Yes, that works and I can obtain F and p values. Here is the output though.

      mixed model output

      As you can see, the output for

        Order

      is NA. I have looked at the structure of the data and it is a factor. There are no warnings aside from the ones created by the output.

      >

      • #107

        henrik
        Keymaster

        I cannot see the figure, you need to add it in another way.

        Are you sure you are using the latest version of afex and R? Does it also occur with other methods (e.g., method = "S")? If yes to both, this is an error I haven’t seen so far and I will not be able to help you without reproducing it (i.e., I will need the data).

        • #108

          I believe I’m running the latest version of afex (0.18.0) and am running the latest version of R. It also occurs with methods.

          This is the output that I get:

          Mixed Model Anova Table (Type 3 tests, KR-method)

          Model: Inference ~ Correction * Narrative + Order + (1 | participant)
          Data: ldf3.2
          Effect df F p.value
          1 Correction 3, 99 34.38 *** <.0001
          2 Narrative 3, 99 7.33 *** .0002
          3 Order 3, NA NA <NA>
          4 Correction:Narrative 6, 99 6.48 *** <.0001

          Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1

          Would it be possible to send you the data? If so, what would be the best method of doing this?

          • #109

            I guess this could be due to the somewhat unorthodox design…

          • #110

            The output seems fine if specify all the factors as main effects rather than interactions.

You must be logged in to reply to this topic.