January 10, 2018 at 20:17 UTC #175
I’m working on what was intended to be a 3x2x3 Repeated-Measures ANOVA. We realized, however that mixed models would handle the complexity better, though we’re less familiar with them.
Could someone help explain one of the interactions and how to discuss it?
The response variable is “Correct”, a binary measure of success, with 0 being incorrect and 1 being correct. Each subject gave 14 responses which were intending, initially, to analyze as proportions. However, as logistic mixed-models can handle binary responses, we went with the raw binary inputs.
The treatment independent variables are Method (3 factors), Levels(3 factors), and MATB (2 factors). SubjectID indicates the individual subject labels.
The interpretation of the initial analysis is pretty straightforward to those used to ANOVA.
> require(afex) package: > full_mixed = mixed(Correct ~ Method * Levels * MATB + (1 | SubjectID),family = binomial, data= data2clean) Mixed Model Anova Table (Type 3 tests, LRT-method) Model: Correct ~ Method * Levels * MATB + (1 | Subject) Data: data2clean Df full model: 19 Effect df Chisq p.value 1 Method 2 174.54 *** <.0001 2 Levels 2 135.95 *** <.0001 3 MATB 1 0.01 .94 4 Method:Levels 4 21.05 *** .0003 5 Method:MATB 2 0.77 .68 6 Levels:MATB 2 4.67 + .10 7 Method:Levels:MATB 4 1.69 .79 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1
Where I’m uncertain is how to analyze the ‘Fixed Effects’ output:
> summary(full_mixed) Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod'] Family: binomial ( logit ) Formula: Correct ~ Method * Levels * MATB + (1 | Subject) Data: data2clean AIC BIC logLik deviance df.resid 4603.3 4725.2 -2282.6 4565.3 4511 Scaled residuals: Min 1Q Median 3Q Max -4.9059 -0.6839 0.3544 0.5920 1.8263 Random effects: Groups Name Variance Std.Dev. Subject (Intercept) 0.756 0.8695 Number of obs: 4530, groups: Subject, 18 Fixed effects: Estimate Std. Error z value Pr(>|z|) <strong>(Intercept) 1.181444 0.208896 5.656 1.55e-08 *** Method1 -0.329280 0.051814 -6.355 2.08e-10 *** Method2 -0.396058 0.051918 -7.629 2.37e-14 *** Levels1 0.608708 0.057102 10.660 < 2e-16 *** Levels2 -0.161221 0.052849 -3.051 0.00228 ** MATB1 0.002876 0.037927 0.076 0.93955 Method1:Levels1 -0.068669 0.076390 -0.899 0.36869 Method2:Levels1 0.183151 0.077251 2.371 0.01775 * Method1:Levels2 0.182732 0.072137 2.533 0.01131 * Method2:Levels2 -0.317815 0.071332 -4.455 8.37e-06 *** Method1:MATB1 0.012756 0.051708 0.247 0.80515 Method2:MATB1 0.035272 0.051646 0.683 0.49464 Levels1:MATB1 -0.121100 0.056619 -2.139 0.03245 * Levels2:MATB1 0.050910 0.052799 0.964 0.33493 Method1:Levels1:MATB1 0.042346 0.076382 0.554 0.57930 Method2:Levels1:MATB1 -0.012933 0.077183 -0.168 0.86693 Method1:Levels2:MATB1 -0.033750 0.072194 -0.467 0.64015 Method2:Levels2:MATB1 0.073972 0.071349 1.037 0.29985 </strong> --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
So, basically, if I’m looking at the output, 1.) I’d assume that the significance of Levels1:MATB1 should be ignored because the main effect for MATB is not present, and 2.) How do I explain what Method 2: Levels 2 means? (I’m pretty certain that I also can’t do a confidence interval because of the logistic transformation.)
- This topic was modified 1 week, 5 days ago by Trey.
January 11, 2018 at 10:31 UTC #177
To be able to provide a more useful answer to your question, I have to ask you why are you looking at the fixed effects in the first place? What is your goal with this?
One of the design goals of
afexis to hide the fixed effects from the user, as they are not particularly useful in cases such as yours (i.e., factorial designs with factors with more than two levels). If your goal is to test specific hypotheses on the level of the factor levels do so using the functionality provided by the
afexv. 0.19.1 onwards using
emmeans). This is described in some detail in the ANOVA vignette and applies in exactly the same way to
mixedmodels as to ANOVA models. For more details on how to set up contrasts (or post-hoc tests) you are interested in, have a lookj at the great vignettes of
emmeansalso automatically calculates CIs for you. This is certainly possible, but needs to be done on the unconstrained (i.e., logistic) scale. Only after calculating the CIs can the values be transformed on the response scale (i.e., probabilities). For example in your case, compare:
emmeans(full_mixed, c("Method", "Levels")) emmeans(full_mixed, c("Method", "Levels"), type = "response")
Note that you can simply specify any type of contrasts on these estimated marginal means. Several examples of this for logistic mixed models are provided in the supplemental material of a recent paper of mine (https://doi.org/10.1016/j.cogpsych.2017.09.002), which can be found at: https://osf.io/3uajq/
One final thought, as discussed in length by Barr, Levy, Scheepers, and Tily (2013), you most likely do not want a random-intercept only model. Anything they write in this paper applies in the same manner to logistic mixed model. The recommended maximal model has random-slopes for all within-subjects factors (e.g.,
Correct ~ Method * Levels * MATB + (Method * Levels * MATB | SubjectID)). The model you estimated likely produces anti-conservatie results.
All I have discussed here is also discussed in more length in our recent chapter: http://singmann.org/download/publications/singmann_kellen-introduction-mixed-models.pdf
Again, this deals with linear mixed models, but the parts relevant to your question apply in the same manner to generalized linear mixed models.
January 16, 2018 at 16:26 UTC #182
First of all, thank you for the detailed response. I had also seen Barr, Levy, Scheepers, and Tily, but was confused as to how it should directly apply to my analysis, so I appreciate the clarification and recommendatoin.
The study’s goal is to assess different interface types (coded as Method, with three conditions) at different cognitive workloads (MATB) and different #’s of signals (3, 4, and 5) on the number of correct responses. So, yes, I’m mainly interested in determining differences at the level of factors.
I was mainly working through Winter’s guide for Mixed Models, prior to this. I’m still digging through the references and may respond back with clarifications, but need to review the materials closely to make sure that I’m asking good questions.
- This reply was modified 6 days, 9 hours ago by Trey.
You must be logged in to reply to this topic.