layout: true <div class="my-footer"><span>ouzhang.me/talk/measurement_invariance</span></div> <!-- this adds the link footer to all slides, depends on my-footer class in css--> --- name: xaringan-title class: left, middle background-image: url(img/front1.jpg) background-size: cover # Measurement Invariance ### .fancy[A Theoretical Framework of MI in CFA] .large[Ou Zhang | Psychometrics conf::2014] <!-- this ends up being the title slide since seal = FALSE--> --- class: right, middle <img class="circle" src="img/Ou_Zhang.jpg" width="150px"/> # Find me at... [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 512 512"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> @zhangou](http://twitter.com/zhangou888) [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 496 512"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> @zhangou](http://github.com/zhangou888) [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 512 512"><path d="M326.612 185.391c59.747 59.809 58.927 155.698.36 214.59-.11.12-.24.25-.36.37l-67.2 67.2c-59.27 59.27-155.699 59.262-214.96 0-59.27-59.26-59.27-155.7 0-214.96l37.106-37.106c9.84-9.84 26.786-3.3 27.294 10.606.648 17.722 3.826 35.527 9.69 52.721 1.986 5.822.567 12.262-3.783 16.612l-13.087 13.087c-28.026 28.026-28.905 73.66-1.155 101.96 28.024 28.579 74.086 28.749 102.325.51l67.2-67.19c28.191-28.191 28.073-73.757 0-101.83-3.701-3.694-7.429-6.564-10.341-8.569a16.037 16.037 0 0 1-6.947-12.606c-.396-10.567 3.348-21.456 11.698-29.806l21.054-21.055c5.521-5.521 14.182-6.199 20.584-1.731a152.482 152.482 0 0 1 20.522 17.197zM467.547 44.449c-59.261-59.262-155.69-59.27-214.96 0l-67.2 67.2c-.12.12-.25.25-.36.37-58.566 58.892-59.387 154.781.36 214.59a152.454 152.454 0 0 0 20.521 17.196c6.402 4.468 15.064 3.789 20.584-1.731l21.054-21.055c8.35-8.35 12.094-19.239 11.698-29.806a16.037 16.037 0 0 0-6.947-12.606c-2.912-2.005-6.64-4.875-10.341-8.569-28.073-28.073-28.191-73.639 0-101.83l67.2-67.19c28.239-28.239 74.3-28.069 102.325.51 27.75 28.3 26.872 73.934-1.155 101.96l-13.087 13.087c-4.35 4.35-5.769 10.79-3.783 16.612 5.864 17.194 9.042 34.999 9.69 52.721.509 13.906 17.454 20.446 27.294 10.606l37.106-37.106c59.271-59.259 59.271-155.699.001-214.959z"/></svg> ouzhang.me](https://ouzhang.me) [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 512 512"><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg> zhangou888@gmail.com](mailto:zhangou888@gmail.com) --- ## .center[Today's Topic] -- What is measurement invariance? -- What matrix we use for measurement invariance analysis? -- 2 Major types of factorial invariance -- * Measurement and Structural -- Sequence of tests for measurement invariance --- name: what is mi? class: center, inverse, middle # .fancy[What is measurement invariance?] --- name: definition title class: center,inverse background-image: url(img/front2.jpg) background-size: cover ## Definition -- ### Measurement Invariance (MI), aka **“factorial invariance”** and **“measurement equivalence”**, concerns the extent to which are the psychometric properties of the observed indicators are transportable (generalizable) across groups or over time/condition (longitudinal study). --- ## .center[What is measurement invariance (cont.)] -- - Are we measuring the same construct in the same way in different groups or over time/condition? -- - Observed scores should depend only on latent construct scores, and not on group membership or occasion -- - Observed differences between groups reflect **TRUE** differences in the amount or variability of the construct -- - Relevant concern in many applied settings - e.g., across cultures, language, age, modality --- ## .center[Most common approach for MI] -- ### Nested Model Comparisons in CFA -- Most common approach for assessing change in model fit is the <span style="color: red;"> likelihood ratio test <span/> (aka, ‐2ΔLL deviance difference test) -- - Implemented via direct difference in model `\(\chi^2\)` values most often, but this is only appropriate when using regular ML estimation -- - If adding a parameter, model fit can be **better** or **not better** (such as when adding factors, error covariances, etc) -- - If removing a parameter, model fit can be **worse** or **not worse** --- name: model comparison likelihood ## .center[Model Comparison via -2ΔLL] -- ### Comparing nested models via a **"likelihood ratio test"** `\(-2\Delta LL\)` (MLR rescaled version) -- * Step 1: Calculate `\(-2\Delta LL = -2 \times (LL_{fewer} - LL_{more})\)` -- * Step 2: Calculate `\(\Delta df = parms_{more} – parms_{fewer}\)` -- * Step 3: Compare rescaled difference to `\(\chi^2\)` with `\(df = \Delta df\)` -- - Add 1 parameter: `\(\Delta LL_{diff} > 3.84\)`, -- - Add 2 parameters: `\(\Delta LL_{diff} > 5.99\)` --- ## .center[What matrix we use for MI - scale] -- Example: -- - Suppose we measure height in inches for two groups. We can compare height difference between two groups because both group use the same height scale (inch/cm). - The hypothetical results are: <img src="img/scale_example1.PNG" width="50%" style="display: block; margin: auto;" /> -- - Now if we calculate height z-score for every individual in the group <img src="img/scale_example2.PNG" width="35%" style="display: block; margin: auto;" /> -- - Are two new results (the z-score variables) on the same scale at both occasions? -- - No, because two groups have different SDs. --- name: what matrix? class: center, inverse, middle # .fancy[What matrix we use for MI?] --- ## .center[What matrix we use for MI] -- - Is it Correlation matrix? -- - Is it Covariance matrix? -- The correct matrix we use for MI is ## .center[<span style="color: red;">**Covariance matrix**<span/>] --- ## .center[2 Major types of factorial invariance] -- There are two types of factorial invariance: -- - <span style="color: red;">**measurement invariance**<span/> -- - <span style="color: red;">**structural invariance**<span/> --- name: mi2 class: center, inverse, middle # .fancy[Measurement invariance vs. Structural invariance] --- ## .center[Measurement invariance] -- **Measurement invariance** concerns how the items measure the latent construct across groups or over time -- - Measurement model invariance: same factor loadings, same item intercepts, (possibly) same residual (co)variances -- - **Measurement model invariance is a precursor to ANY group comparison** (whether it is tested/acknowledged or not) -- Measurement invariance is often **assumed, not tested** -- - Even a t‐test assumes measurement invariance - Model changes overtime while assuming measurement invariance - People tend to accept this assumption unless they try to use a factor model…then they usually insist on testing invariance --- ## .center[Structural invariance] -- **Structural invariance** concerns how the latent factors are distributed and related in the separate populations -- - Structural model invariance: same factor variances and covariances (or same higher‐order structure) and factor means -- - Structural invariance may not hold… and that’s ok - Assuming measurement invariance holds, structural invariance represents 'real' differences in the construct across groups/time - Structural non‐invariance does not indicate a problem with the instrument– group structural differences may be of interest -- - e.g., real growth of factors over time - e.g., differentiation or de‐differentiation of latent traits --- name: MI procedure class: center, inverse, middle # .fancy[What is the MI procedure?] --- ## .center[Sequence of Tests for MI] -- ### Step 0:Omnibus test of equality of the overall indicator covariance matrix across groups * Step 0.1: CFA goodness-of-fit test for each group * Step 0.2: Omnibus test of equality of the overall indicator covariance matrix across groups -- ### Measurement invariance: -- * Step 1/Level 1: Configural invariance (Equal factor structure ) * Step 2/Level 2: Metric / Weak factorial invariance (Equal factor loadings) * Step 3/Level 3: Scalar / strong factorial invariance (Equal indicator intercepts) * Step 4/Level 4: Strict factorial invariance (Equal indicator residuals) --- ## .center[Sequence of Tests for struactual invariance] -- ### Step 0:Omnibus test of equality of the overall indicator covariance matrix across groups * Step 0.1: CFA goodness-of-fit test for each group * Step 0.2: Omnibus test of equality of the overall indicator covariance matrix across groups -- ### Structural Invariance: -- * Step 1: Factor variance invariance * Step 2: Factor covariance invariance * Step 3: Factor means invariance #### .center[ <span style="color: red;">It is OK if structural invariance doesn’t hold.<span/> ] --- ## .center[Sequence of Tests for MI (cont.)] -- <img src="img/cfa_model_example.PNG" width="80%" style="display: block; margin: auto;" /> --- ## .center[Sequence of Tests for MI (cont.)] -- Step 0:Omnibus test of equality of the overall indicator covariance matrix across groups -- Step 0.1: CFA goodness-of-fit test for each group - Does the model fit within each groups? - If not, stop. The MI cannot go further. - If yes, go to step 0.2. -- Step 0.2: Omnibus test of equality of the overall indicator covariance matrix across groups - Do the matrices differ between groups, on the whole? - If there is **no difference**. The invariance is confirmed. Congratulations! -- The omnibus test compares the **full saturated covariance** matrix for both groups - This test is likely to be rejected due to the strictness of its constraints – but if not, the process stops! -- Many people disagree with the necessity or usefulness of this test to begin testing invariance. --- name: Step1 class: center, middle, inverse background-image: url(img/front3.jpg) background-size: cover # Configural invariance --- ## .center[Step 1: configural invariance] -- Appropriate scales for the observed variables and the factors: -- - How the scale should be set up for the factors so that a factor will be the same on the two groups/occasions? -- - Two ways to set up scale: -- - One factor loading ( `\(\lambda_{x_{1}} = 1\)` ) from an observed variable for each factor equal to one ✔️ -- - Factor variance equal to one ( `\(var_{x_{1}} = 1\)` ) - This is not the solution, because we need 2 matrices from two groups have the same scales and if we set up factor variance equal to one, these two matrices are not in the same scale and cannot be compared across groups. --- ## .center[configural invariance (cont.)] -- Are groups having the same factor structure, broadly construed? -- - Same number of factors, same pattern of zero and non-zero loadings -- - same conceptual definition of constructs being measured -- - Estimate a combined model in which all model parameters are allowed to differ across groups -- - This will be the baseline model for further comparisons - Model `\(\chi^2\)` and `\(df\)` will be additive across groups --- ## .center[configural invariance (cont.)] -- <img src="img/configural_example.PNG" width="80%" style="display: block; margin: auto;" /> --- name: Step2 class: center, middle, inverse background-image: url(img/front4.jpg) background-size: cover # Metric/Weak factorial invariance --- name: Metric/Weak Factorial Invariance ## .center[Step 2: metric/weak factorial invariance] -- - Are groups having the **same factor loadings**? - Each **congeneric item** is still allowed to have a different loading - Loadings for same item are constrained to equality across groups -- - **Marker items** (that are fixed=1 for identification) are assumed invariant – because they are already fixed, they cannot be tested - Proposed Solution: For this reason, estimate all factor loadings, but **fix the factor variance(s) to 1 in the reference group only** (still free them in the alternative group) - This allows us to evaluate ALL loadings and still identify the model -- - Compare fit of metric invariance to configural invariance model: - Does the model fit not get worse **( `\(‐2\Delta LL\)` diff not significant)**? - The parameters are taking away, so the model fit can only get worse… - Don’t forget to **fix variance = 1** in reference group only (free in other group)! Otherwise you are imposing a structural constraint too by accident! --- name: Metric/weak factorial invariance2 ## .center[Metric/weak factorial invariance (cont.)] -- <img src="img/metric_example.PNG" width="80%" style="display: block; margin: auto;" /> --- name: Step3 class: center, middle, inverse background-image: url(img/front5.jpg) background-size: cover # Scalar/Strong factorial invariance --- name: Scalar/Strong factorial invariance ## .center[Step 3: scalar/strong factorial invariance] -- - Are groups having the same item intercepts? - Each congeneric item is allowed to have a different intercept - Intercepts for same item are constrained to equality across groups -- - Estimate all intercepts, but **constrain the factor mean(s) to 0 in the reference group** so we can evaluate all intercepts -- - Only test those intercepts for which metric invariance holds -- - Compare fit of scalar invariance to metric invariance model: - Does the model fit not get worse **( `\(‐2\Delta LL\)` diff not significant)**? --- name: Scalar/Strong Factorial Invariance2 ## .center[Scalar/strong factorial invariance (cont.)] -- <img src="img/scalar_example.PNG" width="80%" style="display: block; margin: auto;" /> --- name: Step4 class: center, middle, inverse background-image: url(img/front6.jpg) background-size: cover # Resitual/strict factorial invariance --- name: Residual/Strict Factorial Invariance ## .center[Step 4: residual/strict factorial invariance] -- - Are groups having the same item residual variances? - Each congeneric item is still allowed to have a different residual variance - Residual variances for the same item are constrained to equality across groups - Testing residual variances is the last step in assessing measurement invariance -- - Only test those residual variances for which metric and scalar invariance already hold -- - Compare fit of residual invariance to scalar invariance model: - Does the model fit not get worse **( `\(‐2\Delta LL\)` diff not significant)**? --- name: Residual/Strict Factorial Invariance2 ## .center[Residual/strict factorial invariance (cont.)] -- <img src="img/residual_example.PNG" width="80%" style="display: block; margin: auto;" /> --- name: Structural Invariance ## .center[Structural invariance] -- - Next, Structural Invariance will be tested if needed. -- - Are the factor variances the same across groups? - Is the factor covariance the same across groups? - Are the factor means the same across groups? -- - It is OK if structural invariance doesn’t hold! -- .center[Not covered in this presentation] -- --- name: Conclusion ## .center[Conclusion] -- - The process of testing factorial invariance has two distinct parts: - Measurement invariance: Is your construct being measured in the same way? Let’s hope so! -- - Structural invariance: Do your groups differ in their distribution and/or means of the construct? - Structural differences are real and interpretable differences given measurement invariance of the constructs -- - Measurement invariance is always assumed in any statistical analysis… But can be tested explicitly in a latent trait modeling framework --- class: center, middle # Thanks!