<h1>Informative vs uninformative prior distributions with characteristic curve linking methods</h1> <h2>Brandon LeBeau, Keyu Chen, Wei Cheng Liu, and Aaron McVay</h2> <h3>University of Iowa</h3> # Linking overview - With item response theory (IRT), the ability scale is arbitrarily defined (commonly mean of 0 and sd of 1). - Linking is useful to help place individual ability and IRT item parameters on the same scale. + Particularly when two forms are administered to non-equivalent groups. - Four linking methods are common: + Mean/Mean + Mean/Sigma + Haebara + Stocking Lord # Linking Transformation <img src="/figs/link.PNG" alt="" height = "400" width="800"/> # Linking Designs - Random Groups - Single group with counterbalancing - Common-item nonequivalent group design - More details in Kolen & Brennan (2014). # Common-item NEG Design <img src="/figs/commonitem.png" alt="" height = "500" width="1200"/> # Prior Weights - The proficiency points and weights can be specified to reflect the ability distribution of the original scale. - In addition, proficiency points and weights can be specified to reflect the ability distribution of the new scale. - More details are provided in Kim & Lee (2006). **Research Questions:** 1. To what extent does the prior distribution have an impact on the estimation of the transformation constants? 2. To what extent does the relationship from #1 generalize across the simulation conditions? # Simulation Design <img src="/figs/simulation_conditions.png" alt="" height = "500" width="1200"/> # Simulation Design 2 - The A and B transformation constants were also simulated as a part of the design. + This was done in an attempt to increase generalizeability of study results. - Both were simulated from a random uniform distribution. + A ranged from 0.5 to 1.5 rounded to nearest .05 (21 possibilities) + B ranged from -2 to 2 rounded to nearest 0.10 (41 possibilities) - 1000 replications # Simulation Procedures - A population of 55 items were simulated as Form X from a normal ability distribution. - Form Y consisted of common items from Form X (transformed based on A and B parameters). + Additional items were simulated to fill out Form Y. - Form Y was calibrated with Bilog-MG using a 3PL IRT model. - Transformation constants were computed from calibrated Form Y item parameters and population Form X item parameters. + An R package, plink, was used. # Study Outcomes - Bias in the transformation constants (A and B) were explored descriptively and inferentially: <img src="/figs/bias.PNG" alt="" height = "200" width="800"/> # Simulation recovery <img src="/figs/heatmap_b.png" alt="" height = "500" width="1200"/> # Results <table style="font-size: 26pt;"> <thead> <tr class="header"> <th style="text-align: left;">Variable</th> <th style="text-align: right;">Eta A</th> <th style="text-align: right;">Eta B</th> </tr> </thead> <tbody> <tr class="odd"> <td style="text-align: left;">Ability Dist</td> <td style="text-align: right;">0.699</td> <td style="text-align: right;">0.013</td> </tr> <tr class="even"> <td style="text-align: left;">Prior Dist</td> <td style="text-align: right;">0.012</td> <td style="text-align: right;">0.009</td> </tr> <tr class="odd"> <td style="text-align: left;">A Pop</td> <td style="text-align: right;">0.149</td> <td style="text-align: right;">NA</td> </tr> <tr class="even"> <td style="text-align: left;">B Pop</td> <td style="text-align: right;">0.012</td> <td style="text-align: right;">0.522</td> </tr> <tr class="odd"> <td style="text-align: left;">Ability Dist:Prior Dist</td> <td style="text-align: right;">0.004</td> <td style="text-align: right;">0.003</td> </tr> <tr class="even"> <td style="text-align: left;">Ability Dist:A Pop</td> <td style="text-align: right;">0.045</td> <td style="text-align: right;">NA</td> </tr> <tr class="odd"> <td style="text-align: left;">Ability Dist:B Pop</td> <td style="text-align: right;">0.008</td> <td style="text-align: right;">0.387</td> </tr> <tr class="even"> <td style="text-align: left;">Prior Dist:A Pop</td> <td style="text-align: right;">0.004</td> <td style="text-align: right;">0.002</td> </tr> <tr class="odd"> <td style="text-align: left;">Ability Dist:Prior Dist:B Pop</td> <td style="text-align: right;">0.002</td> <td style="text-align: right;">0.002</td> </tr> </tbody> </table> # Results A Constant <img src="/figs/a_ab_bias_large.png" alt="" height = "500" width="1200"/> # Results B Constant <img src="/figs/b_ab_bias_large.png" alt="" height = "500" width="1200"/> # Conclusions - Prior distribution used for linking the two forms does not have a large impact on the estimation of the A and B constants. - Even correctly specifying the shape of the ability distribution through the weights does not help with non-normal ability distributions. - The ability distribution shape has the most impact on accurate estimation of the A and B constants. + Normalizing transformations of the ability distribution may be helpful to limit bias when estimating these linking constants. # Questions? - Twitter: @blebeau11 - Website: <http://brandonlebeau.org> <br/> <http://www2.education.uiowa.edu/directories/person?id=bleb> - Slides: <http://brandonlebeau.org/2016/04/08/aera2016.html>