<h1>Informative vs uninformative prior distributions with characteristic curve linking methods</h1>
<h2>Brandon LeBeau, Keyu Chen, Wei Cheng Liu, and Aaron McVay</h2>
<h3>University of Iowa</h3>
# Linking overview
- With item response theory (IRT), the ability scale is arbitrarily defined (commonly mean of 0 and sd of 1).
- Linking is useful to help place individual ability and IRT item parameters on the same scale.
+ Particularly when two forms are administered to non-equivalent groups.
- Four linking methods are common:
+ Mean/Mean
+ Mean/Sigma
+ Haebara
+ Stocking Lord
# Linking Transformation
<img src="/figs/link.PNG" alt="" height = "400" width="800"/>
# Linking Designs
- Random Groups
- Single group with counterbalancing
- Common-item nonequivalent group design
- More details in Kolen & Brennan (2014).
# Common-item NEG Design
<img src="/figs/commonitem.png" alt="" height = "500" width="1200"/>
# Prior Weights
- The proficiency points and weights can be specified to reflect the ability distribution of the original scale.
- In addition, proficiency points and weights can be specified to reflect the ability distribution of the new scale.
- More details are provided in Kim & Lee (2006).
**Research Questions:**
1. To what extent does the prior distribution have an impact on the estimation of the transformation constants?
2. To what extent does the relationship from #1 generalize across the simulation conditions?
# Simulation Design
<img src="/figs/simulation_conditions.png" alt="" height = "500" width="1200"/>
# Simulation Design 2
- The A and B transformation constants were also simulated as a part of the design.
+ This was done in an attempt to increase generalizeability of study results.
- Both were simulated from a random uniform distribution.
+ A ranged from 0.5 to 1.5 rounded to nearest .05 (21 possibilities)
+ B ranged from -2 to 2 rounded to nearest 0.10 (41 possibilities)
- 1000 replications
# Simulation Procedures
- A population of 55 items were simulated as Form X from a normal ability distribution.
- Form Y consisted of common items from Form X (transformed based on A and B parameters).
+ Additional items were simulated to fill out Form Y.
- Form Y was calibrated with Bilog-MG using a 3PL IRT model.
- Transformation constants were computed from calibrated Form Y item parameters and population Form X item parameters.
+ An R package, plink, was used.
# Study Outcomes
- Bias in the transformation constants (A and B) were explored descriptively and inferentially:
<img src="/figs/bias.PNG" alt="" height = "200" width="800"/>
# Simulation recovery
<img src="/figs/heatmap_b.png" alt="" height = "500" width="1200"/>
# Results
<table style="font-size: 26pt;">
<thead>
<tr class="header">
<th style="text-align: left;">Variable</th>
<th style="text-align: right;">Eta A</th>
<th style="text-align: right;">Eta B</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Ability Dist</td>
<td style="text-align: right;">0.699</td>
<td style="text-align: right;">0.013</td>
</tr>
<tr class="even">
<td style="text-align: left;">Prior Dist</td>
<td style="text-align: right;">0.012</td>
<td style="text-align: right;">0.009</td>
</tr>
<tr class="odd">
<td style="text-align: left;">A Pop</td>
<td style="text-align: right;">0.149</td>
<td style="text-align: right;">NA</td>
</tr>
<tr class="even">
<td style="text-align: left;">B Pop</td>
<td style="text-align: right;">0.012</td>
<td style="text-align: right;">0.522</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Ability Dist:Prior Dist</td>
<td style="text-align: right;">0.004</td>
<td style="text-align: right;">0.003</td>
</tr>
<tr class="even">
<td style="text-align: left;">Ability Dist:A Pop</td>
<td style="text-align: right;">0.045</td>
<td style="text-align: right;">NA</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Ability Dist:B Pop</td>
<td style="text-align: right;">0.008</td>
<td style="text-align: right;">0.387</td>
</tr>
<tr class="even">
<td style="text-align: left;">Prior Dist:A Pop</td>
<td style="text-align: right;">0.004</td>
<td style="text-align: right;">0.002</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Ability Dist:Prior Dist:B Pop</td>
<td style="text-align: right;">0.002</td>
<td style="text-align: right;">0.002</td>
</tr>
</tbody>
</table>
# Results A Constant
<img src="/figs/a_ab_bias_large.png" alt="" height = "500" width="1200"/>
# Results B Constant
<img src="/figs/b_ab_bias_large.png" alt="" height = "500" width="1200"/>
# Conclusions
- Prior distribution used for linking the two forms does not have a large impact on the estimation of the A and B constants.
- Even correctly specifying the shape of the ability distribution through the weights does not help with non-normal ability distributions.
- The ability distribution shape has the most impact on accurate estimation of the A and B constants.
+ Normalizing transformations of the ability distribution may be helpful to limit bias when estimating these linking constants.
# Questions?
- Twitter: @blebeau11
- Website: <http://brandonlebeau.org> <br/> <http://www2.education.uiowa.edu/directories/person?id=bleb>
- Slides: <http://brandonlebeau.org/2016/04/08/aera2016.html>