Informative vs uninformative prior distributions with characteristic curve linking methods

Brandon LeBeau

2016-04-08

Linking overview

  • With item response theory (IRT), the ability scale is arbitrarily defined (commonly mean of 0 and sd of 1).
  • Linking is useful to help place individual ability and IRT item parameters on the same scale.
    • Particularly when two forms are administered to non-equivalent groups.
  • Four linking methods are common:
    • Mean/Mean
    • Mean/Sigma
    • Haebara
    • Stocking Lord

Linking Transformation

Linking Designs

  • Random Groups
  • Single group with counterbalancing
  • Common-item nonequivalent group design
  • More details in Kolen & Brennan (2014).

Common-item NEG Design

Prior Weights

  • The proficiency points and weights can be specified to reflect the ability distribution of the original scale.
  • In addition, proficiency points and weights can be specified to reflect the ability distribution of the new scale.
  • More details are provided in Kim & Lee (2006).

Research Questions:

  1. To what extent does the prior distribution have an impact on the estimation of the transformation constants?
  2. To what extent does the relationship from #1 generalize across the simulation conditions?

Simulation Design

Simulation Design 2

  • The A and B transformation constants were also simulated as a part of the design.
    • This was done in an attempt to increase generalizeability of study results.
  • Both were simulated from a random uniform distribution.
    • A ranged from 0.5 to 1.5 rounded to nearest .05 (21 possibilities)
    • B ranged from -2 to 2 rounded to nearest 0.10 (41 possibilities)
  • 1000 replications

Simulation Procedures

  • A population of 55 items were simulated as Form X from a normal ability distribution.
  • Form Y consisted of common items from Form X (transformed based on A and B parameters).
    • Additional items were simulated to fill out Form Y.
  • Form Y was calibrated with Bilog-MG using a 3PL IRT model.
  • Transformation constants were computed from calibrated Form Y item parameters and population Form X item parameters.
    • An R package, plink, was used.

Study Outcomes

  • Bias in the transformation constants (A and B) were explored descriptively and inferentially:

Simulation recovery

Results

Variable Eta A Eta B
Ability Dist 0.699 0.013
Prior Dist 0.012 0.009
A Pop 0.149 NA
B Pop 0.012 0.522
Ability Dist:Prior Dist 0.004 0.003
Ability Dist:A Pop 0.045 NA
Ability Dist:B Pop 0.008 0.387
Prior Dist:A Pop 0.004 0.002
Ability Dist:Prior Dist:B Pop 0.002 0.002

Results A Constant

Results B Constant

Conclusions

  • Prior distribution used for linking the two forms does not have a large impact on the estimation of the A and B constants.
  • Even correctly specifying the shape of the ability distribution through the weights does not help with non-normal ability distributions.
  • The ability distribution shape has the most impact on accurate estimation of the A and B constants.
    • Normalizing transformations of the ability distribution may be helpful to limit bias when estimating these linking constants.

Questions?