Reliability
This article is just a quick demonstration of the power of Octopress with Pandoc, and uses all of the common elements of a data analysis write up. In fact, I stole most of the content from a paper I’m working on.
Everything written in this article could be witten directly in an .Rmd file, interactively written and then compiled in R Studio and published (nearly) straight to the web. This post includes code snippets, citations, tables and math. And it looks beautiful!
p.s. This is just a demo and isn’t intended to actually make sense ;-)
The term reliability refers to the ability of a test to consistently assess or measure the same underlying ability or concept, insofar as in a fully reliable test the only source of measurement error is random error. Cronbach’s coefficient alpha (Cronbach 1951) is the most popular metric for evaluating reliability, and is considered a measurement of internal consistency, or the level of inter-item correlation within a test administered to a single group.
In this study I compared the reliability of three final exam formats using the CTT package in R:
require('CTT')
items <- complete.all[, c(67, 39, 3:26)]
items <- scaleMC(items)
# Run item analysis
ital <- list()
ital.mpgpa <- list()
for(level in levels(items$Format)){
# Calculate format-level test reliability
# (ex: across all 'MC+PC' students)
ital[[level]] <- reliability(items[items$Format == level, -1:-2])
}
# Extract alpha values from item analysis
ital.alpha <- c()
for(name in names(ital)){
ital.alpha <- c(ital.alpha, ital[[name]]$alpha)
}
# Print a nice table
t.alpha <- data.frame(
c("Partial Credit", "", "Dichotomous",""),
names(ital),
c('Spring 2013', 'Spring 2012', 'Spring 2012', 'Summer 2013'),
ital.alpha
)
colnames(t.alpha) <- c("Scoring",'Format', 'Semester', "Cronbach's Alpha")
kable(t.alpha)Results
The coefficient alpha estimation of reliability for each of the examination formats and scoring methods is shown in Table 8. For both the CR and MC+PC examination formats, alpha is near 0.74, while the dichotomously scored MC and MC+PC examination formats demonstrated reliability near 0.68.
Cronbach’s Alpha
As described above, Cronbach’s alpha, \(\alpha\), is really just:
\[\alpha = \frac{K}{K-1} \left( 1 - \frac{\sum^{K}_{i=1} \sigma^2_{Y_i}}{\sigma^2_X} \right)\]
where \(\sigma^2_X\) is the variance of the observed total test scores and \(\sigma^2_{Y_i}\) is the variance of component \(i\) for the current sample of persons1.