An Example Course | Dr. Gregor Kappler

Applications

Mon, 26 Sep 2016 00:00:00 +0000

CTT: continuous \(Z\) and \(X\)

Equivalence of Classical Test Theory

With item randomization and definitions of \(\xi\) for the discrete cases, now the application of these definitions to CTT is investigated.

We prove that CTT with \(m\) τ-equivalent (parallel) items is equivalent with the proposed model after a linear transformation.

This investigation helps understanding the similarity with and difference to CTT.

Classical Test Theory with τ-equivalent items

A person \(u \in \Omega_U\) is drawn.
An multivariate observation \(\ym \in \mathbb{R}\) is made of this person \(u\) (manifest random variables \(Y_1,\ldots,Y_m : \Omega \rightarrow \mathbb \mathbb{R}\))
A random variable \(X : \Omega \rightarrow \mathbb \mathbb{R}\) is observed for validation.

Reflective measurement is a special case when \(X\) is also a τ-equivalent item.

Assumptions \(\Ym : \Omega \rightarrow \mathbb{R}\) are τ-equivalent (parallel)

\(\tau\) -equivalence \(\tau := E(Y_1 | U) = \ldots = E(Y_m | U)\)
Error variables \(\epsilon_i=Y_i-\tau\) are uncorrelated with measurement error \(\sigma_{\epsilon}^2=Var(\epsilon_i)\).
Assume also \(X\) is a τ-equivalent item: \(X=\tau + \epsilon_0\).

Definiton of \(\xi\) and Estimator \(\bar \xi\)

With the newly proposed randomization of manifest variables, let \(Z\) be the randomized measurement as defined in \ref{defZ}.¹

Assume the regression \(\xi\) is a linear function²:

\begin{align} \label{cttxialphabeta} \xi = & E(X | Z) = \alpha + \beta \cdot Z \end{align}

We define the \Ym-conditional expectation of \(\xi\)

\begin{align} \bar{\xi} := & E(\xi | \Ym) \\\ \label{cttxibaralphabeta} = & \alpha + \beta \cdot \frac{1}{m}\sum_{i=1}^m Y_{i} \end{align}

The proof of (\ref{cttxibaralphabeta}) is obvious with (\ref{cttxialphabeta}) and marginalisation over \(K\).

Theorem: Equivalence \(\bar{\xi}=\alpha + \beta \bar Y\), with \(\beta=\rho_1\) reliability of one item

The latent variable \(E(\xi|U)\) is related to CTT with \(\bar Y\) (Equation \ref{defYbar}):

estimates of person scores \(\bar \xi\) are linearly transformed \(\bar Y\) with a slope identical to reliability \(\beta=\rho_1\):

\begin{align} \bar{\xi} = & \alpha + \beta \bar{Y} \\\ \beta = & \frac{Cov(X,Z)}{Var(Z)}\\\ = & \rho_{1} \\\ \alpha = & E(X) - \beta E(Z) \\\ = & \mu \cdot (1-\beta) \end{align}
identical reliability \(\rho_{\bar{Y}}=\rho_{\bar{\xi}}\)

Randomized Variable \(Z\)

\begin{align} \label{cttEZ} E(Z) = & \mu \\\ \label{cttVarZ} Var(Z) = & Var(\tau) + \sigma_{\epsilon}^2 \\\ \label{cttCovXZ} Cov(X,Z) = & Var(\tau) \end{align}

Proofs obvious.

Regression random variable \(\xi\)

\begin{align} E(\xi) = & \alpha + \beta \cdot \mu \end{align}

Proof obvious.

\begin{align} Var(\xi) = & \beta^2 Var(Z) \end{align}

Proof:

\begin{align} Var(\xi) = & Var(\alpha + \beta \cdot Z) \\\ = & E\left( (\alpha + \beta \cdot Z -E(\alpha + \beta \cdot Z))^2 \right) \\\ = & \beta^2 E\left( (Z -E(Z))^2 \right) \\\ = & \beta^2 Var(Z) \end{align}

\begin{align} E(\xi | U) = & \alpha + \beta \cdot \tau \end{align}

Proof:

\begin{align} E(\xi | U) = & E \left[ E(\alpha + \beta \cdot Z | U) \right] \\\ = & \sum_{i=1}^m P(K=i | U) E(\alpha + \beta \cdot Z | K=i, U) \\\ = & \sum_{i=1}^m \frac{1}{m} \left( \alpha + \beta \cdot \underbrace{E(Y_i | U)}_{=\tau} \right) \\\ = & \alpha + \beta \cdot \tau \end{align}

With \(K\) being a uniform random variable for the randomization of items (Equation \ref{defK}).

Estimator \(\bar \xi\)

\begin{align} E(\bar{\xi}) = & E(\alpha + \beta \cdot \frac{1}{m}\sum_{i=1}^m Y_{i})\\\ = & \alpha + \beta \mu.\\\ E(\bar{\xi}|U) = & E(\alpha + \beta \cdot \frac{1}{m}\sum_{i=1}^m Y_{i} | U)\\\ = & \alpha + \beta \tau = E(\xi | U).\\\ \label{cttVarxibar} Var(\bar{\xi}) = & \beta^2 Var(\tau) + \frac{\beta^2}{m} \sigma_{\epsilon}^2\\\ \label{cttVarxibarGU} Var(\bar{\xi} | U) = & \frac{\beta^2}{m} \sigma_{\epsilon}^2 \end{align}

Proofs Estimator \(\bar \xi\)

Proof of (\ref{cttVarxibar}):

\begin{align} Var(\bar{\xi}) = & Var \left( \alpha + \beta \cdot \frac{1}{m}\sum_{i=1}^m Y_{i} \right) \\\ = & \beta^2 Var \left( \sum_{i=1}^m \frac{1}{m}(\tau + \epsilon_i) \right) \\\ = & \beta^2 Var\left(\tau + \frac{1}{m}\sum_{i=1}^m \epsilon_i \right) \\\ = & \beta^2 Var(\tau) + \frac{\beta^2}{m} \sigma_{\epsilon}^2 \end{align}

The last identitly follows from the fact that \(\tau, \epsilon_1, \ldots, \epsilon_m\) are stochastically independent and \(Var(\epsilon_i)=\sigma_{\epsilon}^2\).

Proof of (\ref{cttVarxibarGU}):

\begin{align} Var(\bar{\xi} | U) = & E \left[ \left( \bar{\xi} - E(\bar{\xi} | U) \right)^2 \right] \\\ = & E \left[ \left( \alpha + \beta \frac{1}{m} \sum_{i=1}^m Y_i - \alpha - \beta \tau \right)^2 \right] \\\ = & \beta^2 E \left[ \left( \frac{1}{m} \sum_{i=1}^m Y_i - \tau \right)^2 \right] \\\ = & \beta^2 E \left[ \left( \frac{1}{m} \sum_{i=1}^m (\tau + \epsilon_i) - \tau \right)^2 \right] \\\ = & \beta^2 E \left[ \left( \frac{1}{m} \sum_{i=1}^m \epsilon_i \right)^2 \right] \\\ = & \frac{\beta^2}{m^2} Var \left[ \sum_{i=1}^m \epsilon_i \right] = \frac{\beta^2}{m} \sigma_{\epsilon}^2 \end{align}

The last identitly follows from the fact that \(\epsilon_1, \ldots, \epsilon_m\) are stochastically independent and \(Var(\epsilon_i)=\sigma_{\epsilon}^2\).

Reliability

\(\bar{\xi}\) and \(\bar Y\) have equal reliability.

Proof:

\begin{align} \rho_{\bar{\xi}} = & 1- \frac{Var(\bar{\xi} | U)}{Var(\bar{\xi})} \\\ = & 1-\frac{\frac{\beta^2}{m} \sigma_{\epsilon}^2}{\beta^2 Var(\tau) + \frac{\beta^2}{m} \sigma_{\epsilon}^2}\\\ = & 1-\frac{\frac{1}{m} \sigma_{\epsilon}^2}{Var(\tau) + \frac{1}{m} \sigma_{\epsilon}^2}\\\ = & \frac{Var(\tau)}{Var(\tau)+\frac{i}{m}\sigma_{\epsilon}^2} \\\ = & \rho_{\bar{Y}} \end{align}

Discrete \(Z\), continuous \(X\)

Estimating latent variables \(\bar \xi\) from text

In the case of finite possible outcomes for \(Y_i\), given an observation \(\omega\), \(\bar{\xi}(\omega)\) is an unbiased estimate of \(E(\xi|U=U(\omega))\), see Equation (\ref{defxibar}).

\begin{align} \bar{\xi} = & \sum_{z \in \Omega_S} E(X | Z=z) \cdot J_z \end{align}

FRONT \(Z=z\) -conditional expectation of \(X\)

\begin{align} E(X | Z=z) & = \frac{ E \left[ X \cdot J_{Z=z} \right]}{ E(J_{Z=z}) } \end{align}

Proof by marginalisation over all \(Y_i\) on next slide!

VERBESSERN Proof: \(Z=z\) -conditional expectations \(X\)

By construction \(Z\) is \(\mathbf{Y}\) -conditionally independent from \(X\):

\begin{align} \label{zycondiidx} P(Z=z | X=x, \mathbf{Y=Y}(\omega)) &= P(Z=z | \mathbf{Y=Y}(\omega)) \\\ \label{defPz} &= \frac{1}{m} \sum_{i=1}^m I_{Y_i=z}(\omega) \\\ &= \frac{1}{m} \sum_{i=1}^m \delta(Y_i(\omega),z) \end{align}

With vector notation \(\mathbf{Y} := (\Ym)\) and \(\mathbf{Y}(\omega)=(\ym)\), and with Kronecker’s \(\delta(a,b)=1\) if \(a=b\) and \(\delta(a,b)=0\) if \(a \ne b\).

The joint probability of \(X\) and \(Z\) is by the law of total probability, over all possible events in \(\mathbf{y} \in \Omega_O=\mathbf{Y}(\Omega)\):

\begin{align} P(X=x, Z=z) & = \sum_{\mathbf{y} \in \Omega_O} P(Z=z, X=x, \mathbf{Y=y}) \\\ & = \sum_{\mathbf{y} \in \Omega_O} P(Z=z | X=x, \mathbf{Y=y}) \cdot P(X=x, \mathbf{Y=y}) \\\ & \stackrel{(\ref{zycondiidx})}{=} \sum_{\mathbf{y} \in \Omega_O} P(Z=z | \mathbf{Y=y}) \cdot P(X=x, \mathbf{Y=y}) \\\ \label{jointPXZ} & \stackrel{(\ref{defPz})}{=} \sum_{\mathbf{y} \in \Omega_O} \left( \frac{1}{m} \sum_{i=1}^m \delta(y_i,z) \right) \cdot P(X=x, \mathbf{Y=y}) \end{align}

\begin{align} E(X | Z=z) & = \frac{\int_{-\infty}^{\infty} x \cdot P(X=x,Z=z) dx}{P(Z=z)} \\\ & \stackrel{(\ref{jointPXZ})}{=} \frac{1}{P(Z=z)} \int_{-\infty}^{\infty} x \cdot \left[ \sum_{\mathbf{y} \in \Omega_O} \left( \frac{1}{m} \sum_{i=1}^m \delta(y_i,z) \right) \cdot P(X=x, \mathbf{Y=y}) \right] dx \\\ & = \frac{1}{P(Z=z)} \left[ \sum_{\mathbf{y} \in \Omega_O} \left( \frac{1}{m} \sum_{i=1}^m \delta(y_i,z) \right) \cdot \int_{-\infty}^{\infty} x \cdot P(X=x, \mathbf{Y=y}) dx \right] \\\ & = \frac{1}{P(Z=z)} \sum_{\mathbf{y} \in \Omega_O} \left( \frac{1}{m} \sum_{i=1}^m \delta(y_i,z) \right) \cdot E(X | \mathbf{Y=y}) P(\mathbf{Y=y}) \end{align}

Conditional on \(\mathbf{Y}=\mathbf{y}=(y_1, \ldots, y_m)\) the Kronecker’s \(\delta(y_i,z)\) are constant and can be pulled into the expectation:

\begin{align} E(X | Z=z) & = \frac{1}{P(Z=z)} \sum_{\mathbf{y} \in \Omega_O} \left( \frac{1}{m}\sum_{i=1}^m \delta(y_i,z) \right) \cdot E(X | \mathbf{Y=y}) \cdot P(\mathbf{Y=y}) \\\ & = \frac{1}{P(Z=z)} \sum_{\mathbf{y} \in \Omega_O} E \left[ \frac{1}{m}\sum_{i=1}^m \delta(y_i,z) \cdot X | \mathbf{Y=y} \right] \cdot P(\mathbf{Y=y}) \end{align}

\normalsize Also the substitution \(\delta(y_i,z)=I_{Y_i=z}\) is correct in the conditional expectation:

\begin{align} E(X | Z=z) & = \frac{1}{P(Z=z)} \sum_{\mathbf{y} \in \Omega_O} E \left[ \frac{1}{m}\sum_{i=1}^m I_{Y_i=z} \cdot X | \mathbf{Y=y} \right] \cdot P(\mathbf{Y=y}) \\\ & \stackrel{(\ref{defJ})}{=} \frac{1}{P(Z=z)} \sum_{\mathbf{y} \in \Omega_O} E \left[ J_{Z=z} \cdot X | \mathbf{Y=y} \right] \cdot P(\mathbf{Y=y}) \\\ & = \frac{E[ J_{Z=z} \cdot X ]}{P(Z=z)} \qed \end{align}

The last equation follows from law of total expectation.

Discrete \(Z\) and \(X\)

VERBESSERN Person-Conditional Regression of \(X\) on \(Z\):

In the case of discrete \(X\), the regressions of \(I_{=xX}\) on a randomly selected item from the observation \(Z\) are random variables, again.

\begin{align} \label{defxi} \xi_x := & E(I_{X=x} | Z) \end{align}

These regressions reflect what we can learn about the validity criterion \(X\) from a single outcome of the test – if we do not know what item the outcome was for.

The latent variable of interest is the \(U\) -conditional expectation of this regression of \(X\) on \(Z\)

\begin{align} \label{defxigu} \label{defvee} E(\xi_x | U) = & E [ E(I_{X=x} | Z) | U ]\\\ = & \sum_{z \in \Omega_S} P(X=x | Z=z) \cdot \zeta_z \end{align}

The proofs in the continuous case apply, with \(P(X=x | Z=z)=E(I_{X=x} | Z=z)\).

IRT

FRONT Simulation: model estimates display nearly identical correlations

Simulation for the Rasch model of IRT and \(X\) being another item with difficulty 0 (a reflective variable) have been done.

The logits of estimates of latent variables \(\bar\xi\) correlate as well with the true person abilities as Rasch estimates (R package eRm).

Table 1: Simulation Parameters:

N	1000
person abilities	\(\tau \sim N(0, 2)\)
item difficulties	\(\theta=(-1,-0.5,0,0.5,1)\)

Table 2: Correlations:

	tau	ymean	xibar	rasch
tau	1.0000	0.8338	0.8347	0.8355
ymean	0.8338	1.0000	0.9999	0.9995
xibar	0.8347	0.9999	1.0000	0.9998
rasch	0.8355	0.9995	0.9998	1.0000

Simulation: model estimates rescaled (more items)

Selection with independent randomization variable \(K : \Omega \rightarrow \{1, \ldots, m \}\) uniform. ↩︎
This is true if \(X\) is a τ-equivalent item, in the general case I think this assumptions is not required… ↩︎

What is measuring?

Mon, 26 Sep 2016 00:00:00 +0000

Several observations on a unit are combined to estimate an attribute as number(s) with meaning in reality.

What is measuring?

Elements of a Measurement/Assessment

Observations are assessed in order to
measure some attribute (e.g. mass, intelligence)
of a unit (a person or an object).

Typically

measurements are slightly off the “true” value of the attribute: measurement error.
Attributes change in the course of time.

Examples

Hubble Deep Field (1995)

10 day sensor-exposure to an unknown and dark part of the universe¹.
3000 new galaxies were discovered.
Some attributes of these galaxies were estimated from the observed images.

Intelligence Testing

Item structure: Rasch/CTT model assumptions
Items are thoroughly designed and refined
Controlled assessment procedure

“Hard Sciences”

Direct correspondence of assessed data and attribute.

Numerical attributes in physics, chemistry, biology like length, weight, concentration.
“Straight forward” assessment procedures exist
rather clear ontology:

length exists independent of assessment.

“Soft Sciences”

Assessed data are rather “hints” to an attribute.
Observed data is complicated (e.g. behavior, texts)
Designed tests have unclear ontology

“Intelligence is what the intelligence test measures.” – but if intelligence exists outside of the assessment, how is it related to the measurement.

Measurement in social sciences is a challenge.

Current Theories

Lord & Novick: Classical Test Theory
Rasch: Item Response Theory
Steyer: Latent state-trait Theory

Theories rely on observing items of tests with assumptions of conditional independence.

START Reflective Measurements

describe in terms of probability theory the expected values of observations for a unit,
ideally assume that several observations are due to the same one (or few) latent dimensions
provide means to test assumptions and aggregate information from observations
need external validation.

Measurement Theory: Random Experiment

Assessment can be described mathematically in terms of probability theory as a random experiment (Steyer):

A person \(u \in \Omega_U\) is drawn from a population (manifest unit random variable \(U : \Omega \rightarrow \Omega_U\)).
An observation \(y \in \Omega_o\) is made of this person \(u\) (manifest random variables \(Y : \Omega \rightarrow \Omega_S\)) ².

Examples:
1. CTT: real-valued \(\Omega_S=\mathbb{R}\)
2. IRT, Rasch: \(\Omega_S=\{0, 1\}\)
3. IRT, partial credit: \(\Omega_S=\{1,\ldots,k\}\)

Given a probability space \((\Omega, \mathcal{A}, P)\).

Measurement vs. Compression

Not every procedure to compute a number from observations for a unit is a good measurement:

	Latent Variable	Principal Component Analysis
example	IQ Test	machine learning
utility	theoretical	practical
interpretation	measurement	compression

Requirements of a Measurement Theory

A measurement theory must provide

random experiment: a formal description of the assessment process in terms of probability theory,
well-defined latent variable: in terms of a person-conditional expectation of a random variable³,
unbiased estimators for latent variables: from several assessed observations \(\Ym\) for a unit \(u\), and

Otherwise precise interpretations are not possible.

From well defined latent variables one can derive statistical means to test substantive hypotheses on latent variables which

abstract from the observations and
attenuate measurement error.

A Measurement Theory can make Assumptions

A measurement theory can assume a statistical model for several observations \(\Ym\) conditional on the latent variable.

In modern theories this is the case:

CTT: τ-equivalent, essential τ-equivalent or τ-congeneric models
IRT Rasch model: uni-dimensionality, local stochastic independence.

If a measurement theory makes assumtions, then also statistical tests for model assumptions must be provided.⁴

Text data in “soft sciences”

very complicated syntactical structure,
violates assumptions of conditional independence.

There are doubts whether text-based measurement is possible.

Psychometrics with observed text

Text data in “soft sciences” is still considered useful:

Qualitative research: hermeneutic text interpretation
Expert ratings: subsequent rating-based statistics
Statistical/computational text models.

Goal: Psychometrics with Text

We present a well-defined measurement theory of text assessments:

well-defined in terms of a) random experiment, b) latent variables and c) unbiased estimators
but is failing the subtle meanings in a text.⁵

Skript

Eigentlich gilt mein Interesse formalen Sprachen und der Literatur. Gemäß des Beschreibungsinhalts nahezu aller Literatur interessiert mich besonders der Mensch, das dahinterliegende Objekt, und daher Psychologie im weitesten Sinne.

In meinem kürzest zurückliegenden Versuch, die Analyse von Texten in der Psychologie zu etablieren, begab ich mich an einen Lehrstuhl für Psychometrie.

Als Psychometriker sind wir bemüht, Aussagen über den Menschen aufgrund von Beobachtungen machen zu können – spezifisch: mathematisch zu gewährleisten, dass die latenten Variablen, mit welchen wir Personen beschreiben, präzise mathematische Bedeutung haben, und in den Beobachtungen verankert sind. Nur dann können alle Untersuchungen auf Basis dieser Variablen eindeutig interpretiert werden. Natürlich ist das ein sehr hoher Anspruch, und

in Wahrheit werden viele wissenschaftlichen Studien innerhalb der Psychologie diesem Anspruch tatsächlich nicht gerecht.

Es gibt, wie in allen Disziplinen, pragmatische Schulen einerseits und rigorose Schulen andererseits. Die Tradition der Psychometrie baut auf Strukturgleichungsmodellen oder grafischen Modellen auf, und die Konstruktion von Messinstrumenten, die psychometrisch valide sind, gilt im allgemeinen als große Kunst, bei der viel Sorgfalt auf die Auswahl und Formulierung der Fragen in einem Fragebogen zu legen ist.

Es gibt Probleme: Beispiel Desirees Masterarbeit.

Rasch-Modelle sind vor dem Hintergrund der gewaltigen Komplexität des Menschen eine große Herausforderung.

So erscheint ein Unterfangen, Psychometrie mit freiem Text als manifester Variable zu unternehmen, vielleicht erst einmal irrsinnig.

Ich bin also an die mir als am rigorosesten bekannte Institution gegangen, in der Absicht, mich meinerseits zu bemühen, ob auf Basis von Textbeobachtungen solch klar definierte Variablen einer Person abgeleitet werden können, oder irgendeiner anderen beobachteten Einheit wie einen Artikel Review oder ähnlichem.

Was soll ich sagen, es war eine harte Schule! Es ist eine harte Schule. Es herrscht allgemein die Meinung, mit Text könne man vielleicht allgemein ein bisschen herum rechnen, aber latente Variablen zur Personenbeschreibung im rigorosen Sinne seien nicht konstruierbar.

Doch genug von mir, ich nahm viele Anläufe, und Text ist in der Tat von einer Komplexität, die ich nicht nur einmal unterschätzt habe in meinen Ansätzen und Bemühungen. Es zeigte sich, dass in der Anwendung, wie beispielsweise bei den Vorhersagen meine Notizen Überschriften, Probleme zeigten bei Texten mit sehr vielen Texten oder mit sehr wenigen Worten. Es gab heuristische Lösungen, mit fachbegrifflichen Namen wie Prior Verteilungen, die ihrerseits zwar halfen aber in halfen sie für lange Texte waren sie unbrauchbar für kurze Texte, halfen sie für kurze Texte waren sie unbrauchbar für lange Texte.

Häufig Dachte ich, Heureka, ich hab’s. Häufig dachte ich dies in den letzten 20 Jahren. Für mein Interesse an Sprachen las ich das Rätsel der Mythos von Sisyphos von Camus, und jedes Heureka ist eben ein Erreichen des Gipfels gewesen.

Ich experimentierte und promovierte und holte mir Meinungen, und es fanden sich immer wieder Fehler. Meist war es nicht angezeigt, sich im Beweisen zu verstricken, die aufgrund der absehbaren Probleme bei der Programmierung aussichtslos waren, denn beim beweisen kann man sich auch verbeißen.

Nun ist wieder eine Zeit des Heureka.

Also begrüße ich erneut die Zeit des Scheiterns und weiteren Bemühens.

Insofern und wie es wohl bei Sisyphos ist, ist klar, dass dies nicht das Ende der Reise sein kann, und dass, wiewohl ich vielleicht ein Ziel erreicht habe, das überhaupt zu verfolgen anderen vollkommen irrsinnig erschien, für mich ist es von vornherein ein Scheitern an meinem Anspruch, Sprache und Texten tatsächlich gerecht zu werden.

Doch nun zum gemachten!

Current statistical/computational text analysis

Latent Semantic Indexing (LSI)

singular value decomposition for texts,
extracted dimensions have no meaning (no latent variables)

Topic Modeling/Deep Learning

“Latent” variables explaining the data (compression)
without clear interpretation

Approaches provide no measurement theory for texts!

Reading the tea leafs criticism applies to all.
predictions are no person-conditional expectations.

No measurement theory

“Latent” variables that are not well defined

Sometimes “latent” variables are defined by models compressing data (e.g. exploratory factor analysis), not as person (\(U=u\))-conditional expectations of a random variable.

These “latent” variables do not assure well-defined interpretation: what attribute of \(u\) do they reflect (Reading the tea leafs criticism, LDA)?

Violated Model Assumtions

Assumptions of a measurement theory may be violated.

Since typically we test whether model assumption null-hypotheses must be rejected based on the observations, these tests more likely as sample size increases (and as the number of observed variables increases).

START No measurement theory

More on requirements of latent variables?

person-conditional expectation of a random variable
that random variable must first be uniquely defined in terms of observations

Models with latent variables explaining the data are most of the time not identified.

Exploratory Factor analysis: rotations

VERBESSERN From Classical Test Theory to a Text Theory

Outline:

Review of Classical Test Theory introduces concepts (for simplicity with one observed variable).
Then concepts are extended to a new theory
1. Removing stochastical structure with randomisation after the observation.
2. Introducing a well-defined (formative) model for measurement.
These ideas apply for all kinds of observations, real-valued (CTT), dichotmous (IRT) as well as text.
Real-valued observations: Equivalence of the new Theory with CTT (for simplicity assumtions of τ-equivalence).
IRT and Text observations
Proofs for the diligent proofs are in the appendix.

which many argued was empty. ↩︎
Generally, a test of length \(m\) is assessed, consisting of \(m\) items \(y_1, \ldots, y_m \in \Omega_S\), see below. ↩︎
or a function of such an expectation ↩︎
These tests typically are of the form of null-hypotheses that the assumptions are not violated. Generally it is accepted to suffice if tests fail to reject these null-hypotheses. ↩︎
(sometimes you need to break something to make something.) ↩︎

Classical Test/Item Response Theory

Mon, 26 Sep 2016 00:00:00 +0000

Measurement Theory: Latent Variables in CTT

Assessment can be described mathematically in terms of probability theory as a random experiment (Steyer):

A person \(u \in \Omega_U\) is drawn from a population (manifest unit random variable \(U : \Omega \rightarrow \Omega_U\)).
An observation \(y \in \Omega_o\) is made of this person \(u\) (manifest random variables \(Y : \Omega \rightarrow \mathbb R\))

Measurement of a Latent Variable

A well-defined numerical value for the attribute \(Y\) of object \(u\) is the object conditional expectation \(E(Y | U=u)\) (Steyer).

The regression

\begin{align} \eta := & E(Y | U) \end{align}

is the latent random variable (a reflective measurement).

Measurements are unreliable

For a given person \(U=u\) the observation of \(Y\) is still a random variable and has a probability distribution \(P(Y | U=u)\).

The numerical value \(E(Y | U=u)\) is unknown and needs be estimated from assessed observations.

Reliability in Classical Test Theory

The reliability of a measurement in CTT is the proportion of variance in \(Y\) that is “explained” when knowing the unit \(\rho_Y= 1-\frac{Var(Y | U)}{Var(Y)}\), the remaining residual variance is the measurement error \(Var(Y | U)\).

Aggregating Information from multiple items

Multiple observations attenuate measurement errors.

Random Experiment

A person \(u \in \Omega_U\) is drawn.
An multivariate observation \(\ym \in \mathbb{R}\) is made of this person \(u\) (manifest random variables \(Y_1,\ldots,Y_m : \Omega \rightarrow \mathbb \mathbb{R}\))

Assumptions \(\Ym : \Omega \rightarrow \mathbb{R}\) are τ-equivalent (parallel)

\(\tau\) -equivalence¹ \(\tau := E(Y_1 | U) = \ldots = E(Y_m | U)\)
Error variables \(\epsilon_i=Y_i-\tau\) are uncorrelated with measurement error \(\sigma_{\epsilon}^2=Var(\epsilon_i)\).

Reliability of Aggregated Observations

Reliability of Aggregation

In classical test theory a test of length \(m\) displays higher reliability when considering the items’ mean:²

\begin{align} \label{defYbar} \bar{Y} := & \frac{1}{m}\sum_{i=1}^m Y_i\\\ \rho_{\bar{Y}} := & \frac{Var(\tau)}{Var(\tau)+\frac{i}{m}\sigma_{\epsilon}^2} \end{align}

Random Experiment of the the Rasch Model

For items with success/failure, also multiple observations attenuate measurement errors.

Random Experiment

A person \(u \in \Omega_U\) is drawn from a population (manifest unit random variable \(U : \Omega \rightarrow \Omega_U\)).
An multivariate observation \(\ym \in \Omega_o\) is made of this person \(u\) (manifest random variables \(Y_1,\ldots,Y_m : \Omega \rightarrow \mathbb \{0,1\}\))

Assumptions: Rasch Homogeneity

Assumptions and Latent Variables of the Rasch Model

ENTSCHEIDE Ist IRT ein benötigtes Beispiel für Aggregation?

Reference logit latent True-score variables

In the Rasch-Modell the logit of the \(U\) -conditional expectations of the first observable \(Y_1\):

\begin{align} \eta := \tau_1 := \ln \frac{E(Y_1 | U)}{1-E(Y_1 | U)} \end{align}

relaxed definitions for linearly transformed items are often used; τ-equivalence used here for conceptual clarity. ↩︎
cf. Spearman-Brown formula ↩︎

Measuring with Text

Mon, 26 Sep 2016 00:00:00 +0000

Text violates conditional independence

For CTT and Rasch models good test design can result in items \(\Ym\) not violating assumptions of s.i. conditional on latent variables.
For text there does not exist a latent variable explaining the stochastic dependence of words in a text (because text has syntactical structure, among other reasons).
We introduce item randomisation to induce stochastical independence required for aggregation.

Randomisation has a cost: estimation cannot use any attributes of items like difficulty.

Requirements of a Measurement Theory

The introduced theory meets all the requirements of a Measurement Theory.

Text observables, Random experiment

A person \(u \in \Omega_U\) is drawn, with person random variable \(U : \Omega \rightarrow \Omega_U\).
A multivariate observation \(o \in \Omega_O\) is made of this person, consisting of \(m\) words,¹

With manifest random Variables \(Y_i : \Omega \rightarrow \{t_0, t_1,…,t_l\}, i=1, \ldots, m\)

\(\Omega_O = \Omega_S^m\) (set of possible texts) with \(\Omega_S=\{t_0, t_1,…,t_l\}\) (set of \(l\) possible words in a language).

With the product set \(\Omega = \Omega_U \times \Omega_O \times \Omega_X\), the set of possible outcomes of the random experiment. An observation \(\omega \in \Omega\) is \(\omega=(u,\ym)\).

Text is … complicated

Texts have syntactical structure

\(\Ym\) are stochastically dependent.
Joint distribution of all possible texts \(\Ym\) is intractable.

→ Assumptions like τ-equivalence are highly unrealistic

Doubts whether computers can understand text

“recursive nature of language” (Chompski)

First, stochastic dependence due to syntactical structure needs destroying.

To keep notation here simple, we consider all texts to have length \(m\). In practice texts are of finite length, and one can fill with “empty word” \(t_0\). ↩︎

An Example Course | Dr. Gregor Kappler

Applications

CTT: continuous \(Z\) and \(X\)

Equivalence of Classical Test Theory

Classical Test Theory with τ-equivalent items

Assumptions \(\Ym : \Omega \rightarrow \mathbb{R}\) are τ-equivalent (parallel)

Definiton of \(\xi\) and Estimator \(\bar \xi\)

Theorem: Equivalence \(\bar{\xi}=\alpha + \beta \bar Y\), with \(\beta=\rho_1\) reliability of one item

Randomized Variable \(Z\)

Regression random variable \(\xi\)

Estimator \(\bar \xi\)

Proofs Estimator \(\bar \xi\)

Reliability

Proof:

Discrete \(Z\), continuous \(X\)

Estimating latent variables \(\bar \xi\) from text

FRONT \(Z=z\) -conditional expectation of \(X\)

VERBESSERN Proof: \(Z=z\) -conditional expectations \(X\)

Discrete \(Z\) and \(X\)

VERBESSERN Person-Conditional Regression of \(X\) on \(Z\):

IRT

FRONT Simulation: model estimates display nearly identical correlations

Simulation: model estimates rescaled (more items)

What is measuring?

What is measuring?

Elements of a Measurement/Assessment

Typically

Examples

Hubble Deep Field (1995)

Intelligence Testing

Measurement and the Social Sciences

“Hard Sciences”

“Soft Sciences”

FRONT History of Measurement in the Social Sciences

Current Theories

START Reflective Measurements

Measurement Theory: Random Experiment

Measurement vs. Compression

Requirements of a Measurement Theory

A Measurement Theory can make Assumptions

Text data in “soft sciences”

Psychometrics with observed text

Text data in “soft sciences” is still considered useful:

Goal: Psychometrics with Text

Skript

Current statistical/computational text analysis

Latent Semantic Indexing (LSI)

Topic Modeling/Deep Learning

Approaches provide no measurement theory for texts!

No measurement theory

“Latent” variables that are not well defined

Violated Model Assumtions

START No measurement theory

VERBESSERN From Classical Test Theory to a Text Theory

Classical Test/Item Response Theory

Measurement Theory: Latent Variables in CTT

Measurement of a Latent Variable

Measurements are unreliable

Reliability in Classical Test Theory

Aggregating Information from multiple items

Random Experiment

Assumptions \(\Ym : \Omega \rightarrow \mathbb{R}\) are τ-equivalent (parallel)

Reliability of Aggregated Observations

Reliability of Aggregation

Random Experiment of the the Rasch Model

Random Experiment

Assumptions: Rasch Homogeneity

Assumptions and Latent Variables of the Rasch Model

ENTSCHEIDE Ist IRT ein benötigtes Beispiel für Aggregation?

Reference logit latent True-score variables

Measuring with Text

Text violates conditional independence

Requirements of a Measurement Theory

Text observables, Random experiment

Text is … complicated