Alterlab Qualitative Analysis
Published by alterlab-ieu in alterlab-academic-skills
What this skill does
Analyzes qualitative data as a dispatched pipeline module — codebook development, thematic / framework / content analysis, and inter-coder reliability computed correctly (Krippendorff's alpha as primary via the krippendorff package or a bundled stdlib nominal calculator with bootstrap CIs; Cohen's / Fleiss' kappa via statsmodels) with 95% CIs and thresholds (alpha >= .80 reliable, .667-.80 tentative). It BRANCHES by design: coefficient-based ICR for codebook / content-analytic coding, versus con
Add Alterlab Qualitative Analysis to your agent
Review the source and files first. When you are ready, copy the prompt instruction or use the CLI command supported by your environment.
Install with a prompt
Paste this into a compatible coding agent:
add this skill "alterlab-qualitative-analysis" from https://github.com/alterlab-ieu/alterlab-academic-skillsInstall with the CLI
Run this command in a controlled environment after reviewing the repository:
npx skills add https://github.com/alterlab-ieu/alterlab-academic-skills --skill alterlab-qualitative-analysisSkill instructions
Qualitative Analysis — Reliability When It Fits, Consensus When It Doesn't
Skill type: ANALYSIS MODULE. Codes qualitative data and reports agreement correctly. The central discipline is a branch: a reliability coefficient is right for codebook/content-analytic work, but for reflexive thematic analysis the criterion is consensus and reflexivity, not a statistic. The skill does not oversell the number.
Core Mission
KRIPPENDORFF'S ALPHA IS THE PRIMARY COEFFICIENT — BUT A COEFFICIENT IS THE WRONG TOOL FOR REFLEXIVE TA.
BRANCH BY DESIGN.
When to Use This Skill
- "Two RAs coded my interviews with a codebook — compute intercoder reliability."
- "Build/refine a codebook and report agreement."
- "I used an LLM to code open-ended responses — does it agree with my human coding?"
- "Which reliability statistics do I report for a content analysis?"
Does NOT Trigger
| The request is really about… | Route to | Why not this skill |
|---|---|---|
| Topic modeling / embeddings / supervised text classification | alterlab-text-as-data | Computational text, not human-coding reliability. |
| Whether the study is trustworthy (positionality, Lincoln & Guba) | alterlab-ssci-reflexivity-gate | Trustworthiness gate, not the coding itself. |
| Choosing the qualitative method (grounded theory, phenomenology) | alterlab-qualitative-methods | Methodology selection, upstream. |
| Whether a scale is reliable/valid (quantitative) | alterlab-ssci-measurement-gate | Psychometrics, different problem. |
Hand-off rule. When the request matches a row above, open your answer by naming that skill by its exact id (for example alterlab-text-as-data) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work.
Running the code. In Claude Code, run analysis code in a throwaway environment instead of installing packages: python scripts/icr.py coding.csv (stdlib, nominal codes) or uv run --python ">=3.11" --with "krippendorff>=0.8" --with pandas python icr.py. This needs no install approval and leaves the user's environment untouched. If uv or the network is unavailable, say so and hand over the script; never report results you did not compute.
The branch (this is the core discipline)
| Design | Reliability criterion |
|---|---|
| Codebook / content-analytic coding (fixed categories, replicable) | compute an ICR coefficient (Krippendorff's α) with a CI and threshold |
| Reflexive thematic analysis (interpretive, researcher-as-instrument) | consensus + reflexivity — collaborative discussion to understand differences; a coefficient misrepresents the epistemology and is NOT required. Route trustworthiness to alterlab-ssci-reflexivity-gate. |
Do not force α onto reflexive TA, and do not skip α on content-analytic coding.
Correct coefficient choice (verified)
- Krippendorff's α — primary. Handles ≥2 coders, any measurement level (nominal/ordinal/
interval/ratio), and missing data. Python
krippendorff.alpha(reliability_data=<coders × units>, level_of_measurement="nominal")returns a point estimate only — pair it with a bootstrap CI (the bundledscripts/icr.pydoes nominal α + a bootstrap CI + Cohen's κ in pure stdlib; RirrCAC::krippen.alpha.raworicr::krippalpha(..., bootstrap=TRUE)give CIs for all levels). - Cohen's κ (2 coders) / Fleiss' κ (>2, fixed number):
statsmodels.stats.inter_ratercohens_kappa,fleiss_kappa(build the count table withaggregate_raters); sklearncohen_kappa_scoreas a cross-check. - NOT valid ICR measures: chi-square, Cronbach's alpha, and Pearson's r — they measure covariation/internal consistency, not chance-corrected agreement. Refuse these.
Thresholds (Krippendorff): α ≥ .80 reliable; .667 ≤ α < .80 tentative conclusions only; α < .667 unreliable. Report α with a 95% CI, never the bare point estimate.
LLM-assisted coding (supported, never free)
Treat an LLM as another coder: human↔LLM double-coding, compute Krippendorff's α between each LLM and a human gold standard, and use the CI to judge whether the LLM is statistically indistinguishable from human coders. Disclose the model, the prompt, and the agreement — never present LLM coding as uncontrolled or cost-free.
Reporting checklist (put in every ICR report)
BRANCH: content-analytic (coefficient) | reflexive TA (consensus+reflexivity)
ICR MEASURE: Krippendorff alpha (level) | Cohen/Fleiss kappa — and why
CODERS: number of coders; human / LLM
DOUBLE-CODED: % of the data double-coded
ALPHA: point estimate + 95% CI + threshold verdict (.80 / .667)
RESOLUTION: how disagreements were resolved (adjudication / consensus)
References
references/icr_and_reporting.md— coefficient math, verified APIs, LLM-benchmarking, the reporting standard.references/reflexive_vs_codebook.md— when a statistic applies vs. when it does not.scripts/icr.py— stdlib nominal Krippendorff's α + bootstrap CI + Cohen's κ (no numpy / no deps). CSV: a header of ≥ 2 coder names, one row per unit, exactly one cell per coder (blank = missing). α and κ are reported as not estimable when every pairable code is one category.
Part of the AlterLab Academic Skills suite.
Files included
- evals/evals.json
- evals/fixtures/icr_coding.csv
- references/icr_and_reporting.md
- references/reflexive_vs_codebook.md
- scripts/icr.py
- SKILL.md
More skills
Alterlab Survey Design
alterlab-ieu/alterlab-academic-skills
No summary provided.
Audit unavailable97 installsAlterlab Qualitative Methods
alterlab-ieu/alterlab-academic-skills
No summary provided.
Audit unavailable82 installsAlterlab Mixed Methods
alterlab-ieu/alterlab-academic-skills
No summary provided.
Audit unavailable82 installs

