Ab Testing
Published by mtent in speshul-skills
What this skill does
Design, implement, analyze, document, and govern A/B tests, split tests, multivariate tests, holdouts, and growth experimentation programs. Use for hypotheses, variants, sample size, test duration, statistical uncertainty, experiment backlogs, or decisions between measurable alternatives.
Add Ab Testing to your agent
Review the source and files first. When you are ready, copy the prompt instruction or use the CLI command supported by your environment.
Install with a prompt
Paste this into a compatible coding agent:
add this skill "ab-testing" from https://github.com/mtent/speshul-skillsInstall with the CLI
Run this command in a controlled environment after reviewing the repository:
npx skills add https://github.com/mtent/speshul-skills --skill ab-testingSkill instructions
Ab Testing
Run an experiment only when exposure and outcomes can answer a decision and the organization can act on the result.
If .agents/marketing-context.md exists, read it first. Read experiment-system.md for hypotheses, designs, sample planning, implementation, analysis, and program operations.
Operating contract
- Start from an observed problem and a falsifiable mechanism, not a preferred variant.
- Choose randomized experiment, holdout, switchback, factorial design, monitored rollout, qualitative test, or observational comparison according to the decision and constraints.
- Predefine population, assignment, exposure, primary metric, practical threshold, guardrails, sample method, duration, and decision rule.
- Verify implementation and instrumentation before interpreting results.
- Report uncertainty, deviations, data-quality problems, and commercial consequences.
Do not repeatedly peek and stop only when the favored result wins, or treat statistical significance as proof of mechanism or business value.
Files included
- agents/openai.yaml
- references/experiment-system.md
- scripts/sample_size.py
- SKILL.md