A/B Test Design and Analysis

Help me design and analyze an A/B test:

What I'm testing: {{change_description}}
Current metric baseline: {{conversion_rate}}
Minimum detectable effect: {{min_detectable_effect}}
Traffic volume: {{daily_visitors}}
Test duration preference: {{test_duration}}

Provide:

1. **Hypothesis**: Clearly stated H0 and H1

2. **Sample size calculation**:
   - Required sample per variation
   - Estimated test duration based on traffic
   - Statistical power (target 80%)
   - Significance level (α = 0.05)

3. **Test design**:
   - Control and treatment descriptions
   - Randomization strategy
   - Segmentation (should you test on all users or a subset?)
   - Guardrail metrics (what should NOT change)

4. **Implementation checklist**:
   - [ ] Tracking events defined
   - [ ] QA'd on both variations
   - [ ] No other tests running on the same traffic
   - [ ] Sample ratio mismatch monitoring
   - [ ] Data pipeline verified

5. **Analysis plan**:
   - Primary metric and how to calculate it
   - Secondary metrics
   - When to check results (don't peek too early!)
   - How to handle inconclusive results

6. **Results interpretation template**:
   - "The [treatment] showed a [X%] [increase/decrease] in [metric] compared to control"
   - Confidence interval
   - Practical significance vs statistical significance
   - Recommendation: Ship / Iterate / Kill