Test Coverage Audit

Forensic analysis of test coverage quality. Not just "do you have tests" but "are you testing the right things at the right level?"

This skill produces an exhaustive audit report. For recommendations based on this report, use the test-recommendations skill. For implementation planning, use test-implementation-plan.

Philosophy

The Testing Pyramid

         ╱╲
        ╱E2E╲          Few, slow, high-confidence
       ╱──────╲
      ╱ Integ  ╲       Medium count, medium speed
     ╱──────────╲
    ╱    Unit    ╲     Many, fast, focused
   ╱──────────────╲

Comprehensive Testing Level Definitions: concepts/testing-levels.md

Testing at the Right Level

Wrong level → wasted effort, false confidence, or fragile tests

Common AI/LLM Testing Mistakes

When AI generates tests, it often makes systematic errors. Read: concepts/llm-testing-mistakes.md

Audit Process

Phase 1: Complexity Source Detection

Goal: Create an exhaustive inventory of everything that needs testing.

1.1 Architecture Detection

Is this a microservices/distributed system?

Read: detection/microservices.md

Output:

### Architecture Classification
- Type: [Monolith | Modular Monolith | Microservices | Serverless]
- Services detected: [list with protocols]
- Inter-service communication: [HTTP | gRPC | Message Queue | None]
- Contract testing present: [Yes/No]

1.2 Data Interaction Detection

What data does this system touch?

Read: detection/data-interactions.md

Output:

### Data Interactions
| Category | Technology | Locations | Tested? |
|----------|------------|-----------|---------|
| Database | PostgreSQL/SQLAlchemy | models/*.py | ✅/❌ |
| Cache | Redis | services/cache.py | ✅/❌ |
| Files | S3/boto3 | storage/*.py | ✅/❌ |
| Config | pydantic/settings | config.py | ✅/❌ |
| Secrets | AWS SecretsManager | auth/*.py | ✅/❌ |

1.3 External API Detection

What external services does this call?

Read: detection/external-apis.md

Output:

### External API Integrations
| Service | SDK/Client | Criticality | Error Handling? | Tested? |
|---------|------------|-------------|-----------------|---------|
| Stripe | stripe-python | Critical | ⚠️ Partial | ❌ |
| SendGrid | sendgrid | High | ❌ None | ❌ |
| Auth0 | auth0-python | Critical | ✅ Yes | ✅ |

1.4 Interactive/User Input Detection

Does this require user interaction for testing?

Read: concepts/interactive-testing.md

Output:

### Interactive Components
| Component | Type | Can Test in CI? | Current Approach |
|-----------|------|-----------------|------------------|
| Setup wizard | CLI prompts | ✅ (pexpect) | ❌ Untested |
| Tab completion | Shell integration | ✅ (script) | ❌ Untested |
| Dashboard | Full-screen TUI | ⚠️ (with pyte) | ❌ Untested |

Phase 2: Detect Project Type & Language

2.1 Project Type Detection

Identify the scenario to set testing expectations:

Read the appropriate scenario file for testing expectations specific to that project type.

2.2 Language/Framework Detection

Phases 2-5: Forensic Analysis

For the detailed forensic analysis (test inventory, coverage mapping, quality assessment, gap analysis), spawn the test-auditor agent:

Use the Task tool to spawn do:test-auditor agent:

Execute forensic test coverage analysis.

Project: [current working directory]
Framework: [detected framework from Phase 1]
Intensity: [quick|medium|thorough]

Run phases 2-5:
- Phase 3: Test Inventory
- Phase 4: Coverage Mapping
- Phase 5: Quality Assessment
- Phase 6: Gap Analysis

Output: TEST-AUDIT-<timestamp>.md in .agent_planning/

The agent will complete the audit report with all remaining phases.

Output Format

The audit produces a comprehensive report:

# Test Coverage Audit Report
**Project**: [name]
**Date**: [date]
**Auditor**: Claude

## Executive Summary
**Overall Health**: [Healthy | Needs Work | Critical Gaps]
**Architecture**: [type]
**Coverage Distribution**: Unit n% | Integration n% | E2E n%
**Critical Issues**: [count]

---

## 1. Architecture Analysis
[From Phase 1.1]

## 2. Complexity Source Inventory
### 2.1 Data Interactions
[From Phase 1.2]

### 2.2 External APIs
[From Phase 1.3]

### 2.3 Interactive Components
[From Phase 1.4]

---

## 3. Test Inventory
[From Phase 3]

---

## 4. Coverage Matrix
[From Phase 4]

---

## 5. Quality Assessment
### 5.1 Red Flags Detected
[From Phase 5.1]

### 5.2 Quality Checklist Results
[From Phase 5.2]

---

## 6. Gap Analysis

### P0 - Critical (Must Fix)
[From Phase 6]

### P1 - Significant (Should Fix)
[From Phase 6]

### P2 - Minor (Nice to Have)
[From Phase 6]

---

## 7. Risk Assessment

| Risk | Impact | Likelihood | Current Mitigation |
|------|--------|------------|-------------------|
| Payment failures undetected | High | Medium | ❌ None |
| Auth bypass possible | Critical | Low | ⚠️ Partial |

---

## 8. Appendix

### A. Files Analyzed
[List of all files examined]

### B. Test File Inventory
[Complete list of test files]

### C. Detection Commands Used
[Commands run during audit]

Intensity Levels

Reference Documents

Concepts

Detection

Scenarios (15)

Languages (6)

Integration

This skill is invoked as a dimension of /do:plan audit:

• Trigger: "audit tests", "test coverage audit", "testing audit"
• Can run alongside other audit dimensions

Related Skills