Python Development

Production-quality Python development with fail-fast philosophy, type safety, and modern patterns.

Standard Tools

Use the best industry-standard tool for each job:

• Package management: uv
• Testing: pytest
• Git hooks: pre-commit
• Type checking: mypy
• Linting: ruff

Core Philosophy

1. Fail-Fast: No Defaults, No Fallbacks

[[fail-fast.md]]

Critical rule: Silent defaults corrupt research data. Fail immediately so problems are fixed, not hidden.

2. Type Safety Always

[[type-safety.md]]

All function signatures, class attributes, and complex data structures must have type hints.

3. Testing: Mock Only at Boundaries

[[testing.md]]

Never mock your own code. Mock only at system boundaries you don't control.

4. Modern Python Patterns

[[modern-python.md]]

Use pathlib, f-strings, comprehensions, and modern Python idioms.

5. Code Quality Standards

[[code-quality.md]]

Docstrings, clear names, focused functions, organized imports.

Development Workflow

Step 1: Design Before Code

Before writing any code:

1. Define the interface - Function signatures with type hints
2. Write docstring - What it does, Args, Returns, Raises
3. Consider edge cases - Empty inputs, None values, invalid data

Example:

def process_records(
    records: List[Dict[str, Any]],
    output_path: Path,
) -> int:
    """Process records and write to file.

    Args:
        records: List of record dictionaries
        output_path: Where to save results

    Returns:
        Number of records processed

    Raises:
        ValueError: If records is empty
        IOError: If output_path cannot be written
    """
    if not records:
        raise ValueError("records cannot be empty")

    # Implementation follows design

Step 2: Write Test First (TDD)

CRITICAL - Test Infrastructure Requirements:

Before writing ANY test, check:

1. Use EXISTING test infrastructure - Check conftest.py for fixtures
2. Connect to EXISTING live data - Use project configs to find data locations
3. Test against REAL data - No fake data, no new databases
4. Never create new test data - Don't run pipelines, don't create databases

Strict Prohibitions:

• ❌ Creating new databases/collections
• ❌ Running vectorization/indexing pipelines
• ❌ Creating new configs (use existing project configs)
• ❌ Generating fake/mock data (unless external API)

Checklist before writing test:

• Checked conftest.py for existing fixtures?
• Identified live data location from project config?
• Confirmed test uses real data, not mocks?
• Verified NOT creating new databases/configs?

Example using existing infrastructure:

def test_process_records_basic(db_session):  # Uses existing fixture
    """Test basic record processing with real database."""
    # Connects to existing database via fixture
    records = db_session.query(Record).limit(10).all()
    output_path = tmp_path / "output.json"

    count = process_records(records, output_path)

    assert count == len(records)
    assert output_path.exists()


def test_process_records_empty_raises():
    """Test that empty records raises ValueError."""
    with pytest.raises(ValueError, match="records cannot be empty"):
        process_records([], Path("output.json"))

Step 3: Implement with Quality

Code quality checklist:

• Type hints on all parameters and returns
• Docstring with Args/Returns/Raises
• No .get() with defaults for required config
• No bare except: clauses
• Uses pathlib for file operations
• Uses f-strings for formatting
• Pydantic for configuration/validation
• Clear variable names
• Single responsibility

Step 4: Run and Debug

# Run specific test
uv run pytest tests/test_module.py::test_function -xvs

# Run with coverage
uv run pytest tests/ --cov=src

# Type checking
uv run mypy src/

# Linting
uv run ruff check src/

DO NOT:

• Create test.py, debug.py, verify.py scripts
• Use python -c "..." to check things
• Add try/except to hide errors
• Guess at fixes

DO:

• Fix the root cause
• Add test for the bug
• Verify fix with pytest
• Use logger (not print) for debugging

Subagent Delegation Patterns

When delegating TDD tasks to subagents, use these specific prompt templates to ensure quality.

1. Test Creation Prompt

Task(subagent_type="general-purpose", prompt="
Create ONE failing test.

**FIRST**: Invoke Skill(skill='python-dev') to load coding standards and test patterns.

Acceptance criterion being tested: {criterion}
Behavior to test: {behavior}
File: tests/test_{name}.py

Test requirements (python-dev skill enforces):
- Use EXISTING test infrastructure (check conftest.py for fixtures)
- Connect to EXISTING live data using project configs
- Use REAL production data (NO fake data, NO new databases)
- NEVER create new databases/collections for testing
- NEVER create new configs - use existing project configs
- NEVER run vectorization/indexing to create test data
- NEVER mock internal code (only external APIs)
- Integration test pattern testing complete workflow
- Test should fail with: {expected_error}

After completing, STOP and report:
- Test location: tests/test_{name}.py::test_{function_name}
- Run command: uv run pytest {test_location} -xvs
- Actual failure message received
- Confirm failure is due to missing implementation (not test setup error)
")

2. Implementation Prompt

Task(subagent_type="general-purpose", prompt="
Implement MINIMAL code to make this ONE test pass.

**FIRST**: Invoke Skill(skill='python-dev') to load coding standards.

Test: tests/test_{name}.py::{function}
Error message: {error_message}

Implementation requirements:
- File to modify: {file}
- Function/method: {location}
- Behavior needed: {behavior}
- Constraints: Minimal change, fail-fast principles (no .get(), no defaults)

Tools to use:
1. Read {file} to understand current implementation
2. Edit {file} to add functionality
3. Run test using Bash: uv run pytest {test_path} -xvs to verify
4. If test passes, run all tests: uv run pytest

After implementation, STOP and report:
- What you changed (describe the logic you added)
- Files modified
- Test results (specific test and full suite)
")

3. Quality Check Prompt (Before Commit)

Task(subagent_type="general-purpose", prompt="
Validate code quality and commit this change.

**FIRST**: Invoke Skill(skill='python-dev') to load quality standards.

Changes summary: {what_was_implemented}
Test: {test_name}
Files modified: {files}

Validation checklist (python-dev skill enforces):
1. Code quality against fail-fast principles (no .get(key, default), no defaults, no fallbacks)
2. Test patterns (real fixtures used, no mocked internal code)
3. Type safety and code structure
4. Execute commit ONLY if all validation passes

If validation FAILS:
- Report ALL violations with file:line numbers
- STOP and report violations
- DO NOT commit

If validation PASSES:
- Create commit automatically
- Commit message follows conventional commits format
- Report commit hash
")

Critical Rules

NEVER:

• Modify research configurations, flows, or pipelines substantively - This is a CRITICAL violation of academic integrity (AXIOM #24). Report problems, propose fixes, WAIT for explicit approval.
• Use .get() with defaults for required configuration
• Use bare except: or overly broad exception handling
• Create standalone test/debug/verify scripts outside tests/
• Use print() for logging in production code
• Use os.path when pathlib works
• Skip type hints on public functions
• Add defaults to Pydantic required fields

ALWAYS:

• Fail immediately on missing required config
• Use specific exception types
• Use pytest for all testing
• Use logger for debugging/info messages
• Use pathlib for file operations
• Add type hints to all function signatures
• Validate at system boundaries with Pydantic

Quick Reference

Template for new function:

from typing import List, Dict, Any
from pathlib import Path


def function_name(
    param1: str,
    param2: List[int],
) -> Dict[str, Any]:
    """Short description.

    Args:
        param1: Description
        param2: Description

    Returns:
        Description of return value

    Raises:
        ValueError: When and why
    """
    if not param1:
        raise ValueError("param1 cannot be empty")

    result = {}
    return result

Template for Pydantic config:

from pydantic import BaseModel, Field, field_validator
from pathlib import Path


class MyConfig(BaseModel):
    """Configuration with validation."""

    # Required (no defaults)
    required_field: str
    required_path: Path

    # Optional (with defaults)
    optional_field: int = 10

    @field_validator("required_path")
    @classmethod
    def validate_path(cls, v: Path) -> Path:
        if not v.exists():
            raise ValueError(f"Path does not exist: {v}")
        return v

Common commands:

# Run tests
uv run pytest tests/test_module.py -xvs

# Type check
uv run mypy src/

# Lint
uv run ruff check --fix src/

Reference Files

Core Python Standards

• [[references/fail-fast.md]] - No defaults, explicit config
• [[references/type-safety.md]] - Type hints, Pydantic validation
• [[references/testing.md]] - Testing philosophy and pytest patterns
• [[references/modern-python.md]] - Modern Python idioms
• [[references/code-quality.md]] - Docstrings, naming, organization

Major Libraries

• [[references/pandas.md]] - Data manipulation and analysis
• [[references/hydra.md]] - Configuration management
• [[references/fastapi.md]] - High-performance web APIs
• [[references/fastmcp.md]] - Model Context Protocol servers
• [[references/bigquery.md]] - Google BigQuery data warehouse