No end-to-end (E2E) test coverage

[oyale]

E2E Test Coverage for FilmAffinity Backup Tool

Problem Statement

The current test suite lacks end-to-end (E2E) test coverage for the complete backup and upload workflows. While unit tests and integration tests exist for individual components, there's no automated testing of the full user journey from FilmAffinity data scraping to IMDb upload completion.

Current Test Coverage Analysis

Existing Test Types

Unit Tests (pytest with selenium mocking):

• ✅ Browser automation functions
• ✅ Data processing logic
• ✅ CSV validation
• ✅ Reporting utilities
• ✅ Command-line interface

Integration Tests (pytest -m integration):

• ✅ FilmAffinity HTTP scraping
• ✅ IMDbPY client initialization
• ✅ CSV format validation
• ✅ Data export pipelines

Missing Coverage:

• ❌ Full backup workflow (scrape → process → upload)
• ❌ Real browser interactions for IMDb login/upload
• ❌ End-to-end data integrity validation
• ❌ Error recovery scenarios
• ❌ Cross-platform compatibility

Proposed E2E Test Scenarios

Primary E2E Workflow

# Complete backup and upload cycle
filmaffinity-backup scrape --user 12345 --output data/
filmaffinity-backup upload --csv data/watched.csv --imdb-user test@example.com

Test Scenarios:

1. Happy Path: Complete successful backup and upload
2. Partial Upload: Resume interrupted upload
3. Data Validation: Verify uploaded ratings match source data
4. Error Recovery: Handle network failures, rate limiting, login issues

Secondary Scenarios

• Multi-user backup workflows
• Large dataset processing (>1000 movies)
• Different export formats (Letterboxd, JSON)
• Concurrent upload sessions

Technical Approach

Test Framework Selection

Recommendation: Playwright for browser automation + pytest-playwright

Rationale:

• Modern, reliable browser automation
• Cross-browser support (Chrome, Firefox, Safari, Edge)
• Built-in test generation and debugging tools
• Better stability than Selenium for E2E testing
• Active maintenance and community support

Alternative: Selenium with existing infrastructure (higher maintenance)

Test Environment Setup

Local Development:

# Install Playwright browsers
playwright install

# Run E2E tests
pytest tests/e2e/ --headed  # Visual debugging
pytest tests/e2e/ --browser chromium --headed

CI/CD Integration:

# GitHub Actions example
- name: Install Playwright
  run: |
    pip install playwright pytest-playwright
    playwright install-deps
    playwright install chromium

- name: Run E2E Tests
  run: pytest tests/e2e/ --browser chromium --video retain-on-failure

Test Data Strategy

Test Accounts:

• Dedicated FilmAffinity test user with known ratings
• IMDb test account for upload validation
• Pre-populated test datasets

Data Isolation:

• Use test-specific user IDs/accounts
• Clean up test data after execution
• Rate limiting awareness (FilmAffinity is strict)

Implementation Plan

Phase 1: Infrastructure Setup

1. Add Playwright dependencies

   # pyproject.toml
   [tool.pytest.ini_options]
   addopts = "--browser chromium"
   
   [project.optional-dependencies]
   e2e = ["pytest-playwright", "playwright"]

2. Create E2E test directory structure

   tests/e2e/
   ├── conftest.py          # Playwright fixtures
   ├── test_full_backup.py  # Main E2E scenarios
   ├── test_error_scenarios.py
   ├── test_data_integrity.py
   └── fixtures/
       ├── test_users.json
       └── sample_data/
   

3. Set up test fixtures

   # tests/e2e/conftest.py
   @pytest.fixture(scope="session")
   def browser_context_args(browser_context_args):
       return {
           **browser_context_args,
           "record_video_dir": "test-results/videos/",
           "record_har_path": "test-results/har/",
       }

Phase 2: Core E2E Tests

Test 1: Complete Backup Workflow

def test_full_backup_workflow(page, test_user):
    """Test complete FilmAffinity → IMDb backup cycle."""

    # Step 1: Scrape FilmAffinity data
    run_cli_command(f"filmaffinity-backup scrape --user {test_user['fa_id']}")

    # Step 2: Verify scraped data
    csv_path = f"data/{test_user['fa_id']}/watched.csv"
    assert_csv_has_expected_structure(csv_path)

    # Step 3: Upload to IMDb (browser automation)
    page.goto("https://www.imdb.com/")
    login_to_imdb(page, test_user['imdb_credentials'])

    # Step 4: Upload ratings
    run_cli_command(f"filmaffinity-backup upload --csv {csv_path}")

    # Step 5: Verify upload success
    verify_ratings_on_imdb(page, expected_ratings)

Test 2: Error Recovery

def test_upload_resume_after_failure(page, test_user):
    """Test resuming upload after network interruption."""

    # Simulate partial upload
    # Trigger network failure mid-upload
    # Verify resume functionality
    # Confirm all ratings eventually uploaded

Test 3: Data Integrity

def test_data_integrity_preservation(page, test_user):
    """Verify uploaded ratings match source data exactly."""

    # Compare FilmAffinity ratings with IMDb ratings
    # Validate all metadata (titles, years, scores)
    # Check for data corruption or loss

Phase 3: CI/CD Integration

1. GitHub Actions workflow
2. Parallel test execution
3. Artifact collection (videos, HAR files, screenshots)
4. Test result reporting

Phase 4: Maintenance & Monitoring

1. Test stability monitoring
2. IMDb/FilmAffinity website change detection
3. Automated test data refresh
4. Performance regression tracking

Acceptance Criteria

Functional Requirements

• E2E test covers complete backup workflow
• Tests run successfully in CI environment
• Test execution time < 10 minutes
• 95%+ test reliability (flaky test detection)
• Cross-browser compatibility (Chrome minimum)

Quality Requirements

• Clear test failure diagnostics
• Video recordings for failed tests
• Comprehensive error logging
• Test data isolation and cleanup

Documentation Requirements

• E2E test setup instructions in CONTRIBUTING.md
• Test data management documentation
• CI/CD pipeline documentation
• Troubleshooting guide for common failures

Dependencies & Prerequisites

External Dependencies

• IMDb test account with known ratings
• FilmAffinity test user profile
• Stable internet connection for CI
• Sufficient rate limit allowances

Internal Dependencies

• Completion of selenium mocking improvements (Add retry mechanism to FilmAffinity scraper #57)
• Stable CLI interface
• Reliable browser automation functions

Risk Assessment

High Risk

• Rate Limiting: FilmAffinity/IMDb may block test accounts
• Website Changes: IMDb/FilmAffinity UI updates break tests
• Test Data Management: Maintaining test accounts and data

Mitigation Strategies

• Rate Limiting: Use dedicated test accounts, implement delays
• Website Changes: Regular test maintenance, change detection alerts
• Test Data: Automated test data refresh, multiple test accounts

Success Metrics

• Coverage: E2E tests cover 80%+ of user workflows
• Reliability: <5% flaky test rate
• Speed: E2E test suite completes in <15 minutes
• Maintenance: <2 hours/month for test upkeep

References

• Playwright Documentation
• Testing Pyramid
• E2E Testing Best Practices

[m13v]

Good call on Playwright for the E2E tests. A few things from our experience running Playwright against sites that require authentication (which FilmAffinity and IMDb both do):

For the login flow testing, consider persisting the authenticated session state to a JSON file (using Playwright's storageState) and reloading it in subsequent test runs. This way you do not need to go through the full login flow for every test execution, which saves time and avoids triggering rate limits or CAPTCHAs on repeated logins.

For the concurrent upload sessions scenario, you will need to isolate browser profiles per test worker. Playwright workers sharing a browser context will step on each other's cookie state, causing flaky auth failures. Each worker should get its own browser instance with its own storage directory.

For error recovery tests (network failures, rate limiting), Playwright's route interception is great for simulating these without actually hitting the real servers. You can intercept requests and return 429s or drop connections to test your retry logic deterministically.

[m13v]

Here is the lock script we use for isolating concurrent Playwright sessions per test worker: https://github.com/m13v/browser-lock/blob/main/playwright-lock.sh

It wraps browser launches with file-based locks so parallel test workers do not corrupt each other's session state.