Logging Configuration Migration Guide

Overview

This guide documents the migration from scattered logging.basicConfig() calls to the centralized LoggingManager system in autobot-backend/utils/logging_manager.py.

Issue: #42 - Logging Configuration Standardization

Why Centralized Logging?

Problems with `logging.basicConfig()`:

Configuration Conflicts - Multiple calls overwrite each other
No Categorization - All logs mixed together, hard to filter
No Rotation - Log files grow unbounded
Inconsistent Formatting - Different formats across codebase
No Environment Control - Can't configure via env vars

Benefits of Centralized Logging:

✅ Single Configuration Point - autobot-backend/utils/logging_manager.py
✅ Category-Based Loggers - backend, frontend, llm, debug, audit
✅ Automatic Log Rotation - 10MB max size, 5 backups
✅ Environment-Based Config - AUTOBOT_LOG_LEVEL, AUTOBOT_LOGS_DIR
✅ Consistent Formatting - Same format across all modules
✅ Backward Compatible - Convenience functions for easy migration

Migration Pattern

Core Production Files (COMPLETE - 6/6 ✅)

Migrated Files:

✅ backend/main.py
✅ backend/app_factory_enhanced.py
✅ backend/utils/redis_compatibility.py
✅ src/project_state_manager.py
✅ autobot-backend/agents/research_agent.py
✅ autobot-backend/utils/system_context.py

Before Migration:

import logging

# Module-level logging setup
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger(__name__)

# ... code ...

if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG)
    # ... test code ...

After Migration:

from src.utils.logging_manager import get_logger

# Get centralized logger
logger = get_logger(__name__, "backend")

# ... code ...

if __name__ == "__main__":
    # Logging configured via centralized logging_manager
    # ... test code ...

Step-by-Step Migration

Step 1: Update Imports

Remove:

import logging

Add:

from src.utils.logging_manager import get_logger

Step 2: Replace Logger Initialization

Replace:

logger = logging.getLogger(__name__)

With:

logger = get_logger(__name__, "backend")  # or appropriate category

Step 3: Remove `logging.basicConfig()` Calls

Remove all instances of:

logging.basicConfig(level=logging.INFO)
logging.basicConfig(level=logging.DEBUG, format="...")
logging.basicConfig(...)

Replace with comment:

# Logging configured via centralized logging_manager

Step 4: Choose Appropriate Category

Select the correct category for your logger:

Category	Use Case	Example Modules
`backend`	Backend API, services, utilities	`autobot-backend/api/.py`, `autobot-backend/utils/.py`
`frontend`	Frontend-related backend code	Vue build scripts, frontend helpers
`llm`	LLM interactions, prompts	`src/llm_interface.py`, LLM services
`debug`	Debug/development logging	Test utilities, debug scripts
`audit`	Security audit logs	Security layer, auth modules

Common Migration Scenarios

Scenario 1: Simple Module Logger

Before:

import logging

logger = logging.getLogger(__name__)

def my_function():
    logger.info("Processing data")

After:

from src.utils.logging_manager import get_logger

logger = get_logger(__name__, "backend")

def my_function():
    logger.info("Processing data")

Scenario 2: Module with `basicConfig()`

Before:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

After:

from src.utils.logging_manager import get_logger

logger = get_logger(__name__, "backend")

Scenario 3: `main` Block with Logging

Before:

if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
    main()

After:

if __name__ == "__main__":
    # Logging configured via centralized logging_manager
    main()

Scenario 4: Test Script with Multiple Loggers

Before:

import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
test_logger = logging.getLogger("test")

def test_function():
    logger.debug("Test starting")
    test_logger.info("Running tests")

After:

from src.utils.logging_manager import get_logger

logger = get_logger(__name__, "debug")
test_logger = get_logger("test", "debug")

def test_function():
    logger.debug("Test starting")
    test_logger.info("Running tests")

Scenario 5: Async Module with Logging

Before:

import asyncio
import logging

logger = logging.getLogger(__name__)

async def async_task():
    logger.info("Async task started")

After:

import asyncio
from src.utils.logging_manager import get_logger

logger = get_logger(__name__, "backend")

async def async_task():
    logger.info("Async task started")

Testing Your Migration

1. Verify Imports

# Check that old logging imports are removed
grep -r "import logging" backend/ src/ --include="*.py"

# Should only see centralized logging_manager imports
grep -r "from src.utils.logging_manager import get_logger" backend/ src/

2. Verify No `basicConfig()` Calls

# Check for remaining basicConfig calls (excluding logging_manager.py itself)
grep -r "logging.basicConfig" backend/ src/ --include="*.py" \
  --exclude="logging_manager.py"

3. Test Logging Output

# Test script to verify logging works
from src.utils.logging_manager import get_logger

logger = get_logger("test_migration", "debug")
logger.debug("Debug message")
logger.info("Info message")
logger.warning("Warning message")
logger.error("Error message")

Expected output in logs/backend.log:

2025-01-14 10:30:00 - test_migration - DEBUG - Debug message
2025-01-14 10:30:00 - test_migration - INFO - Info message
2025-01-14 10:30:00 - test_migration - WARNING - Warning message
2025-01-14 10:30:00 - test_migration - ERROR - Error message

4. Verify Log Rotation

# Check that log files are created with rotation
ls -lh logs/
# Should see: backend.log, backend.log.1, backend.log.2, etc.

Environment Configuration

Control logging via environment variables:

# Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
export AUTOBOT_LOG_LEVEL=DEBUG

# Set logs directory
export AUTOBOT_LOGS_DIR=/path/to/logs

# Set backup directory
export AUTOBOT_LOGS_BACKUP_DIR=/path/to/backups

Remaining Files to Migrate

Scripts (Low Priority)

Pattern for scripts:

Replace logging.basicConfig() with centralized logger
Use debug category for most scripts
Keep print statements for user-facing output

Examples:

scripts/detect-hardcoded-values.sh  # Shell script - no migration needed
scripts/test_*.py                   # Use "debug" category
scripts/utilities/*.py              # Use "debug" category

Tests (Low Priority)

Pattern for tests:

Use debug category for test logging
Keep test framework logging separate
Only migrate test utilities, not test assertions

Examples:

tests/unit/*.py          # Use "debug" category
tests/integration/*.py   # Use "debug" category

Migration Commands

# Find all files with logging.basicConfig
find backend/ src/ scripts/ tests/ -name "*.py" -exec grep -l "logging.basicConfig" {} \;

# Count remaining migrations
grep -r "logging.basicConfig" backend/ src/ scripts/ tests/ --include="*.py" \
  --exclude="logging_manager.py" | wc -l

Backward Compatibility

The centralized logging system provides convenience functions for backward compatibility:

# All of these work and map to appropriate categories:
from src.utils.logging_manager import (
    get_logger,           # General-purpose (preferred)
    get_backend_logger,   # backend category
    get_frontend_logger,  # frontend category
    get_llm_logger,       # llm category
    get_debug_logger,     # debug category
    get_audit_logger,     # audit category
)

# Usage:
backend_logger = get_backend_logger("my_module")
llm_logger = get_llm_logger("my_llm_code")

Troubleshooting

Issue: "No handlers could be found for logger"

Solution: Ensure LoggingManager.initialize() is called during app startup

Already configured in backend/main.py and backend/app_factory.py
For standalone scripts, logging is initialized on first get_logger() call

Issue: Logs not appearing in file

Solution: Check log directory permissions and environment variables

# Verify logs directory exists and is writable
mkdir -p logs
chmod 755 logs

# Check environment variables
echo $AUTOBOT_LOG_LEVEL
echo $AUTOBOT_LOGS_DIR

Issue: Log rotation not working

Solution: Verify RotatingFileHandler configuration in logging_manager.py

Default: 10MB max size, 5 backups
Check disk space and permissions

Issue: Duplicate log entries

Solution: Remove any remaining logging.basicConfig() calls

# Find remaining basicConfig calls
grep -r "logging.basicConfig" . --include="*.py" --exclude="logging_manager.py"

Migration Checklist

For each file being migrated:

Replace import logging with from src.utils.logging_manager import get_logger
Replace logger = logging.getLogger(__name__) with logger = get_logger(__name__, category)
Remove all logging.basicConfig() calls
Add comment # Logging configured via centralized logging_manager
Choose appropriate category (backend, frontend, llm, debug, audit)
Test that logging works correctly
Verify no duplicate log entries
Check log files are created in logs/ directory

Migration Progress

Core Production Files: 6/6 (100%) ✅

✅ backend/main.py
✅ backend/app_factory_enhanced.py
✅ backend/utils/redis_compatibility.py
✅ src/project_state_manager.py
✅ autobot-backend/agents/research_agent.py
✅ autobot-backend/utils/system_context.py

Remaining (Low Priority):

Scripts: ~20-30 files
Tests: ~70-80 files

Total Migrated: 6 core files (critical paths complete) Remaining: Non-critical scripts and tests (can be migrated incrementally)

References

Centralized Logging Implementation: autobot-backend/utils/logging_manager.py
GitHub Issue: #42 - Logging Configuration Standardization
Python Logging Documentation: https://docs.python.org/3/library/logging.html
RotatingFileHandler: https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler

Last Updated: 2025-01-14 Migration Status: Core production files complete (6/6)

FilesExpand file tree

LOGGING_MIGRATION_GUIDE.md

Latest commit

History

LOGGING_MIGRATION_GUIDE.md

File metadata and controls

Logging Configuration Migration Guide

Overview

Why Centralized Logging?

Problems with logging.basicConfig():

Benefits of Centralized Logging:

Migration Pattern

Core Production Files (COMPLETE - 6/6 ✅)

Before Migration:

After Migration:

Step-by-Step Migration

Step 1: Update Imports

Step 2: Replace Logger Initialization

Step 3: Remove logging.basicConfig() Calls

Step 4: Choose Appropriate Category

Common Migration Scenarios

Scenario 1: Simple Module Logger

Scenario 2: Module with basicConfig()

Scenario 3: __main__ Block with Logging

Scenario 4: Test Script with Multiple Loggers

Scenario 5: Async Module with Logging

Testing Your Migration

1. Verify Imports

2. Verify No basicConfig() Calls

3. Test Logging Output

4. Verify Log Rotation

Environment Configuration

Remaining Files to Migrate

Scripts (Low Priority)

Tests (Low Priority)

Migration Commands

Backward Compatibility

Troubleshooting

Issue: "No handlers could be found for logger"

Issue: Logs not appearing in file

Issue: Log rotation not working

Issue: Duplicate log entries

Migration Checklist

Migration Progress

Core Production Files: 6/6 (100%) ✅

Remaining (Low Priority):

References

Problems with `logging.basicConfig()`:

Step 3: Remove `logging.basicConfig()` Calls

Scenario 2: Module with `basicConfig()`

Scenario 3: `main` Block with Logging

2. Verify No `basicConfig()` Calls