Electrs gets stuck during Bitcoin mainnet indexing on high-spec hardware

[pxmtYann]

Description

Liquid is working fine and Bitcoind syncs successfully to latest block but for the Bitcoin case, Electrs repeatedly gets stuck in uninterruptible sleep state during Bitcoin mainnet indexing, despite using high-performance hardware. The process appears to be waiting indefinitely on disk I/O operations during database compaction cycles.

Environment

Hardware:

Instance: AWS EC2 r6i.2xlarge

• 8 vCPUs
• 64GB RAM

Storage: EBS gp3 volume

• Size: 3TB
• IOPS: 16,000
• Throughput: 1,000 MB/s

Configuration

services:
  esplora-bitcoin:
    image: blockstream/esplora:latest
    command: ["/srv/explorer/run.sh", "bitcoin-mainnet", "explorer"]
    environment:
      - ENABLE_LIGHTMODE=1
    ports:
      - "50001:50001"
      - "8080:80"
    volumes:
      - ./data_bitcoin_mainnet:/data
    restart: unless-stopped
    stdin_open: true
    tty: true
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
    networks:
      - bitcoin-net

  esplora-liquid:
    image: blockstream/esplora:latest
    command: ["/srv/explorer/run.sh", "liquid-mainnet", "explorer"]
    ports:
      - "50002:50001"
      - "8082:80"
    volumes:
      - ./data_liquid_mainnet:/data
    restart: unless-stopped
    stdin_open: true
    tty: true
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
    networks:
      - liquid-net

networks:
  bitcoin-net:
  liquid-net:

Issue Details

What works:

• ✅ Liquid explorer: Indexes and runs successfully
• ✅ Bitcoin daemon (bitcoind): Syncs to 100% without issues
• ✅ Disk performance: Confirmed with iostat showing 95-96% utilization during operations

What fails:

• ❌ Bitcoin electrs indexing: Gets stuck repeatedly during indexing process
• ❌ Process state: Enters "D" state (uninterruptible sleep) and remains stuck for days
• ❌ Each container restart causes electrs to start indexing from block 0 again

Observed Behavior

1. Bitcoind syncs successfully to latest block (e.g., block 919,039)
2. Electrs starts indexing: INFO hash=... height=919038 (919039 left to index)
3. Electrs process shows high CPU usage initially (20-25%)
4. After some time, process enters "D" state with 0% CPU:

   PID  USER  STATE  %CPU  %MEM  COMMAND
   76   root  D      0.0   22.1  electrs

5. Process remains stuck in D state for 24-48+ hours
6. No progress in logs - last log entry remains unchanged for days
7. Database grows to ~1.1TB but indexing never completes

Process State Details

When stuck, top output shows:

76 root  20  0  14.6g  13.7g  4352  D  0.0  22.1  10:16  electrs

• State: D (Uninterruptible Sleep - waiting on disk I/O)
• CPU: 0.0% (completely blocked)
• Memory: Normal usage

Disk I/O During Stuck State

iostat -x output during stuck periods:

Device   r/s    rkB/s  w/s    wkB/s  %util
nvme0n1  1687   11754  10.5   54     95-96%

1. Any insights into if what we are seeing is normal?
2. What is the recommended specs?
3. Is it possible to get an pre indexed version of the db and start for there?

[philippem]

what is the last log entry from electrs?
After retrieving blocks from bitcoin core, electrs will enter a compaction phase which may take 48 hours + even on this hardware configuration. If possible, can you consider a higher level of storage? (io2)

[pxmtYann]

Thanks @philippem .

The last electrs log entry has remained unchanged for over 48 hours:

docker-compose exec esplora-bitcoin tail -1 /data/logs/electrs/current                           
2-2025-10-10T15:59:35.134+00:00 - INFO hash=00000000000000000001285042e8ae0cdcdf0d180af08b7f73570fadfff97328 height=918462 @ 2025-10-10T15:57:11Z (918463 left to index)

Questions:

1. During compaction, what logs should I monitor to confirm compaction is actually happening vs being stuck?
2. The process has been in this state for 2+ days now - is this expected duration even with gp3 16K IOPS?

Regarding io2:

1. What io2 specifications would you recommend?
2. Would io2 with similar or higher specs resolve the compaction hang issues?
3. What IOPS/throughput targets should I aim for?

[philippem]

it may take several day for sync and compaction to complete, and you will see electrs in D state during that time.
You can get some feedback from electr's progress by setting an additional environment variable ELECTRS_AGS=-vvvv which will enable tracing. There won't be a percentage complete, but there will trace messages about blocks being indexed.