[BUG] Photon not starting after initial download

[rouke-broersma]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

photon-docker configuration:

Please add your docker compose and environment variables.

System Info

• Host OS: Talos Linux
• Host Type: Kubernetes
• Hardware details:
  • CPU: Plenty
  • Available RAM: Plenty
  • Storage Type: Ceph
  • Storage Size: 500Gib

Debug Logs

[INFO] 2025-06-09 12:17:26 - Environment variables:
[INFO] 2025-06-09 12:17:26 - UPDATE_STRATEGY=PARALLEL
[INFO] 2025-06-09 12:17:26 - UPDATE_INTERVAL=96h
[INFO] 2025-06-09 12:17:26 - LOG_LEVEL=DEBUG
[INFO] 2025-06-09 12:17:26 - BASE_URL=https://download1.graphhopper.com/public/experimental
[INFO] 2025-06-09 12:17:26 - FORCE_UPDATE=false
[INFO] 2025-06-09 12:17:26 - SKIP_MD5_CHECK=false
[DEBUG] 2025-06-09 12:17:26 - Verifying directory structure at: /photon/photon_data
[INFO] 2025-06-09 12:17:26 - Found existing valid opensearch index
[INFO] 2025-06-09 12:17:26 - Ensuring correct ownership and permissions for /photon/photon_data/node_1
[DEBUG] 2025-06-09 12:17:26 - Current state for /photon/photon_data/node_1: Perms: 755, Owner: ubuntu (1000), Group: ubuntu (1000), Name: /photon/photon_data/node_1
[DEBUG] 2025-06-09 12:17:27 - Successfully changed ownership of /photon/photon_data/node_1 to 1000:1000.
[DEBUG] 2025-06-09 12:17:27 - Successfully changed permissions of /photon/photon_data/node_1 to 755.
[INFO] 2025-06-09 12:17:27 - Post-permission state for /photon/photon_data/node_1: Perms: 755, Owner: ubuntu (1000), Group: ubuntu (1000), Name: /photon/photon_data/node_1
[INFO] 2025-06-09 12:17:27 - Ensuring correct ownership and permissions for /photon/photon_data/node_1
[DEBUG] 2025-06-09 12:17:27 - Current state for /photon/photon_data/node_1: Perms: 755, Owner: ubuntu (1000), Group: ubuntu (1000), Name: /photon/photon_data/node_1
[DEBUG] 2025-06-09 12:17:27 - Successfully changed ownership of /photon/photon_data/node_1 to 1000:1000.
[DEBUG] 2025-06-09 12:17:27 - Successfully changed permissions of /photon/photon_data/node_1 to 755.
[INFO] 2025-06-09 12:17:27 - Post-permission state for /photon/photon_data/node_1: Perms: 755, Owner: ubuntu (1000), Group: ubuntu (1000), Name: /photon/photon_data/node_1
[INFO] 2025-06-09 12:17:27 - Starting Photon service
[INFO] 2025-06-09 12:17:27 - Photon service started successfully with PID: 41
[INFO] 2025-06-09 12:17:27 - Update strategy: PARALLEL
[INFO] 2025-06-09 12:17:27 - Update interval: 96h (345600 seconds)
[INFO] 2025-06-09 12:17:27 - Sleeping for 345600 seconds until next update
Cluster Name: photon
Base Path:    /photon/photon_data
Num Of Node:  1
Node Name:      Node 1
HTTP Port:      9201
Data Directory: /photon/photon_data/node_1/data
Log Directory:  /photon/photon_data/node_1/logs
[2025-06-09T12:17:28,009][INFO ][o.o.n.Node               ] version[2.19.1], pid[41], build[unknown/unknown/unknown], OS[Linux/6.12.28-talos/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.5/21.0.5+11-LTS]
[2025-06-09T12:17:28,017][INFO ][o.o.n.Node               ] JVM home [/opt/java/openjdk]
[2025-06-09T12:17:28,041][INFO ][o.o.n.Node               ] JVM arguments []
[2025-06-09T12:17:28,055][INFO ][o.o.p.PluginsService     ] no modules loaded
[2025-06-09T12:17:28,057][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.analysis.common.CommonAnalysisPlugin]
[2025-06-09T12:17:28,057][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.geo.GeoModulePlugin]
[2025-06-09T12:17:28,057][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.transport.Netty4Plugin]
[2025-06-09T12:17:28,118][WARN ][stderr                   ] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-06-09T12:17:28,118][WARN ][stderr                   ] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-06-09T12:17:28,118][WARN ][stderr                   ] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-06-09T12:17:28,205][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2025-06-09T12:17:28,210][INFO ][o.o.e.NodeEnvironment    ] using [1] data paths, mounts [[/photon/photon_data (/dev/rbd0)]], net usable_space [306.5gb], net total_space [491gb], types [ext4]
[2025-06-09T12:17:28,215][INFO ][o.o.e.NodeEnvironment    ] heap size [3.9gb], compressed ordinary object pointers [true]
[2025-06-09T12:17:28,312][WARN ][o.a.l.i.v.VectorizationProvider] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
[2025-06-09T12:17:28,428][INFO ][o.o.n.Node               ] node name [Node 1], node ID [5Ex_37lWRqiSwyIt3MAhDQ], cluster name [photon], roles [data, cluster_manager]
[2025-06-09T12:17:28,463][INFO ][o.o.e.ExtensionsManager  ] ExtensionsManager initialized
[2025-06-09T12:17:29,796][INFO ][o.o.t.NettyAllocator     ] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=512kb, suggested_max_allocation_size=512kb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=2mb}]
[2025-06-09T12:17:29,905][INFO ][o.o.d.DiscoveryModule    ] using discovery type [single-node] and seed hosts providers [settings]
[2025-06-09T12:17:30,194][WARN ][o.o.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-06-09T12:17:30,435][INFO ][o.o.n.Node               ] initialized
[2025-06-09T12:17:30,436][INFO ][o.o.n.Node               ] starting ...
[2025-06-09T12:17:30,515][INFO ][o.o.t.TransportService   ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2025-06-09T12:17:31,385][INFO ][o.o.c.c.Coordinator      ] cluster UUID [9MwEhVdDR768Po-hesCVNw]
[2025-06-09T12:17:31,679][INFO ][o.o.c.s.MasterService    ] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper, count:3 and sample tasks: elected-as-cluster-manager ([1] nodes joined)[{Node 1}{5Ex_37lWRqiSwyIt3MAhDQ}{hkN5oN2oThy2sIBFPz0j_g}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 4, version: 19, delta: cluster-manager node changed {previous [], current [{Node 1}{5Ex_37lWRqiSwyIt3MAhDQ}{hkN5oN2oThy2sIBFPz0j_g}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}
[2025-06-09T12:17:32,008][INFO ][o.o.c.s.ClusterApplierService] cluster-manager node changed {previous [], current [{Node 1}{5Ex_37lWRqiSwyIt3MAhDQ}{hkN5oN2oThy2sIBFPz0j_g}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}, term: 4, version: 19, reason: Publication{term=4, version=19}
[2025-06-09T12:17:32,038][INFO ][o.o.d.PeerFinder         ] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Node 1}{5Ex_37lWRqiSwyIt3MAhDQ}{hkN5oN2oThy2sIBFPz0j_g}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]
[2025-06-09T12:17:32,043][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:17:32,043][INFO ][o.o.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}
[2025-06-09T12:17:32,044][INFO ][o.o.n.Node               ] started
[2025-06-09T12:17:32,058][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:17:32,530][INFO ][o.o.g.GatewayService     ] recovered [1] indices into cluster_state
[2025-06-09T12:17:32,565][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:17:32,646][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:17:32,688][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:17:32,709][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:17:33,756][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/M-E3ihQCQ3ycHOQ34sL63g]
[2025-06-09T12:17:34,283][WARN ][o.o.c.r.a.AllocationService] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2025-06-09T12:18:01,953][WARN ][stderr                   ] Exception in thread "main" org.codelibs.opensearch.runner.OpenSearchRunnerException: ensureYellow timed out, cluster state:
[2025-06-09T12:18:01,953][WARN ][stderr                   ] cluster uuid: 9MwEhVdDR768Po-hesCVNw [committed: true]
[2025-06-09T12:18:01,953][WARN ][stderr                   ] version: 22
[2025-06-09T12:18:01,953][WARN ][stderr                   ] state uuid: 6sEHBubqQqep4O__5eWOeA
[2025-06-09T12:18:01,953][WARN ][stderr                   ] from_diff: false
[2025-06-09T12:18:01,953][WARN ][stderr                   ] meta data version: 6
[2025-06-09T12:18:01,953][WARN ][stderr                   ]    coordination_metadata:
[2025-06-09T12:18:01,954][WARN ][stderr                   ]       term: 4
[2025-06-09T12:18:01,954][WARN ][stderr                   ]       last_committed_config: VotingConfiguration{5Ex_37lWRqiSwyIt3MAhDQ}
[2025-06-09T12:18:01,954][WARN ][stderr                   ]       last_accepted_config: VotingConfiguration{5Ex_37lWRqiSwyIt3MAhDQ}
[2025-06-09T12:18:01,954][WARN ][stderr                   ]       voting tombstones: []
[2025-06-09T12:18:01,954][WARN ][stderr                   ]    [photon/M-E3ihQCQ3ycHOQ34sL63g]: v[19], mv[2], sv[3], av[1]
[2025-06-09T12:18:01,954][WARN ][stderr                   ]       0: p_term [4], isa_ids [jSGpB7fuR-adKzpPpp9YqA]
[2025-06-09T12:18:01,955][WARN ][stderr                   ]       1: p_term [4], isa_ids [uPEKeSD1QTaZHESRGPfi_Q]
[2025-06-09T12:18:01,956][WARN ][stderr                   ]       2: p_term [4], isa_ids [K_Et7CxXRNOHK76GoG9_hw]
[2025-06-09T12:18:01,956][WARN ][stderr                   ]       3: p_term [3], isa_ids [AtrAedK3Tva3oXdaix_vuQ]
[2025-06-09T12:18:01,956][WARN ][stderr                   ]       4: p_term [2], isa_ids [1dZaB5VeRDCluCLnR-CG7w]
[2025-06-09T12:18:01,956][WARN ][stderr                   ] metadata customs:
[2025-06-09T12:18:01,956][WARN ][stderr                   ]    index-graveyard: IndexGraveyard[[]]
[2025-06-09T12:18:01,956][WARN ][stderr                   ] nodes:
[2025-06-09T12:18:01,956][WARN ][stderr                   ]    {Node 1}{5Ex_37lWRqiSwyIt3MAhDQ}{hkN5oN2oThy2sIBFPz0j_g}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}, local, cluster-manager
[2025-06-09T12:18:01,956][WARN ][stderr                   ] routing_table (version 3):
[2025-06-09T12:18:01,957][WARN ][stderr                   ] -- index [[photon/M-E3ihQCQ3ycHOQ34sL63g]]
[2025-06-09T12:18:01,957][WARN ][stderr                   ] ----shard_id [photon][0]
[2025-06-09T12:18:01,957][WARN ][stderr                   ] --------[photon][0], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=jSGpB7fuR-adKzpPpp9YqA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,957][WARN ][stderr                   ] --------[photon][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,957][WARN ][stderr                   ] ----shard_id [photon][1]
[2025-06-09T12:18:01,957][WARN ][stderr                   ] --------[photon][1], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=uPEKeSD1QTaZHESRGPfi_Q], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,958][WARN ][stderr                   ] --------[photon][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,958][WARN ][stderr                   ] ----shard_id [photon][2]
[2025-06-09T12:18:01,958][WARN ][stderr                   ] --------[photon][2], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=K_Et7CxXRNOHK76GoG9_hw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,958][WARN ][stderr                   ] --------[photon][2], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,959][WARN ][stderr                   ] ----shard_id [photon][3]
[2025-06-09T12:18:01,959][WARN ][stderr                   ] --------[photon][3], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=AtrAedK3Tva3oXdaix_vuQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,959][WARN ][stderr                   ] --------[photon][3], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,959][WARN ][stderr                   ] ----shard_id [photon][4]
[2025-06-09T12:18:01,959][WARN ][stderr                   ] --------[photon][4], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[deciders_throttled]]
[2025-06-09T12:18:01,960][WARN ][stderr                   ] --------[photon][4], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,960][WARN ][stderr                   ] routing_nodes:
[2025-06-09T12:18:01,960][WARN ][stderr                   ] -----node_id[5Ex_37lWRqiSwyIt3MAhDQ][V]
[2025-06-09T12:18:01,961][WARN ][stderr                   ] --------[photon][0], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=jSGpB7fuR-adKzpPpp9YqA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,962][WARN ][stderr                   ] --------[photon][1], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=uPEKeSD1QTaZHESRGPfi_Q], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,963][WARN ][stderr                   ] --------[photon][2], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=K_Et7CxXRNOHK76GoG9_hw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,963][WARN ][stderr                   ] --------[photon][3], node[5Ex_37lWRqiSwyIt3MAhDQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=AtrAedK3Tva3oXdaix_vuQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[fetching_shard_data]]
[2025-06-09T12:18:01,964][WARN ][stderr                   ] ---- unassigned
[2025-06-09T12:18:01,965][WARN ][stderr                   ] --------[photon][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,966][WARN ][stderr                   ] --------[photon][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,970][WARN ][stderr                   ] --------[photon][2], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,972][WARN ][stderr                   ] --------[photon][3], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,974][WARN ][stderr                   ] --------[photon][4], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[deciders_throttled]]
[2025-06-09T12:18:01,975][WARN ][stderr                   ] --------[photon][4], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-06-09T12:17:32.054Z], delayed=false, allocation_status[no_attempt]]
[2025-06-09T12:18:01,976][WARN ][stderr                   ] tasks: (0):
[2025-06-09T12:18:01,979][WARN ][stderr                   ] 	at org.codelibs.opensearch.runner.OpenSearchRunner.onFailure(OpenSearchRunner.java:1101)
[2025-06-09T12:18:01,983][WARN ][stderr                   ] 	at org.codelibs.opensearch.runner.OpenSearchRunner.ensureYellow(OpenSearchRunner.java:758)
[2025-06-09T12:18:01,983][WARN ][stderr                   ] 	at de.komoot.photon.Server.startInternal(Server.java:88)
[2025-06-09T12:18:01,983][WARN ][stderr                   ] 	at de.komoot.photon.Server.start(Server.java:48)
[2025-06-09T12:18:01,983][WARN ][stderr                   ] 	at de.komoot.photon.App.runPhoton(App.java:66)
[2025-06-09T12:18:01,983][WARN ][stderr                   ] 	at de.komoot.photon.App.main(App.java:37)

[rouke-broersma]

I did another test with a country-specific database and this managed to start up. Could it be a resource constraint issue? If so how can I investigate? The container seems to only claim about 4Gib memory and I have more available if necessary.

[rtuszik]

I think that this may either be a resource constraint or permissions issue. What kind of disk speeds are you getting with ceph?

It seems like these issues often happen with "external" storage. Photon likes NVME. But I haven't been able to replicate this on a low end machine either, so I am not entirely sure. Given that you have plenty of resources, I assume that ram and cpu are not the problem here.

[emilghen]

I had the same issue when trying to have the storage on an NFS attached drive. I moved the container to my NAS running directly from the drive and it was solved.

[rtuszik]

I have implemented a feature that allows setting the PUID and PGID manually using the environment variables. This should solve any permission issues with NFS. If you are willing to test, I have pushed it to rtuszik/photon-docker:dev.

You can get the PUID and PGID for your host using id -u and id -g, respectively.

Edit: Doesn't work right now. Working on a fix.

[rsteckler]

This is happening to me running inside docker. It was working previously.
Everything is running as root inside the container:

UID          PID    PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root           1       0  0  2006  4000  18 13:58 ?        00:00:00 /bin/bash /photon/start-photon.sh
root          41       1  0  1427  1920  14 13:58 ?        00:00:00 sleep 86400
root         124       0  0   707  1760   7 14:06 pts/0    00:00:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c "/bin/bash" /dev/null || exec /bin/bash) || exec /bin/sh
root         130     124  0   707   804  16 14:06 pts/0    00:00:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c "/bin/bash" /dev/null || exec /bin/bash) || exec /bin/sh
root         131     130  0  1432  2080  10 14:06 pts/0    00:00:00 /usr/bin/script -q -c /bin/bash /dev/null
root         132     131  0   706  1600  15 14:06 pts/1    00:00:00 sh -c /bin/bash
root         133     132  0  1973  3840  14 14:06 pts/1    00:00:00 /bin/bash
root         144       0  0   707  1600   6 14:29 pts/2    00:00:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c "/bin/bash" /dev/null || exec /bin/bash) || exec /bin/sh
root         151     144  0   707   480  16 14:29 pts/2    00:00:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c "/bin/bash" /dev/null || exec /bin/bash) || exec /bin/sh
root         152     151  0  1432  1920  17 14:29 pts/2    00:00:00 /usr/bin/script -q -c /bin/bash /dev/null
root         153     152  0   706  1600  14 14:29 pts/3    00:00:00 sh -c /bin/bash
root         154     153  0  1970  4480  18 14:29 pts/3    00:00:00 /bin/bash
root         175     154  0  2791  4640   2 14:44 pts/3    00:00:00 ps -AF

Here are the log files and photon_data files:
komoot/photon#925

I'm not sure what to check next.

[lonvia]

This really looks more and more like a slow disk problem. Have you considered running a composite docker setup with an OpenSearch instance in its own container and photon connecting to it externally using the --transport-address parameter? Then hopefully Docker would take care of waiting for the OpenSearch container to be fully ready before spinning up photon. If you do try that, please be aware that photon can only connect using http, while OpenSearch defaults to using https.

I've looked into the OpenSearch startup procedure trying to detect if we are just waiting for slow disks but it is really tricky. Not to mention that I haven't been able to replicate this locally either.

[rsteckler]

Will give it a shot. Could you give me a little guidance on how that works? I'm assuming:

1. Setup an OpenSearch docker container and expose the service with it. This is where the large volume (200+ gigs) needs to attach?
2. Point the photon container to that OS service via PHOTON_PARAMS and --transport-addresses
3. Wipe the old photon_data directory?
4. Restart everything and photon should start downloading the data and storing it in OpenSearch

I can report back and see if that helps. I don't think it's solely a slow disk though - this is running a complicated storage setup, but it's very fast (12-disk ZFS RAID exposing a dedicated disk to Photon via iSCSI over 10gbps network).

[rtuszik]

From what I can tell, there are two separate issues that come up time and time again.

There are issues with remote storage locations due to root squashing, which I am working on solving (#82), and there is definitely a resource constraint issue. (#95)
In the case I was able to pin down as disk speed issue, the index did end up loading, but then failed on requests.

In this case I can see it go either way.

I have previously tried to setup the separate ElasticSearch instance with an older version but had some issues getting that to work, but I'll try to get that running at some point. I'll see if I can integrate that with my rewrite (#50).

I will also work on a troubleshooting section that may help with triage.

[rtuszik]

I can report back and see if that helps. I don't think it's solely a slow disk though - this is running a complicated storage setup, but it's very fast (12-disk ZFS RAID exposing a dedicated disk to Photon via iSCSI over 10gbps network).

iSCSI doesn't rood squash, so its not a permissions issue (or at least not one I have been able to replicate so far).

1. Setup an OpenSearch docker container and expose the service with it.  This is where the large volume (200+ gigs) needs to attach?

2. Point the photon container to that OS service via PHOTON_PARAMS and --transport-addresses

3. Wipe the old photon_data directory?

4. Restart everything and photon should start downloading the data and storing it in OpenSearch

You will have to create the OpenSearch instance yourself and also download the index manually. Then you would spin up the OpenSearch instance, wait for it to be ready and then start photon without download functionality, using the transport-address. Correct me if im wrong here @lonvia.

[rsteckler]

Thanks for the help. I got opensearch setup and now downloading the photon data from graphopper/experimental. Will be ~8 hours so will report back tomorrow