CUDA failed with error an illegal memory access was encountered

[RoboMWM]

edit-mind-ml               |     result = await self.transcription_service.process_async(
edit-mind-ml               |   File "/app/python/services/base_service.py", line 69, in process_async
edit-mind-ml               |     result = await loop.run_in_executor(
edit-mind-ml               |   File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
edit-mind-ml               |     result = self.fn(*self.args, **self.kwargs)
edit-mind-ml               |   File "/app/python/services/transcription/service.py", line 61, in _process_sync
edit-mind-ml               |     raise TranscriptionError(f"Transcription failed: {e}")
edit-mind-ml               | core.errors.TranscriptionError: Transcription failed: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               | 2026-02-07 12:52:58,608 - INFO - Finished transcription: /media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 19.10.53.18.DVR_output_stripped..mp4 (active: 0)
edit-mind-ml               | 2026-02-07 12:52:58,914 - INFO - Started transcription: /media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.58.13.17.DVR_output_stripped..mp4 (active: 1)
edit-mind-ml               | 2026-02-07 12:52:58,917 - INFO - Starting transcription: /media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.58.13.17.DVR_output_stripped..mp4
edit-mind-ml               | 2026-02-07 12:53:01,212 - ERROR - Transcription failed: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               | 2026-02-07 12:53:01,212 - ERROR - Transcription error: Transcription failed: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               | Traceback (most recent call last):
edit-mind-ml               |   File "/app/python/services/transcription/service.py", line 44, in _process_sync
edit-mind-ml               |     result = self._transcribe_video(
edit-mind-ml               |   File "/app/python/services/transcription/service.py", line 73, in _transcribe_video
edit-mind-ml               |     segments, info = model.transcribe(
edit-mind-ml               |   File "/app/.venv/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 942, in transcribe
edit-mind-background-jobs  | {"level":50,"time":1770468781263,"pid":18,"hostname":"6573ccca44f4","jobId":"GNmuB9iPwRC4c3jdsSkk3","videoPath":"/media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.58.13.17.DVR_output_stripped..mp4","error":{},"stack":"Error: Transcription failed: CUDA failed with error an illegal memory access was encountered\n    at WebSocket.<anonymous> (/app/packages/shared/dist/services/pythonService.js:185:35)\n    at WebSocket.emit (node:events:519:28)\n    at Receiver.receiverOnMessage (/app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/websocket.js:1220:20)\n    at Receiver.emit (node:events:519:28)\n    at Receiver.dataMessage (/app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/receiver.js:596:14)\n    at /app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/receiver.js:530:12\n    at /app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/permessage-deflate.js:309:9\n    at /app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/permessage-deflate.js:392:7\n    at afterWrite (node:internal/streams/writable:710:5)\n    at onwrite (node:internal/streams/writable:688:7)\n    at Zlib.cb (node:internal/streams/transform:191:7)\n    at Zlib.processCallback (node:zlib:607:8)","msg":"Error during transcription"}
edit-mind-ml               |     ) = self.detect_language(
edit-mind-ml               |   File "/app/.venv/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 1819, in detect_language
edit-mind-ml               |     encoder_output = self.encode(
edit-mind-ml               |   File "/app/.venv/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 1400, in encode
edit-mind-ml               |     return self.model.encode(features, to_cpu=to_cpu)
edit-mind-ml               | RuntimeError: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               |
edit-mind-ml               | During handling of the above exception, another exception occurred:
edit-mind-ml               |
edit-mind-ml               | Traceback (most recent call last):
edit-mind-ml               |   File "/app/python/services/websocket/handlers.py", line 190, in handle_transcribe
edit-mind-ml               |     result = await self.transcription_service.process_async(
edit-mind-ml               |   File "/app/python/services/base_service.py", line 69, in process_async
edit-mind-ml               |     result = await loop.run_in_executor(
edit-mind-ml               |   File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
edit-mind-ml               |     result = self.fn(*self.args, **self.kwargs)
edit-mind-ml               |   File "/app/python/services/transcription/service.py", line 61, in _process_sync
edit-mind-ml               |     raise TranscriptionError(f"Transcription failed: {e}")
edit-mind-ml               | core.errors.TranscriptionError: Transcription failed: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               | 2026-02-07 12:53:01,216 - INFO - Finished transcription: /media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.58.13.17.DVR_output_stripped..mp4 (active: 0)
edit-mind-ml               | 2026-02-07 12:53:01,372 - INFO - Started transcription: /media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.35.42.16.DVR_output_stripped..mp4 (active: 1)
edit-mind-ml               | 2026-02-07 12:53:01,397 - INFO - Starting transcription: /media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.35.42.16.DVR_output_stripped..mp4
edit-mind-web              | 2026-02-07T12:53:01.931Z - GET /
edit-mind-web              | GET / 200 - - 14.736 ms
edit-mind-background-jobs  | {"level":30,"time":1770468783916,"pid":18,"hostname":"6573ccca44f4","msg":"Extracted 5 frames from scene (17.999s - 18.999s)"}
edit-mind-ml               | 2026-02-07 12:53:04,302 - ERROR - Transcription failed: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               | 2026-02-07 12:53:04,302 - ERROR - Transcription error: Transcription failed: CUDA failed with error an illegal memory access was encountered
edit-mind-ml               | Traceback (most recent call last):
edit-mind-background-jobs  | {"level":50,"time":1770468784327,"pid":18,"hostname":"6573ccca44f4","jobId":"HqFenM4ZZxkQP7elTWzHL","videoPath":"/media/videos/NVIDIA/Marvel Rivals/Marvel Rivals 2025.01.14 - 18.35.42.16.DVR_output_stripped..mp4","error":{},"stack":"Error: Transcription failed: CUDA failed with error an illegal memory access was encountered\n    at WebSocket.<anonymous> (/app/packages/shared/dist/services/pythonService.js:185:35)\n    at WebSocket.emit (node:events:519:28)\n    at Receiver.receiverOnMessage (/app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/websocket.js:1220:20)\n    at Receiver.emit (node:events:519:28)\n    at Receiver.dataMessage (/app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/receiver.js:596:14)\n    at /app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/receiver.js:530:12\n    at /app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/permessage-deflate.js:309:9\n    at /app/node_modules/.pnpm/ws@8.18.3/node_modules/ws/lib/permessage-deflate.js:392:7\n    at afterWrite (node:internal/streams/writable:710:5)\n    at onwrite (node:internal/streams/writable:688:7)\n    at Zlib.cb (node:internal/streams/transform:191:7)\n    at Zlib.processCallback (node:zlib:607:8)","msg":"Error during transcription"}
edit-mind-ml               |   File "/app/python/services/transcription/service.py", line 44, in _process_sync
edit-mind-ml               |     result = self._transcribe_video(
edit-mind-ml               |   File "/app/python/services/transcription/service.py", line 73, in _transcribe_video
edit-mind-ml               |     segments, info = model.transcribe(
edit-mind-ml               |   File "/app/.venv/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 942, in transcribe
edit-mind-ml               |     ) = self.detect_language(
edit-mind-ml               |   File "/app/.venv/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 1819, in detect_language
edit-mind-ml               |     encoder_output = self.encode(
edit-mind-ml               |   File "/app/.venv/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 1400, in encode
edit-mind-ml               |     return self.model.encode(features, to_cpu=to_cpu)
edit-mind-ml               | RuntimeError: CUDA failed with error an illegal memory access was encountered

Not sure if this was occurring before or after I woke the PC from sleep, but seems you may be able to mitigate this by adding torch.cuda.empty_cache() when done with using the GPU after a task. https://stackoverflow.com/questions/68106457/pytorch-cuda-error-an-illegal-memory-access-was-encountered

I tried to check if other videos that were reported as processed had a transcription or not, but I don't see where to read that in the UI currently?

[RoboMWM]

I tried to check if other videos that were reported as processed had a transcription or not, but I don't see where to read that in the UI currently?

Seems the only way right now is via enabling captions when watching a video I think. Would be nice for the search to include searching transcriptions as well as the scene descriptions; probably should be a new issue as a feature request unless I'm missing something.

[IliasHad]

Seems the only way right now is via enabling captions when watching a video I think. Would be nice for the search to include searching transcriptions as well as the scene descriptions; probably should be a new issue as a feature request unless I'm missing something.

Yes, you can check from the scene description where you see transcription ("The transcription reads"). I have search via transcription, but it missed that there's a bug with the search (https://github.com/IliasHad/edit-mind/blob/8ec3b49b9b3d04ab658ab5010431a8c2f3f18bba/packages/search/src/utils/filters.ts).

[RoboMWM]

The scene description I see is for each analyzed image, and I don't see anything that mentions "The transcription reads," since it's describing just the image of that extracted frame.

[IliasHad]

@RoboMWM, but if you click on the captions button, you can see the transcription, right?

[RoboMWM]

Yes I do see the transcription in the form of captions yes, as the video plays.

As for this original issue, I think this may have been caused by my GPU crashing (which apparently is happening when I wake the monitor while GPU is under load), so the suggested mitigation may not be necessary.

[RoboMWM]

"The transcription reads"

Oh I see this now, it has to be at those scenes. I thought there'd be a transcription for the entire video printed somewhere.

[RoboMWM]

I'll go ahead and close this for now, and either reopen or make a more focused issue if it occurs again, as it seems the cause is due to the gpu crashing.

Or you could reopen if you want to add the ability for it to recover from a gpu crash.

[IliasHad]

Oh I see this now, it has to be at those scenes. I thought there'd be a transcription for the entire video printed somewhere.

Yes, exactly over the active scene card below the video player.

[IliasHad]

@RoboMWM, I'll keep this issue open until we have a recovery option for the GPU crash over the ML service