Skip to content

TimeOut error #474

Description

@yannachen

This is the timeout error during measuring Sn at 29200 eV +-30 eV, with step size 1 ev and 2 eV:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/queues.py", line 186, in get
    await getter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 498, in observe_signals_value
    item = await asyncio.wait_for(q.get(), iteration_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 37, in wait_with_error_message
    await awaitable
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 205, in _notify_watchers_from
    async for update in iterator:
    ...<6 lines>...
            self._update_watcher(watcher, self._last_update)
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/devices/undulator.py", line 289, in set
    async for signal, value in observe_signals_value(
    ...<24 lines>...
            break
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 500, in observe_signals_value
    raise TimeoutError(
    ...<3 lines>...
    ) from exc
TimeoutError: Timeout Error while waiting 11.661087036132812s to update ['derived://undulator-energy', 'ca://S25ID:DSID:BusyDeviceM.VAL']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff329881750>, 29184.338912963867)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 296, in _execute_plan
    result = func()
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 486, in start_plan_func
    return self._RE(plan_func(*plan_args, **plan_kwargs), {"all": [self._run_reg_cb]}, **plan_meta)
           ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 976, in __call__
    plan_return = self._resume_task(init_func=_build_task)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1121, in _resume_task
    raise exc
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1756, in _run
    raise err
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1610, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/aps_suspenders.py", line 122, in install_suspenders
    yield from plan
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 88, in open_shutters_wrapper
    return_val = yield from new_plan
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 159, in open_on_kickoff_wrapper
    response = yield msg
               ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  [Previous line repeated 5 more times]
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 2378, in _status_object_completed
    raise FailedStatus(ret) from exc
bluesky.utils.FailedStatus: <WatchableAsyncStatus, device: undulator-energy, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7ff4c092a3e0>, errored: TimeoutError("Timeout Error while waiting 11.661087036132812s to update ['derived://undulator-energy', 'ca://S25ID:DSID:BusyDeviceM.VAL']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7ff329881750>, 29184.338912963867)")>

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions