Skip to content

bug: GraphQL subscription schema mismatch breaks bot startups #314

@flexus-teams

Description

@flexus-teams

Original Logs

20260414 11:09:39.395 fclnt [INFO] FlexusClient service_name=karen_100020189_r_ api_key=None http://backend-v1-service.flexus.svc/v1/jailed-bot
20260414 11:09:39.396 stexe [INFO] Connecting ws://backend-v1-service.flexus.svc/v1/jailed-bot
20260414 11:09:39.396 btexe [INFO] Subscribing to ERP tables: ['crm_contact', 'crm_activity', 'crm_deal', 'com_shop', 'com_product', 'com_product_variant', 'com_order', 'com_order_item', 'com_refund']
20260414 11:09:39.451 stexe [INFO] got TransportQueryError (attempt 1/3), sleep 60...
20260414 11:11:39.549 stexe [ERROR] 🛑 3 exceptions in 5 min, exiting
Traceback (most recent call last):
  File "/app/flexus_client_kit/ckit_service_exec.py", line 26, in run_typical_single_subscription_with_restart_on_network_errors
    await subscribe_and_do_something(fclient, ws_client, *func_args, **func_kwargs)
  File "/app/flexus_client_kit/ckit_bot_exec.py", line 412, in subscribe_and_produce_callbacks
    async for r in ws.subscribe(
  File "/usr/local/lib/python3.11/site-packages/gql/client.py", line 1426, in subscribe
    async for result in inner_generator:
  File "/usr/local/lib/python3.11/site-packages/gql/client.py", line 1337, in _subscribe
    async for result in inner_generator:
  File "/usr/local/lib/python3.11/site-packages/gql/transport/common/base.py", line 298, in subscribe
    answer_type, execution_result = await listener.get()
                                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gql/transport/common/listener_queue.py", line 35, in get
    raise item
  File "/usr/local/lib/python3.11/site-packages/gql/transport/common/base.py", line 221, in _receive_data_loop
    answer_type, answer_id, execution_result = self._parse_answer(
                                               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gql/transport/websockets_protocol.py", line 423, in _parse_answer
    return self._parse_answer_apollo(json_answer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gql/transport/websockets_protocol.py", line 384, in _parse_answer_apollo
    raise TransportQueryError(

gql.transport.exceptions.TransportQueryError: {'message': "Cannot query field 'news_payload_task' on type 'FBotThreadsCallsTasks'. Did you mean 'news_payload_auth', 'news_payload_id', 'news_payload_task_new', 'news_payload_task_old', or 'news_payload_thread'?", 'locations': [{'line': 96, 'column': 5}]}

Error Summary

Bot pod flexus-pod-bot-karen-100020189-rx in namespace isolated crashloops on every startup because the backend GraphQL schema no longer has news_payload_task on FBotThreadsCallsTasks.

Stacktrace

TransportQueryError: Cannot query field 'news_payload_task' on type 'FBotThreadsCallsTasks'

Root Cause

  • File: flexus_client_kit/ckit_bot_exec.py:412
  • Function: subscribe_and_produce_callbacks
  • Why: the client subscription still asks for a removed field name, while the backend exposes news_payload_task_new, news_payload_task_old, etc.
  • Git blame: see gql_utils.py / subscription helpers; schema mismatch is in the client kit path

Code Snippet

# from ckit_service_exec.py
except (
    gql.transport.exceptions.TransportError,
    asyncio.exceptions.TimeoutError,
) as e:
    ...
    if len(exception_times) >= 3:
        logger.exception("3 exceptions in 5 min, exiting")
        raise

Affected

  • Pods: flexus-pod-bot-karen-100020189-rx
  • Namespaces: isolated
  • Occurrences: 1 crashing pod, repeated on every restart

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions