Summary
After the frontend auth handshake completes successfully, the first SQL_STMT_EXECUTE (a plain SELECT 1) never produces a response frame back to the client. Surfaced while bringing the mysqlx-soak-g1 TAP group green — three of the soak tests (test_mysqlx_route_drop_inflight-t, test_mysqlx_soak_behavioral-t, test_mysqlx_soak_stress-t) all fail past the handshake stage for this reason.
Distinct from the auth/connect-surface bugs fixed in PR #5852 (commit b8a710c1f: MYSQL41 AuthContinue parser accepting standard X-Protocol shapes, start_connect adding DNS resolution, mapped-mode backend password fallback). Those three were necessary for the client to reach the SQL stage at all; this is what blocks it from completing.
Observed symptoms
Two failure shapes, possibly the same bug surfaced by different client timings:
Shape A — mysql-connector-python (mysqlx-soak-g1 behavioral/stress harnesses):
mysqlx.get_session(...) returns successfully (frontend auth OK)- proxysql log shows backend
start_connect returning rc=1 EINPROGRESS, then check_connect rc=0 (TCP connect completes)
s.sql(\"SELECT 1\").execute() then hangs indefinitely- No further plugin log lines after the backend connect completes
Shape B — raw TAP client (test_mysqlx_route_drop_inflight-t):
- Stage 1: 5/5 handshakes succeed (clients reach AuthOK)
- Stage 2:
exec_select_1 returns false on all 5 fds within ~160ms each → pre-drop SELECT 1 succeeded on 0/5 sessions
- proxysql log shows the 5 handshakes — and then nothing. No
start_connect lines fire for any of the 5 sessions during Stage 2, even though forward_to_backend should have triggered the lazy backend connection.
Hypotheses (not yet confirmed)
State machine transition after BACKEND_AUTH_DONE (see handler_connecting_server() in plugins/mysqlx/src/mysqlx_session.cpp, around the tail that sets status_ = WAITING_CLIENT_XMSG; to_process = true;). Candidates:
- The next handler tick isn't re-entering
handler_waiting_client_msg / dispatch_client_message, so the buffered StmtExecute frame is never popped.
forward_to_backend is entered but server_ds().write_to_net() doesn't actually write to the backend fd (e.g. server_ds_ not properly bound to backend_conn_->get_fd() after fresh auth).- The backend response side (
handler_waiting_server_msg) is never invoked because server_ds_ isn't subscribed for read events.
- The session is being marked
healthy = false somewhere on the AuthOK→first-query transition and closed without logging.
Shape A and Shape B may diverge on which hypothesis applies — A clearly gets to BACKEND_AUTH_DONE; B may not even get the SELECT 1 into forward_to_backend (the absence of start_connect logs in Shape B is the most surprising data point).
Diagnostic plan
Targeted proxy_info traces at:
dispatch_client_message entry: log msg_type, status_, backend_conn_ != nullptrforward_to_backend entry/exit: log server_ds().get_status(), backend_conn_->get_state(), whether the frame was enqueuedhandler_waiting_server_msg entry: log how often it's called and whether server_ds_ has any data- Every
healthy = false assignment post-send_auth_ok() (grep for the assignments and add a trace)
Plus tcpdump -i any -w /tmp/cap.pcap port 6603 or port 13306 from the test-runner container to confirm whether the SELECT 1 frame ever leaves ProxySQL toward the backend, and whether the backend ever responds.
Reproduction (local)
# In a worktree of v3.0 with PR #5852 merged:
make debug && PROXYSQL40=1 PROXYSQLGENAI=1 OPTZ=\"-O0 -ggdb -DDEBUG\" make -C plugins/mysqlx
WORKSPACE=\$(pwd) INFRA_ID=dev-soak-\$USER TAP_GROUP=mysqlx-soak-g1 test/infra/control/ensure-infras.bash
WORKSPACE=\$(pwd) INFRA_ID=dev-soak-\$USER TAP_GROUP=mysqlx-soak-g1 test/infra/control/run-tests-isolated.bash
# Or just Shape A:
docker run --rm --network dev-soak-\${USER}_backend python:3.11-slim bash -c \\
'pip install mysqlx-connector-python -q && python3 -c \"
import mysqlx
s = mysqlx.get_session({\\\"host\\\":\\\"proxysql.dev-soak-${USER}\\\",\\\"port\\\":6603,
\\\"user\\\":\\\"alice\\\",\\\"password\\\":\\\"alicepass\\\",
\\\"ssl-mode\\\":\\\"DISABLED\\\",\\\"compression\\\":\\\"disabled\\\",\\\"auth\\\":\\\"MYSQL41\\\"})
print(\\\"OPEN\\\")
print(s.sql(\\\"SELECT 1\\\").execute().fetch_all()) # hangs here
\"'Direct backend (bypassing ProxySQL, same alice creds, port 13306) works end-to-end, so the backend itself is fine.
Related
Summary
After the frontend auth handshake completes successfully, the first
SQL_STMT_EXECUTE(a plainSELECT 1) never produces a response frame back to the client. Surfaced while bringing themysqlx-soak-g1TAP group green — three of the soak tests (test_mysqlx_route_drop_inflight-t,test_mysqlx_soak_behavioral-t,test_mysqlx_soak_stress-t) all fail past the handshake stage for this reason.Distinct from the auth/connect-surface bugs fixed in PR #5852 (commit
b8a710c1f: MYSQL41 AuthContinue parser accepting standard X-Protocol shapes,start_connectadding DNS resolution,mapped-mode backend password fallback). Those three were necessary for the client to reach the SQL stage at all; this is what blocks it from completing.Observed symptoms
Two failure shapes, possibly the same bug surfaced by different client timings:
Shape A —
mysql-connector-python(mysqlx-soak-g1behavioral/stressharnesses):mysqlx.get_session(...)returns successfully (frontend auth OK)start_connectreturningrc=1 EINPROGRESS, thencheck_connect rc=0(TCP connect completes)s.sql(\"SELECT 1\").execute()then hangs indefinitelyShape B — raw TAP client (
test_mysqlx_route_drop_inflight-t):exec_select_1returnsfalseon all 5 fds within ~160ms each →pre-drop SELECT 1 succeeded on 0/5 sessionsstart_connectlines fire for any of the 5 sessions during Stage 2, even thoughforward_to_backendshould have triggered the lazy backend connection.Hypotheses (not yet confirmed)
State machine transition after
BACKEND_AUTH_DONE(seehandler_connecting_server()inplugins/mysqlx/src/mysqlx_session.cpp, around the tail that setsstatus_ = WAITING_CLIENT_XMSG; to_process = true;). Candidates:handler_waiting_client_msg/dispatch_client_message, so the buffered StmtExecute frame is never popped.forward_to_backendis entered butserver_ds().write_to_net()doesn't actually write to the backend fd (e.g.server_ds_not properly bound tobackend_conn_->get_fd()after fresh auth).handler_waiting_server_msg) is never invoked becauseserver_ds_isn't subscribed for read events.healthy = falsesomewhere on the AuthOK→first-query transition and closed without logging.Shape A and Shape B may diverge on which hypothesis applies — A clearly gets to
BACKEND_AUTH_DONE; B may not even get the SELECT 1 intoforward_to_backend(the absence ofstart_connectlogs in Shape B is the most surprising data point).Diagnostic plan
Targeted
proxy_infotraces at:dispatch_client_messageentry: logmsg_type,status_,backend_conn_ != nullptrforward_to_backendentry/exit: logserver_ds().get_status(),backend_conn_->get_state(), whether the frame was enqueuedhandler_waiting_server_msgentry: log how often it's called and whetherserver_ds_has any datahealthy = falseassignment post-send_auth_ok()(grep for the assignments and add a trace)Plus
tcpdump -i any -w /tmp/cap.pcap port 6603 or port 13306from the test-runner container to confirm whether the SELECT 1 frame ever leaves ProxySQL toward the backend, and whether the backend ever responds.Reproduction (local)
Direct backend (bypassing ProxySQL, same alice creds, port 13306) works end-to-end, so the backend itself is fine.
Related
behavioral_validation.py(blocked by this)stress.py(blocked by this)