Move the IoT Edge runtime and daemon to the 1.6 LTS line

[Ewerton Scaboro da Silva (ewertons)]

What

Move the IoT Edge release line from 1.5 to 1.6. Three defaults:

File	Setting
bin/deploy/edge_configuration.py	EDGE_RELEASE_LINE — azureiotedge-agent/-hub image tags
scripts/new/install-iotedge.sh	aziot_edge_line — host aziot-edge/aziot-identity-service packages
test-runner/README.md	documents the default

IOTHUB_E2E_EDGE_RELEASE_LINE overrides all three together, so images and daemon cannot drift.

Why now

1.6 is the current LTS, serviced through 2028-11-14. 1.5 is the previous LTS.

This PR was blocked until now: edgeHub 1.6.0/1.6.1 could not complete a TLS handshake on the MQTT head, so every edgehub_module job failed. That was an IoT Edge defect — the 1.6 images resolved DotNetty.Handlers/Codecs/Codecs.Mqtt down to 0.6.0 (October 2018) instead of 0.7.6, because the .NET 10 + device SDK update removed the transitive reference that had been raising them. The details are in the comments below.

Fixed in edgeHub 1.6.2 (2026-08-26), which pins those packages back to 0.7.6. The :1.6 tag now resolves to it.

Gate result

Full gate on this branch, rebased on current master, is green: 20/20 jobs, including 6/6 edgehub_module — the jobs that were 0/6 on 1.6.1.

Daemon and images match, both on the 1.6 line:

Installing aziot-identity-service=1.6.0-1 and aziot-edge=1.6.0-1
iotedge 1.6.0

The daemon stays at 1.6.0 — 1.6.2 updated only the module images.

Blast radius

Changes edgeAgent/edgeHub and the host daemon for every language gate — Java, Python, Node, C, C#. Nothing else pins the version: no pipeline definition and no consuming repo's horton-e2e.yaml sets IOTHUB_E2E_EDGE_RELEASE_LINE or the image overrides. A gate needing to stay on 1.5 can set IOTHUB_E2E_EDGE_RELEASE_LINE=1.5 in its own pipeline without reverting this.

Validation

Local, before the gate:

• bash -n scripts/new/install-iotedge.sh clean; python -m py_compile bin/deploy/edge_configuration.py clean.
• Loaded edge_configuration.py and asserted behaviour: default EDGE_RELEASE_LINE is 1.6; _verify_image_on_release_line accepts azureiotedge-hub:1.6 and :1.6.2 and rejects :1.5.
• latest_version_on_line awk exercised against real apt-cache madison output: 1.6 → 1.6.0-1, 1.5 → 1.5.21-1, a line with no packages → hard error rather than a silent fall through to latest.
• Confirmed aziot-edge=1.6.0-1 and aziot-identity-service=1.6.0-1 are published for ubuntu-22.04 (jammy) and ubuntu-24.04 (noble), the only Linux agent images these pipelines use.

Then the full gate, as above.

[Ewerton Scaboro da Silva (ewertons)]

Gate result: this does not work. Do not merge.

I was wrong about #435's blocker, and the gate proved it.

Controlled A/B

Same repo content, three lines apart, on the same gate pipeline:

Branch	Line	edgehub_module jobs
master @ a71dd0b	1.5	6 / 6 green
this PR @ 5344b14	1.6	0 / 6 green

On 1.6: 7/7 iothub jobs green (they deploy no edge runtime) and 0/6 edgehub_module jobs green. The split is exactly "does this job deploy edgeHub", which is the cleanest possible attribution to the version change.

The mechanical part of the change is fine

Installing aziot-identity-service=1.6.0-1 and aziot-edge=1.6.0-1
Unpacking aziot-edge (1.6.0-1) ...
iotedge 1.6.0

wait for iot edge containers, wait for testMod container and Wait for EdgeHub scope cache propagation all succeeded. So install-iotedge.sh and edge_configuration.py do what they are supposed to. The problem is edgeHub 1.6 itself.

What actually fails

C / C# / Node — the four friendMod tests from #435, unchanged:

test_method_call_invoked_on_friend
  RetryError on .../module/ModuleClient_1/connect2
  (Caused by ResponseError('too many 500 error responses'))
test_inputoutput_module_to_friend_routing
  ReadTimeoutError on .../module/ModuleClient_1/inputMessage/fromtestMod
  (read timeout=150)

Python is considerably worse — 16 failures per job, and not confined to friendMod. testMod fails connect2 too, and so does a leaf device:

test_method_call_invoked_on_leaf_device      .../device/DeviceClient_1/connect2
test_twin_reported_props                     .../module/moduleObject_29/connect2
test_send_telemetry_to_iothub[127K object]   .../module/moduleObject_35/connect2

So it is not one wrapper, one language, or one module identity. Clients broadly cannot establish connections to edgeHub 1.6 in this deployment.

Where my #439/#440 reasoning was right, and where it was wrong

Right: the wrapper no longer dies. Previously the follow-on failures were ConnectionRefused and RemoteDisconnected — the node process was gone, and two tests ERRORed in setup. Now those are replaced by ResponseError('too many 500 error responses') and ReadTimeoutError (read timeout=150): the wrapper is alive and answering, it just cannot connect. #439/#440 did remove the process-death cascade.

Wrong: I treated the crash as the whole story. It was not. With the crash fixed, a real connection failure against edgeHub 1.6 is still there — previously hidden behind the louder process death. #435's blocker is genuine.

Consequence

Staying on 1.5 is correct. 1.5 is a supported LTS (1.5.44), so there is no end-of-life exposure — that was resolved by #436 when it moved off 1.4. This PR is the next step, not an urgent one, and it should not merge until edgeHub 1.6 works here.

Marking as draft. Investigation continues; the python job is the better reproducer, since it fails more broadly and includes a leaf device.

[Ewerton Scaboro da Silva (ewertons)]

Correction and root cause direction

My previous comment implied the failure was TLS 1.3 specific. An interleaved re-run shows that was an artefact of test ordering. The finding is broader.

edgeHub 1.6's MQTT head on port 8883 mostly never completes a TLS handshake at all.

Interleaved control

Raw TLS + MQTT CONNECT probe against the deployed edgeHub, 40 attempts, client max TLS version alternating 1.2/1.3 attempt by attempt so any drift in server health hits both variants equally. A CONNACK of any return code counts as answered — the point is whether the server responds.

edgeHub	answered	latency
1.5.44	40 / 40	18–46 ms
1.6.1	1 / 40	the one success took 17,766 ms

Same probe, same pipeline, same VM image, same deployment manifest. The only difference is IOTHUB_E2E_EDGE_RELEASE_LINE. Confirmed from the container logs: Version - 1.5.44.176952487 vs Version - 1.6.1.176221263, both reporting Enabling SSL protocols: Tls12, Tls13. Reproduced across two runs.

Failure shape

edgeHub's DotNetty buffer-pool debug lines appear at the exact millisecond the client connects, so the connection is accepted and reaches the MQTT head:

23:49:51.324  friendMod   MqttBase uri: mqtts://<host>
23:49:51.417  edgeHub     [DBG] [DotNetty.Buffers.AbstractByteBuffer] -Dio.netty.buffer.bytebuf.checkAccessible: True
              ......... 60 s, edgeHub logs nothing about this client .........
23:50:51.395  friendMod   received 'close' from mqtt client, error supplied is: no error supplied
23:50:51.397  friendMod   UnauthorizedError: mqtt.js returned Failure on first connection (Not authorized)

Why it looked language-specific

friendMod is always the node wrapper (friendMod_language = "node" in bin/deploy/edge_deployment.py). Across three 1.6 runs it authenticated zero times:

$ grep -o "ClientAuthenticated, [^,\"]*" edgeHub.log | sort | uniq -c
     11 ClientAuthenticated, ..._iotedge/testMod        <- python, 55 minute run
      0 ClientAuthenticated, ..._iotedge/friendMod      <- node, never

Scope

Only the DotNetty MQTT head is affected. edgeHub's Kestrel HTTPS on 443 is unaffected — friendMod invokes direct methods over it in the same runs where its MQTT connection never completes.

SslProtocols=Tls12 is not a workaround: 4/20 and 2/20 answered, with handshakes up to 6.7 s.

Gate stays on 1.5. See the next comment for the root cause.

[Copilot]

Pull request overview

Moves the repo’s default IoT Edge “release line” setting from 1.5 to 1.6 so the default edgeAgent/edgeHub image tags and the host aziot-edge/aziot-identity-service package line stay aligned via IOTHUB_E2E_EDGE_RELEASE_LINE.

Changes:

• Update EDGE_RELEASE_LINE default from 1.5 → 1.6 for edgeAgent/edgeHub image tags.
• Update install-iotedge.sh default package line from 1.5 → 1.6.
• Update test-runner documentation to reflect the new default release line.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.

File	Description
bin/deploy/edge_configuration.py	Changes the default IoT Edge release line used for module image tags.
scripts/new/install-iotedge.sh	Changes the default IoT Edge release line used to select/pin host daemon packages.
test-runner/README.md	Updates documentation for the default IOTHUB_E2E_EDGE_RELEASE_LINE value.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Ewerton Scaboro da Silva (ewertons)]

Root cause: edgeHub 1.6 ships a 2018 build of DotNetty's TLS handler

The 1.6 images resolve DotNetty.Handlers down to 0.6.0 (October 2018), while keeping DotNetty.Buffers/Common/Transport at 0.7.6. DotNetty.Handlers contains TlsHandler, which terminates TLS for the MQTT head on port 8883. 1.5 ships 0.7.6 throughout.

Read from the running containers (docker exec edgeHub cat /app/*.deps.json):

package	edgeHub 1.5	edgeHub 1.6
DotNetty.Handlers	0.7.6	0.6.0
DotNetty.Codecs	0.7.6	0.6.0
DotNetty.Codecs.Mqtt	0.7.6	0.6.0
DotNetty.Buffers / Common / Transport	0.7.6	0.7.6
runtime (runtimeconfig.json)	net8.0	net10.0

DotNetty's own RELEASE_NOTES.md shows what 0.6.0 predates:

• 0.7.6 (2024-02-09) — "Fix TLS handshake for net8"
• 0.7.5 (2022-08-31) — "Fix infinite loop caused by TlsHandler"
• 0.7.3 (2022-07-28) — "Adopted TlsHandler for the case when Handshake completion callback is dispatched asynchronously to thread pool"
• 0.6.0 (2018-10-09) — no TLS fixes

So 1.6 runs a TLS handler that predates the fix explicitly made for net8, on net10.

How the downgrade happened

A transitive-resolution regression, not an intentional pin. Both versions are Transitive in packages.lock.json:

• edgeHub's MQTT head gets DotNetty via Microsoft.Azure.Devices.ProtocolGateway.Core 2.0.1, which declares DotNetty.Handlers: 0.6.0, DotNetty.Codecs.Mqtt: 0.6.0 — the floor.
• In 1.5, Microsoft.Azure.Devices.Client 1.36.10 also declared DotNetty.Handlers 0.7.6 / DotNetty.Codecs.Mqtt 0.7.6, so resolution lifted the whole graph to 0.7.6.
• 1.6 updated that SDK to 1.43.0-edgeLts, which replaced DotNetty with MQTTnet 5.1.0.1559 and dropped both dependencies.
• With nothing lifting them, resolution fell back to ProtocolGateway's 0.6.0. DotNetty.Transport stayed at 0.7.6 only because an explicit PackageReference Include="DotNetty.Transport" Version="0.7.6" was added to Microsoft.Azure.Devices.Edge.Hub.CloudProxy.csproj in 1.6 — Handlers and Codecs got no such pin.

Present in 1.6.0-rc.1, 1.6.0 and 1.6.1; absent in 1.5.43 and 1.5.44. That matches the observed history: the first 1.6 trial ran 1.6.0, before 1.6.1 existed, so the 1.6.1 base-image refresh is not involved.

Effect

Raw TLS + MQTT CONNECT probe against the deployed edgeHub, client max TLS version alternating 1.2/1.3 per attempt (a CONNACK of any return code counts as answered):

edgeHub	answered	latency
1.5 (Handlers 0.7.6, net8)	20 / 20	18–43 ms
1.6 (Handlers 0.6.0, net10)	1 / 20	the one success took 11,747 ms

Reproduced across three runs. Failure shape: TCP is accepted and reaches DotNetty — its buffer-pool debug lines appear at the exact millisecond — then nothing. No ServerHello, no CONNACK, and no log line naming the client even at RuntimeLogLevel=debug. Clients wait their full timeout; mqtt.js reports it as UnauthorizedError: Failure on first connection (Not authorized) at 60 s, which is what made this look like an auth problem.

Only the DotNetty MQTT head is affected. edgeHub's Kestrel HTTPS on 443 is unaffected — friendMod invokes direct methods over it in the same runs where its MQTT connection never completes. Consistent with a fault in the DotNetty TLS path rather than certificates, SNI, or configuration.

edge-hub's own MqttProtocolHead.cs is byte-identical between 1.5.44 and 1.6.1, which is why nothing in the product source diff points at this.

Why it looks language-specific

friendMod is always the node wrapper (friendMod_language = "node"). Across three 1.6 runs it authenticated zero times, while the python testMod got through 11 times in one 55-minute run. All six edgehub_module job failures trace back to stalled connections; the 7/7 iothub jobs pass because they deploy no edge runtime.

Secondary observation

On 1.6 arms the agent reached 98% memory (18 warnings in one run, 10 in another); the 1.5 arms running the identical probe produced zero. Consistent with stalled handshakes retaining buffers. Not required for the failure — the full gate run where all six edgehub_module jobs failed had no memory warnings at all. A per-container memory measurement is running to confirm whether edgeHub itself is the consumer.

Fix

For the product, in edge-hub: pin DotNetty.Handlers, DotNetty.Codecs and DotNetty.Codecs.Mqtt to 0.7.6, alongside the DotNetty.Transport pin that already exists, so the whole DotNetty graph is consistent and includes the net8+ TLS handshake fixes.

Confirming that 0.7.6 is sufficient on net10 requires building the edgeHub image, which needs the product repo's toolchain and has not been done. DotNetty 0.8.0 (2026-01-08) carries further TLS work — "Deferred processing of pending data packets to ensure upstream handlers receive TlsHandshakeCompletionEvent before application data" — and later commits mention TLS 1.3 connection drops when application data is sent before authentication completes. Consistent with that: a probe that waited 250 ms after the handshake before sending CONNECT occasionally succeeded on 1.6 where an immediate send did not.

For this repo

No workaround exists on our side — the defect is inside the published image and cannot be configured around. SslProtocols=Tls12 does not help (4/20 and 2/20 answered, handshakes to 6.7 s).

The gate stays on 1.5 (1.5.44, supported LTS, 6/6 edgehub_module green). This PR stays draft until a fixed 1.6 image is published.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.

[Ewerton Scaboro da Silva (ewertons)]

Mechanism confirmed: each stalled handshake leaks memory in edgeHub 1.6

Host and per-container memory measured immediately before and after the same 20-attempt probe, on both lines:

	edgeHub 1.5 (Handlers 0.7.6)	edgeHub 1.6 (Handlers 0.6.0)
edgeHub RSS before	117.7 MiB (1.48%)	1.344 GiB (17.34%)
edgeHub RSS after	120.9 MiB (1.52%)	6.972 GiB (89.94%)
growth over 20 attempts	+3.2 MiB	+5.63 GiB (~288 MiB per attempt)
host memory available after	6,616 MiB	119 MiB
probe answered	20 / 20	0 / 20

On 1.6 the other containers were squeezed out as edgeHub grew — friendMod 57.64 → 11.77 MiB, testMod 48.73 → 1.78 MiB.

This ties the whole failure together:

• Each connection that stalls in TlsHandler retains buffers instead of releasing them, so the host is progressively exhausted.
• edgeHub was already at 1.344 GiB before the probe even started, from the wrapper connection attempts made during deployment — on 1.5 it was at 117.7 MiB at the same point.
• It explains the agent's 98% memory warnings on 1.6 arms only, the occasional successes taking 11–17 seconds rather than ~20 ms, and why later tests in a run fail more broadly than earlier ones.

DotNetty 0.7.5 (2022-08-31) is "Fix infinite loop caused by TlsHandler" — an unbroken loop in the handler matches both symptoms exactly: no bytes are ever written back, and buffers accumulate without bound. 0.6.0 predates that fix, and 1.6 ships 0.6.0.

Memory pressure is a consequence, not the trigger: the earlier full gate run where all six edgehub_module jobs failed recorded no memory warnings at all.

The fix is unchanged — pin DotNetty.Handlers, DotNetty.Codecs and DotNetty.Codecs.Mqtt to 0.7.6 in edge-hub, matching the DotNetty.Transport pin that already exists.

[Ewerton Scaboro da Silva (ewertons)]

Unblocked: green on edgeHub 1.6.2

edgeHub 1.6.2 shipped 2026-08-26 and contains the fix for the MQTT/TLS failure described above — its changelog lists "Fix Edge Hub MQTT/TLS failures", and packages.lock.json at that tag now resolves DotNetty.Handlers, DotNetty.Codecs and DotNetty.Codecs.Mqtt to 0.7.6 instead of 0.6.0. Only the module images changed; the daemon stays at 1.6.0.

Re-ran the full gate on this branch, rebased on current master:

	1.6.1 (before)	1.6.2 (now)
edgehub_module jobs	0 / 6	6 / 6
all jobs	—	20 / 20

Installing aziot-identity-service=1.6.0-1 and aziot-edge=1.6.0-1
iotedge 1.6.0

Same three-line change as before; only the runtime it deploys against has changed. Taking this out of draft.

One thing to know when reading this: the :1.6 tag floats to the newest 1.6.x, so this picks up 1.6.2 automatically and will pick up later 1.6.x too. That is the same behaviour the repo has always had for :1.5 and :1.4, and IOTHUB_E2E_EDGE_RELEASE_LINE can pin a specific version if a run ever needs one.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.