Recently some builds were failing with message such as:
2026-06-26 11:23:22 INFO container.mc-bufuwuyu Running /usr/bin/systemd-run --machine=mc-bufuwuyu --wait --quiet --working-directory=/ --pipe --uid=0 --gid=0 /srv/moncic-ci/scripts/tmpe3jep10u
2026-06-26 11:23:22 INFO container.mc-bufuwuyu stderr: Failed to connect to system scope bus via machine transport: Connection refused
(e.g.: https://simc.arpae.it/moncic-ci/dballe/202606260914/master/fedora44/build.log)
Looking at the system it seemed that a lot of containers were stuck with systemd-userwork: waiting... messages and/or a generic process still active /usr/lib/systemd/systemd systemd.hostname=mc-*****
e.g.:
1105071 ? Ss 0:02 /usr/sbin/systemd-nspawn --quiet --directory=/moncic-ci/fedora44 --machine=mc-bejanugo --boot --notify-ready=yes --resolv-conf=replace-host --timezone=copy --bind-ro=/tmp/tmp19_hpwzy
1105073 ? Ss 0:02 \_ /usr/lib/systemd/systemd --system --deserialize=29 systemd.hostname=mc-bejanugo
1105101 ? Ss 0:00 \_ /usr/lib/systemd/systemd-journald
1105107 ? Ss 0:08 \_ /usr/lib/systemd/systemd-userdbd
3649154 ? S 0:00 | \_ systemd-userwork: waiting...
3651704 ? S 0:00 | \_ systemd-userwork: waiting...
3651705 ? S 0:00 | \_ systemd-userwork: waiting...
1105119 ? Ss 1:42 \_ /usr/lib/systemd/systemd-oomd
1105133 ? Ss 0:00 \_ /usr/lib/systemd/systemd-homed
1105134 ? Ss 0:00 \_ /usr/lib/systemd/systemd-logind
1105137 ? Ss 0:00 \_ /usr/bin/dbus-broker-launch --scope system --audit
1105149 ? S 0:00 | \_ dbus-broker --log 10 --controller 9 --machine-id eefd4b60c510479781212e01f3bd47bd --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit
1105152 pts/0 Ss+ 0:00 \_ /usr/sbin/agetty --noreset --noclear --issue-file=/etc/issue:/etc/issue.d:/run/issue.d:/usr/lib/issue.d --keep-baud 115200,57600,38400,9600 - vt220
2756491 ? Ss 0:00 /usr/lib/systemd/systemd systemd.hostname=mc-bogaluca
547703 ? Ss 0:00 /usr/lib/systemd/systemd systemd.hostname=mc-sasosa
2249161 ? Ss 0:00 /usr/lib/systemd/systemd systemd.hostname=mc-birufowe
2293258 ? Ss 0:00 /usr/lib/systemd/systemd systemd.hostname=mc-bisucisa
3450728 ? Ss 0:00 /usr/lib/systemd/systemd systemd.hostname=mc-bubikibe
167656 ? Ss 0:00 /usr/lib/systemd/systemd systemd.hostname=mc-foposa
I've updated and rebooted the CI VM and so far the issue did not recur, I'll leave this issue open for a while just in case
Recently some builds were failing with message such as:
(e.g.: https://simc.arpae.it/moncic-ci/dballe/202606260914/master/fedora44/build.log)
Looking at the system it seemed that a lot of containers were stuck with
systemd-userwork: waiting...messages and/or a generic process still active/usr/lib/systemd/systemd systemd.hostname=mc-*****e.g.:
I've updated and rebooted the CI VM and so far the issue did not recur, I'll leave this issue open for a while just in case