feat: update docs for NetworkManager #897
Conversation
github-actions
bot
requested review from
akashraj4261,
dariavladykina and
jillian-maroket
|
|
Note to self: upgrade docs also need to point out that any additional manual network config (e.g. custom cloudinit CRDs that set up ifcfg- files) will need manual changes to migrate to NetworkManager. |
tserong
force-pushed
the
wip-networkmanager
branch
from
3ab9c7b to
438cfbe
Compare
|
Rebased to include #922. This should be reasonably complete now, in terms of replacing all existing wicked customization content with NetworkManager equivalents. Still TODO:
|
tserong
force-pushed
the
wip-networkmanager
branch
3 times, most recently
from
841085a to
669f9f0
Compare
tserong
requested review from
Vicente-Cheng,
ibrokethecloud,
ihcsim,
rrajendran17 and
w13915984028
|
OK, this should, I hope, be complete now. |
|
On thing I may not have made clear when talking about users who've made manual changes to network configuration needing to do extra work during upgrade: here I'm only talking about changes to management interface configuration, i.e. if users have been tweaking things in I am not talking about Cluster Networks, VM Networks, etc. that were configured directly via Harvester (and are managed by the network controller) as described in the Networking chapter. Those things should continue to work just fine after upgrade with no extra steps required. |
| :::info important | ||
| Example, where the primary VLAN ID is 2017 and the secondary VLAN ID is 2018: | ||
| ``` | ||
| $ nmcli con modify bond-mgmt bridge-port.vlans '1 pvid untagged, 2017, 2018' |
There was a problem hiding this comment.
The question here and similar places are:
We assume the /oem/90_custom.yaml is the source of truth, all additional changes are suggested to change upon this file, and then it can survive after node is rebooted or upgraded (Harvester upgrade script adds/changes the file content selectively).
If user only do actions via command nmcli and don't write&save to above yaml, how to ensure the desired configurations are persistant? Or the example here is under the cap of /oem/90_custom.yaml?
thanks.
There was a problem hiding this comment.
We've added /etc/NetworkManager to the list of persistent paths, so when the user runs nmcli, it makes changes to connection profiles in that directory, and they are persisted immediately. Thus there's no need to make changes to /oem/90_custom.yaml for network configuration. This is mentioned in docs/install/update-harvester-configuration.md, but I haven't made it explicit in other places like docs/networking/clusternetwork.md as you mention here.
Should I maybe add a note in each place where we suggest running nmcli to mention that the change will persist automatically?
There was a problem hiding this comment.
I have following concerns:
In the past, most of our documents/guides told cusomter/support-engineer to change the /oem/90_custom.yaml or other yaml files to make changes, network, password, dhcp server... if they can only be done via those low level operations.
Now, for the network part, which is fully? deletgated to $nmcli command and make changes off the /oem/90_custom.yaml. It might bring those challenges:
(1) How to check the current persistant configurations, via nmcli
(2) The basic mgmt network related configurations are still saved on /oem, but others might be only on nmcli, information might conflict, e.g. only change mgmt-bo MTU via nmcli, but don't update yaml
(3) It could also make user's exsting tools/scripts more difficult to wirte
Could we simplify users operation, to guide all changes still upon /oem/90_custom.yaml file? thanks.
There was a problem hiding this comment.
Re: the points above:
- The current persistent configurations for NetworkManager (i.e. network config and DNS) are now in files in
/etc/NetworkManager/system-connections/. These can be manipulated using thenmclitool, or, by editing those files directly, but its really easier with thenmclitool. - There isn't any network interface or DNS configuration saved in
/oemanymore. I think the only thing left in there is the system hostname, and that should never be changed by the user anyway. - I'm not sure how this makes existing tools/scripts more difficult to write. Can you provide an example?
Given the NetworkManager config was made persistent, I think trying to also change it through /oem/90_custom.yaml would make the whole situation more confusing.
There was a problem hiding this comment.
I checked the SB on this issue harvester/harvester#9697
after upgrade to v17* cluster, the /oem/90_custom.yaml has following contents:
but seems to be leftover information on the yaml file
on fresh v170 cluster, it does not have such info;
BTW, exclude those interfaces like cali* looks helpful, I did not have latest cluster, but hopefully, the nmcli will not show them after adding the exclude list.
harv41:/home/rancher # nmcli
mgmt-br: connected to bridge-mgmt
"mgmt-br"
bridge, 52:54:00:03:3A:E4, sw, mtu 1500
ip4 default
inet4 192.168.122.141/24
inet4 192.168.122.142/32
route4 192.168.122.0/24 metric 425
route4 default via 192.168.122.1 metric 425
lo: connected (externally) to lo
"lo"
loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
inet4 127.0.0.1/8
inet6 ::1/128
vip-440cdc2d: connected (externally) to vip-440cdc2d
"vip-440cdc2d"
macvlan, 00:00:6C:D5:3A:2E, sw, mtu 1500
inet4 192.168.122.100/32
flannel.1: connected (externally) to flannel.1
"flannel.1"
vxlan, 6A:94:15:A6:72:B3, sw, mtu 1450
inet4 10.52.0.0/32
cn2-bo: connected (externally) to cn2-bo
"cn2-bo"
bond, 52:54:00:9C:83:1D, sw, mtu 1500
mgmt-bo: connected to bond-mgmt
"mgmt-bo"
bond, 52:54:00:03:3A:E4, sw, mtu 1500
master mgmt-br
cn2-br: connected (externally) to cn2-br
"cn2-br"
bridge, CE:D9:77:E8:FB:8B, sw, mtu 1500
ens3: connected to bond-slave-ens3
"Intel 82540EM"
ethernet (e1000), 52:54:00:03:3A:E4, hw, mtu 1500
master mgmt-bo
ens8: connected (externally) to ens8
"Intel 82540EM"
ethernet (e1000), 52:54:00:9C:83:1D, hw, mtu 1500
master cn2-bo
cali09d107a80c1: unmanaged
"cali09d107a80c1"
ethernet (veth), EE:EE:EE:EE:EE:EE, sw, mtu 1450
cali111c46b7208: unmanaged
"cali111c46b7208"
ethernet (veth), EE:EE:EE:EE:EE:EE, sw, mtu 1450
and the NetworkManager service log can also exclude them
...
Dec 09 19:39:12.208242 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.2082] manager: (cali7f0ea92b273): new Veth device (/org/freedesktop/NetworkManager/Devices/28)
Dec 09 19:39:12.216274 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.2162] device (cali7f0ea92b273): carrier: link connected
Dec 09 19:39:12.223781 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.2237] manager: (cali72e3cb4bcfd): new Veth device (/org/freedesktop/NetworkManager/Devices/29)
Dec 09 19:39:12.226758 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.2267] device (cali72e3cb4bcfd): carrier: link connected
Dec 09 19:39:12.347583 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.3475] manager: (cali4fff037aa4a): new Veth device (/org/freedesktop/NetworkManager/Devices/30)
Dec 09 19:39:12.355521 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.3554] device (cali4fff037aa4a): carrier: link connected
Dec 09 19:39:12.372544 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.3725] manager: (calic06bfeb03de): new Veth device (/org/freedesktop/NetworkManager/Devices/31)
Dec 09 19:39:12.373557 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.3735] device (calic06bfeb03de): carrier: link connected
Dec 09 19:39:12.390454 david-demo-hp17 NetworkManager[2154]: <info> [1765309152.3904] manager: (calic1f430c1486): new Veth device (/org/freedesktop/NetworkManager/Devices/32)
...
There was a problem hiding this comment.
ok, besides the above leftover information, the networkmanager seems to have a /oem/91_network-manager.yaml file after upgrade
in this case, if later nmcli operaion conflicts with it, how do we handle such?
example from above SB
name: Harvester Network Configuration
stages:
initramfs:
- files:
- path: /etc/NetworkManager/system-connections/bond-mgmt.nmconnection
permissions: 384
owner: 0
group: 0
content: |-
[connection]
id=bond-mgmt
type=bond
interface-name=mgmt-bo
master=mgmt-br
slave-type=bridge
[ethernet]
[bond]
miimon=100
mode=active-backup
[bridge-port]
vlans=1 pvid untagged,2013
encoding: ""
ownerstring: ""
- path: /etc/NetworkManager/system-connections/bond-slave-eno49.nmconnection
permissions: 384
owner: 0
group: 0
content: |
[connection]
id=bond-slave-eno49
type=ethernet
interface-name=eno49
master=mgmt-bo
slave-type=bond
encoding: ""
ownerstring: ""
There was a problem hiding this comment.
BTW, from above SB, I did not find network manager configurations, besides the /oem/91_.. file (not fully sure is it generated via upgrade, of SB tool).
could you double check, if the SB tool have been updated to fetch all networkmanager related configurations? thanks.
seems we need to add all/some of below files to to collector
ls -r /etc/NetworkManager/
VPN system-connections dispatcher.d conf.d
There was a problem hiding this comment.
but seems to be leftover information on the yaml file
Correct. All those wicked config things - ifcfg-* files, the setup_bond.sh script, etc. are not processed or used at all by NetworkManager. That they remain in /oem/90_custom.yaml after an upgrade is harmless.
ok, besides the above leftover information, the networkmanager seems to have a
/oem/91_network-manager.yamlfile after upgrade
This was there in the original NM implementation, but was removed for v1.7.0-rc6 with harvester/harvester#9581.
could you double check, if the SB tool have been updated to fetch all networkmanager related configurations? thanks.
rancher/support-bundle-kit#160 should capture NM logs (wicked logs were removed in rancher/support-bundle-kit#157). @m-ildefons opened a PR earlier in the year to gather more network info, which will pick up a lot more detail - see rancher/support-bundle-kit#135 - but this hasn't been merged yet.
Related-to: harvester/harvester#3418 Signed-off-by: Tim Serong <[email protected]>
tserong
force-pushed
the
wip-networkmanager
branch
from
669f9f0 to
4fbffad
Compare
|
Hopefully the second commit I just added makes things a little clearer about |
Signed-off-by: Tim Serong <[email protected]>
tserong
force-pushed
the
wip-networkmanager
branch
from
4fbffad to
4e26397
Compare
Co-authored-by: Jillian Maroket <[email protected]> Signed-off-by: Tim Serong <[email protected]>
jillian-maroket
left a comment
jillian-maroket
left a comment
There was a problem hiding this comment.
PTAL at the last few suggestions.
Co-authored-by: Jillian Maroket <[email protected]> Signed-off-by: Tim Serong <[email protected]>
Signed-off-by: Tim Serong <[email protected]>
Co-authored-by: Jillian Maroket <[email protected]> Signed-off-by: Tim Serong <[email protected]>
4 participants
Problem:
Harvester v1.7.0 uses NetworkManager instead of Wicked
Solution:
/etc/NetworkManagerso changes made usingnmcliwill automatically persist. There's no need to edit YAML in/oem/90_custom.yamlto tweak NetworkManager config as was previously necessary with the wicked ifcfg files.Related Issue(s):
harvester/harvester#3418
Test plan:
N/A
Additional documentation or context
Note: there's a link to the NetworkManager HEP in here, which is a link to the PR, rather than the final document in the source tree. That's because the HEP PR isn't actually merged yet. Need to update once that PR is merged.