feat: update docs for NetworkManager

Related-to: harvester/harvester#3418

Signed-off-by: Tim Serong <[email protected]>

Author: tserong

docs/install/harvester-configuration.md

@@ -280,10 +280,10 @@ DNS resolution is unavailable in the `after-install-chroot stage`, and the `name
 ```yaml
 os:
   after_install_chroot_commands:
-    - "rm -f /etc/resolv.conf && echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf"
+    - "echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf"
     - "mkdir /usr/local/bin"
     - "curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh"
-    - "rm -f /etc/resolv.conf && ln -s /var/run/netconfig/resolv.conf /etc/resolv.conf"
+    - "rm -f /etc/resolv.conf"
 ```
 
 

docs/install/requirements.md

@@ -90,7 +90,7 @@ Harvester nodes require the following port connections or inbound rules. Typical
 | TCP       | 9796                     | Harvester management and compute nodes  | Node-exporter                         |
 | TCP       | 30000-32767             |  Harvester management and compute nodes  | NodePort port range                   |
 | TCP       | 22                       | Harvester management and compute nodes  | sshd                                  |
-| UDP       | 68                       | Harvester management and compute nodes  | Wicked                                |
+| UDP       | 68                       | Harvester management and compute nodes  | NetworkManager                        |
 | TCP       | 3260                     | Harvester management and compute nodes |   iscsid                                |
 
 ### Port Requirements for Integrating Harvester with Rancher

docs/install/update-harvester-configuration.md

@@ -20,23 +20,25 @@ This page describes how to edit some of the most-requested Harvester configurati
 
 If you upgrade from a version before `v1.1.2`, the `cloud-init` file in examples will be `/oem/99_custom.yaml`. Please substitute the value if needed.
 
+If you upgrade from a version before `v1.7.0`, any changes to network and DNS configuration need to be made in the `/oem/91_networkmanager.yaml` file instead of in `/oem/90_custom.yaml`. Please substitute the value if needed.
+
 :::
 
 ## DNS servers
 
 ### Runtime change
 
 1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
-1. Edit `/etc/sysconfig/network/config` and update the following line. Use a space to separate DNS server addresses if there are multiple servers.
+1. If the management interface _is not_ configured to use a VLAN, run the following command:
 
     ```
-    NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"
+    nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br
     ```
 
-1. Update and reload the configuration with the following command:
+1. If the management interface _is_ configured to use a VLAN, run the following commands. Replace `VLAN_ID` with the actal ID of the VLAN. If in doubt, run `nmcli con` to see the configured connections and devices.
 
     ```
-    netconfig update
+    nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.VLAN_ID
     ```
 
 1. Confirm the file `/etc/resolv.conf` contains the correct DNS servers with the `cat` command:
@@ -55,28 +57,39 @@ If you upgrade from a version before `v1.1.2`, the `cloud-init` file in examples
 
 ### Configuration persistence
 
-Beginning with v1.1.2, the persistent name of the cloud-init file is `/oem/90_custom.yaml`. Harvester now uses a newer version of Elemental, which creates the file during installation.
+:::note
 
-When upgrading from an earlier version to `v1.1.2` or later, Harvester retains the old file name (`/oem/99_custom.yaml`) to avoid confusion. You can manually rename the file to `/oem/90_custom.yaml` if necessary.
+If you upgrade from a version before `v1.7.0`, the changes below need to be made in the `/oem/91_networkmanager.yaml` file instead of in `/oem/90_custom.yaml`.
+
+:::
 
 1. Backup the elemental `cloud-init` file `/oem/90_custom.yaml` as follows:
 
     ```
     cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
     ```
 
-1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.initramfs[0].commands`. The `commands` array must contain a line to manipulate the `NETCONFIG_DNS_STATIC_SERVERS` config. Add the line if the line doesn't exist. 
+1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.network[0].commands`. The `commands` array must contain a line to manipulate the NetworkManager DNS configuration for the management interface. This is the exact same command used above when making the change at runtime. Add the line if the line doesn't exist.
 
-    The following example adds a line to change the `NETCONFIG_DNS_STATIC_SERVERS` config:
+    The following example adds a line to configure DNS servers when not using a VLAN on the management interface:
 
     ```
     stages:
-      initramfs:
+      network:
+        - commands:
+            - nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br
+    ```
+
+    The following example adds a line to configure DNS servers when using VLAN 2017 on the management interface:
+
+    ```
+    stages:
+      network:
         - commands:
-            - sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"/' /etc/sysconfig/network/config
+            - nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.2017
     ```
 
-    Replace the DNS server addresses and save the file. Harvester sets up new servers after rebooting.
+    Replace the DNS server addresses and VLAN ID if applicable and save the file. Harvester sets up new servers after rebooting.
 
 
 ## NTP servers
@@ -113,78 +126,80 @@ You can update the slave interfaces of Harvester's management bonding interface
 1. Identify the interface names with the following command:
 
     ```
-    ip a
+    $ nmcli device
+    DEVICE           TYPE      STATE                   CONNECTION
+    mgmt-br          bridge    connected               bridge-mgmt
+    ...
+    mgmt-bo          bond      connected               bond-mgmt
+    ens6             ethernet  connected               bond-slave-ens6
+    ens7             ethernet  disconnected            --
+    ...
     ```
 
-1. Edit `/etc/sysconfig/network/ifcfg-mgmt-bo` and update the lines associated with bonding slaves and bonding mode:
+1. Use `nmcli` to create a connection for the interface and attach it to the management bond, for example:
 
     ```
-    BONDING_SLAVE_0='ens5'
-    BONDING_SLAVE_1='ens6'
-    BONDING_MODULE_OPTS='miimon=100 mode=balance-tlb '
-    ```
+    $ nmcli con add type bond-slave ifname ens7 master mgmt-bo
+    Connection 'bond-slave-ens7' (5a379328-178a-4167-b065-b5426facd659) successfully added.
 
-1. Restart the network with the `wicked ifreload` command:
+1. You should now be able to see the device is connected:
 
     ```
-    wicked ifreload mgmt-bo
+    $ nmcli device
+    DEVICE           TYPE      STATE                   CONNECTION
+    mgmt-br          bridge    connected               bridge-mgmt
+    ...
+    mgmt-bo          bond      connected               bond-mgmt
+    ens6             ethernet  connected               bond-slave-ens6
+    ens7             ethernet  connected               bond-slave-ens7
     ```
 
-    :::caution
-
-    A mistake in the configuration may disrupt the SSH session.
+### Configuration persistence
 
-    :::
+:::note
 
-### Configuration persistence
+If you upgrade from a version before `v1.7.0`, the changes below need to be made in the `/oem/91_networkmanager.yaml` file instead of in `/oem/90_custom.yaml`.
 
+:::
 
 1. Backup the elemental cloud-init file `/oem/90_custom.yaml` as follows:
 
     ```
     cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
     ```
 
-1. Edit `/oem/90_custom.yaml` and update the yaml path `stages.initramfs[0].files`. More specifically, update the content of the `/etc/sysconfig/network/ifcfg-mgmt-bo` file and edit the `BONDING_SLAVE_X` and `BONDING_MODULE_OPTS` entries accordingly:
+1. Edit `/oem/90_custom.yaml` and add a entry under `stages.initramfs[0].files` to create the NetworkManager connection profile for the new device. The most straightforward thing to do is to copy an existing bonding slave device and update the `path`, `id` and `interface-name` fields to match the new device. The following example shows the creation of `/etc/NetworkManager/system-connections/bond-slave-ens7.nmconnection` based on the existing `/etc/NetworkManager/system-connections/bond-slave-ens6.nmconnection` connection profile:
 
     ```
     stages:
       initramfs:
-      - ...
-        files:
-        - path: /etc/sysconfig/network/ifcfg-mgmt-bo
-          permissions: 384
-          owner: 0
-          group: 0
-          content: |+
-              STARTMODE='onboot'
-              BONDING_MASTER='yes'
-              BOOTPROTO='none'
-              POST_UP_SCRIPT="wicked:setup_bond.sh"
-    
-    
-              BONDING_SLAVE_0='ens5'
-              BONDING_SLAVE_1='ens6'
-    
-              BONDING_MODULE_OPTS='miimon=100 mode=balance-tlb '
-    
-              DHCLIENT_SET_DEFAULT_ROUTE='no'
-    
-          encoding: ""
-          ownerstring: ""
-        - path: /etc/sysconfig/network/ifcfg-ens6
-          permissions: 384
-          owner: 0
-          group: 0
-          content: |
-            STARTMODE='hotplug'
-            BOOTPROTO='none'
-          encoding: ""
-          ownerstring: ""
-    ```
-
-    :::note
-
-    If you didn't select an interface during installation, you must add an entry to initialize the interface. Please check the `/etc/sysconfig/network/ifcfg-ens6` file creation in the above example. The file name should be `/etc/sysconfig/network/ifcfg-<interface-name>`.
-
-    :::
+        - files:
+            - ...
+            - path: /etc/NetworkManager/system-connections/bond-slave-ens6.nmconnection
+              permissions: 384
+              owner: 0
+              group: 0
+              content: |
+                [connection]
+                id=bond-slave-ens6
+                type=ethernet
+                interface-name=ens6
+                master=mgmt-bo
+                slave-type=bond
+              encoding: ""
+              ownerstring: ""
+            - path: /etc/NetworkManager/system-connections/bond-slave-ens7.nmconnection
+              permissions: 384
+              owner: 0
+              group: 0
+              content: |
+                [connection]
+                id=bond-slave-ens7
+                type=ethernet
+                interface-name=ens7
+                master=mgmt-bo
+                slave-type=bond
+              encoding: ""
+              ownerstring: ""
+            - ...
+    ```

docs/logging/harvester-logging.md

@@ -47,6 +47,7 @@ In Harvester's practice, the `Logging`, `Audit` and `Event` shares one architect
 The Harvester logging infrastructure allows you to aggregate Harvester logs into an external service such as [Graylog](https://www.graylog.org), [Elasticsearch](https://www.elastic.co/elasticsearch/), [Splunk](https://www.splunk.com/), [Grafana Loki](https://grafana.com/oss/loki/) and others.
 
 ### Collected Logs
+
 See below for a list logs that are collected:
  - Logs from all cluster `Pods`
  - Kernel logs from each `node`
@@ -55,7 +56,7 @@ See below for a list logs that are collected:
    - `rke2-agent`
    - `rancherd`
    - `rancher-system-agent`
-   - `wicked`
+   - `NetworkManager`
    - `iscsid`
 
 :::note

docs/networking/best-practice.md

@@ -45,28 +45,40 @@ Before any action is taken, it is important to collect the current network infor
     Example: A NIC named `ens3` was added to the `mgmt-bo` bond interface.
 
     ```
-    - path: /etc/sysconfig/network/ifcfg-mgmt-bo
+    - path: /etc/NetworkManager/system-connections/bond-mgmt.nmconnection
       permissions: 384
       owner: 0
       group: 0
-      content: |+
-        STARTMODE='onboot'
-        BONDING_MASTER='yes'
-        BOOTPROTO='none'
-        POST_UP_SCRIPT="wicked:setup_bond.sh"
-        BONDING_SLAVE_0='ens3'
-        BONDING_MODULE_OPTS='miimon=100 mode=active-backup '
-        DHCLIENT_SET_DEFAULT_ROUTE='no'
+      content: |-
+        [connection]
+        id=bond-mgmt
+        type=bond
+        interface-name=mgmt-bo
+        master=mgmt-br
+        slave-type=bridge
+
+        [ethernet]
+
+        [bond]
+        miimon=100
+        mode=active-backup
+
+        [bridge-port]
+        vlans=1 pvid untagged
       encoding: ""
       ownerstring: ""
 
-    - path: /etc/sysconfig/network/ifcfg-ens3
+    - path: /etc/NetworkManager/system-connections/bond-slave-ens3.nmconnection
       permissions: 384
       owner: 0
       group: 0
       content: |
-        STARTMODE='hotplug'
-        BOOTPROTO='none'
+        [connection]
+        id=bond-slave-ens3
+        type=ethernet
+        interface-name=ens3
+        master=mgmt-bo
+        slave-type=bond
       encoding: ""
       ownerstring: ""
     ```