Skip to content

Commit 841085a

Browse files
tserongtserong
committed
feat: update docs for NetworkManager
Related-to: harvester/harvester#3418 Signed-off-by: Tim Serong <[email protected]>
1 parent cc6b84a commit 841085a

File tree

8 files changed

+117
-211
lines changed

8 files changed

+117
-211
lines changed

‎docs/install/harvester-configuration.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -280,10 +280,10 @@ DNS resolution is unavailable in the `after-install-chroot stage`, and the `name
280280
```yaml
281281
os:
282282
after_install_chroot_commands:
283-
- "rm -f /etc/resolv.conf && echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf"
283+
- "echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf"
284284
- "mkdir /usr/local/bin"
285285
- "curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh"
286-
- "rm -f /etc/resolv.conf && ln -s /var/run/netconfig/resolv.conf /etc/resolv.conf"
286+
- "rm -f /etc/resolv.conf"
287287
```
288288

289289

‎docs/install/requirements.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ Harvester nodes require the following port connections or inbound rules. Typical
9090
| TCP | 9796 | Harvester management and compute nodes | Node-exporter |
9191
| TCP | 30000-32767 | Harvester management and compute nodes | NodePort port range |
9292
| TCP | 22 | Harvester management and compute nodes | sshd |
93-
| UDP | 68 | Harvester management and compute nodes | Wicked |
93+
| UDP | 68 | Harvester management and compute nodes | NetworkManager |
9494
| TCP | 3260 | Harvester management and compute nodes | iscsid |
9595

9696
### Port Requirements for Integrating Harvester with Rancher

‎docs/install/update-harvester-configuration.md‎

Lines changed: 49 additions & 126 deletions
Original file line numberDiff line numberDiff line change
@@ -12,179 +12,102 @@ description: How to update Harvester configuration after installation
1212
<link rel="canonical" href="https://docs.harvesterhci.io/v1.6/install/update-harvester-configuration"/>
1313
</head>
1414

15-
Harvester's OS has an immutable design, which means most files in the OS revert to their pre-configured state after a reboot. The Harvester OS loads the pre-configured values of system components from configuration files during the boot time.
15+
Harvester's OS has an immutable design, which means most files in the OS revert to their pre-configured state after a reboot. The Harvester OS loads the pre-configured values of system components from configuration files during the boot time.
1616

17-
This page describes how to edit some of the most-requested Harvester configurations. To update a configuration, you must first update the runtime value in the system and then update configuration files to make the changes persistent between reboots.
17+
This page describes how to edit some of the most-requested Harvester configurations. To update a configuration, you must first update the runtime value in the system and then update configuration files to make the changes persistent between reboots.
18+
19+
One notable exception to this starting in Harvester v1.7.0 is network configuration, as the `/etc/NetworkManager` directory was added to the list of persistent paths. This means that changes to network configuration made using the `nmcli` tool will persist immediately.
1820

1921
:::note
2022

2123
If you upgrade from a version before `v1.1.2`, the `cloud-init` file in examples will be `/oem/99_custom.yaml`. Please substitute the value if needed.
2224

2325
:::
2426

25-
## DNS servers
27+
## Password of user `rancher`
2628

2729
### Runtime change
2830

29-
1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
30-
1. Edit `/etc/sysconfig/network/config` and update the following line. Use a space to separate DNS server addresses if there are multiple servers.
31-
32-
```
33-
NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"
34-
```
35-
36-
1. Update and reload the configuration with the following command:
37-
38-
```
39-
netconfig update
40-
```
41-
42-
1. Confirm the file `/etc/resolv.conf` contains the correct DNS servers with the `cat` command:
43-
44-
```
45-
cat /etc/resolv.conf
46-
```
47-
1. Restart rke2-coredns:
48-
```
49-
kubectl rollout restart deployment/rke2-coredns-rke2-coredns -n kube-system
50-
```
51-
1. Confirm rke2-coredns was rolled out successfully:
52-
```
53-
kubectl rollout status deployment/rke2-coredns-rke2-coredns -n kube-system
54-
```
31+
1. Log in to a Harvester node as user `rancher`. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
32+
1. To reset the password for the user `rancher`, run the command `passwd`.
5533

5634
### Configuration persistence
5735

58-
Beginning with v1.1.2, the persistent name of the cloud-init file is `/oem/90_custom.yaml`. Harvester now uses a newer version of Elemental, which creates the file during installation.
59-
60-
When upgrading from an earlier version to `v1.1.2` or later, Harvester retains the old file name (`/oem/99_custom.yaml`) to avoid confusion. You can manually rename the file to `/oem/90_custom.yaml` if necessary.
61-
6236
1. Backup the elemental `cloud-init` file `/oem/90_custom.yaml` as follows:
6337

6438
```
6539
cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
6640
```
6741
68-
1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.initramfs[0].commands`. The `commands` array must contain a line to manipulate the `NETCONFIG_DNS_STATIC_SERVERS` config. Add the line if the line doesn't exist.
69-
70-
The following example adds a line to change the `NETCONFIG_DNS_STATIC_SERVERS` config:
71-
72-
```
73-
stages:
74-
initramfs:
75-
- commands:
76-
- sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"/' /etc/sysconfig/network/config
77-
```
78-
79-
Replace the DNS server addresses and save the file. Harvester sets up new servers after rebooting.
80-
42+
1. Edit `/oem/90_custom.yaml` and update the yaml path `stages.initramfs[0].users.rancher.passwd`. Refer to the configuration [`os.password`](./harvester-configuration.md#ospassword) for details on how to specify the password in an encrypted form.
8143
8244
## NTP servers
8345
84-
We introduce the new mechanism for the NTP configuration in Harvester v1.2.0.
46+
We introduced a new mechanism for NTP configuration in Harvester v1.2.0.
8547
86-
For more information about NTP settings in Harvester v1.2.0 and later versions, see the [NTP servers](../host/host.md#ntp-configuration).
87-
88-
## Password of user `rancher`
48+
For more information about NTP settings in Harvester v1.2.0 and later versions, see [NTP servers](../host/host.md#ntp-configuration).
8949
90-
### Runtime change
50+
## DNS servers
9151
92-
1. Log in to a Harvester node as user `rancher`. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
93-
1. To reset the password for the user `rancher`, run the command `passwd`.
52+
1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
53+
1. If the management interface _is not_ configured to use a VLAN, run the following command:
9454
95-
### Configuration persistence
55+
```
56+
nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br
57+
```
9658
97-
1. Backup the elemental `cloud-init` file `/oem/90_custom.yaml` as follows:
59+
1. If the management interface _is_ configured to use a VLAN, run the following commands. Replace `VLAN_ID` with the actal ID of the VLAN. If in doubt, run `nmcli con` to see the configured connections and devices.
9860
9961
```
100-
cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
62+
nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.VLAN_ID
10163
```
10264
103-
1. Edit `/oem/90_custom.yaml` and update the yaml path `stages.initramfs[0].users.rancher.passwd`. Refer to the configuration [`os.password`](./harvester-configuration.md#ospassword) for details on how to specify the password in an encrypted form.
65+
1. Confirm the file `/etc/resolv.conf` contains the correct DNS servers with the `cat` command:
10466
67+
```
68+
cat /etc/resolv.conf
69+
```
70+
1. Restart rke2-coredns:
71+
```
72+
kubectl rollout restart deployment/rke2-coredns-rke2-coredns -n kube-system
73+
```
74+
1. Confirm rke2-coredns was rolled out successfully:
75+
```
76+
kubectl rollout status deployment/rke2-coredns-rke2-coredns -n kube-system
77+
```
10578
10679
## Bonding slaves
10780
10881
You can update the slave interfaces of Harvester's management bonding interface `mgmt-bo`.
10982
110-
### Runtime change
111-
11283
1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
11384
1. Identify the interface names with the following command:
11485
11586
```
116-
ip a
117-
```
118-
119-
1. Edit `/etc/sysconfig/network/ifcfg-mgmt-bo` and update the lines associated with bonding slaves and bonding mode:
120-
121-
```
122-
BONDING_SLAVE_0='ens5'
123-
BONDING_SLAVE_1='ens6'
124-
BONDING_MODULE_OPTS='miimon=100 mode=balance-tlb '
87+
$ nmcli device
88+
DEVICE TYPE STATE CONNECTION
89+
mgmt-br bridge connected bridge-mgmt
90+
...
91+
mgmt-bo bond connected bond-mgmt
92+
ens6 ethernet connected bond-slave-ens6
93+
ens7 ethernet disconnected --
94+
...
12595
```
12696
127-
1. Restart the network with the `wicked ifreload` command:
97+
1. Use `nmcli` to create a connection for the interface and attach it to the management bond, for example:
12898
12999
```
130-
wicked ifreload mgmt-bo
131-
```
132-
133-
:::caution
134-
135-
A mistake in the configuration may disrupt the SSH session.
100+
$ nmcli con add type bond-slave ifname ens7 master mgmt-bo
101+
Connection 'bond-slave-ens7' (5a379328-178a-4167-b065-b5426facd659) successfully added.
136102
137-
:::
138-
139-
### Configuration persistence
140-
141-
142-
1. Backup the elemental cloud-init file `/oem/90_custom.yaml` as follows:
103+
1. You should now be able to see the device is connected:
143104
144105
```
145-
cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
106+
$ nmcli device
107+
DEVICE TYPE STATE CONNECTION
108+
mgmt-br bridge connected bridge-mgmt
109+
...
110+
mgmt-bo bond connected bond-mgmt
111+
ens6 ethernet connected bond-slave-ens6
112+
ens7 ethernet connected bond-slave-ens7
146113
```
147-
148-
1. Edit `/oem/90_custom.yaml` and update the yaml path `stages.initramfs[0].files`. More specifically, update the content of the `/etc/sysconfig/network/ifcfg-mgmt-bo` file and edit the `BONDING_SLAVE_X` and `BONDING_MODULE_OPTS` entries accordingly:
149-
150-
```
151-
stages:
152-
initramfs:
153-
- ...
154-
files:
155-
- path: /etc/sysconfig/network/ifcfg-mgmt-bo
156-
permissions: 384
157-
owner: 0
158-
group: 0
159-
content: |+
160-
STARTMODE='onboot'
161-
BONDING_MASTER='yes'
162-
BOOTPROTO='none'
163-
POST_UP_SCRIPT="wicked:setup_bond.sh"
164-
165-
166-
BONDING_SLAVE_0='ens5'
167-
BONDING_SLAVE_1='ens6'
168-
169-
BONDING_MODULE_OPTS='miimon=100 mode=balance-tlb '
170-
171-
DHCLIENT_SET_DEFAULT_ROUTE='no'
172-
173-
encoding: ""
174-
ownerstring: ""
175-
- path: /etc/sysconfig/network/ifcfg-ens6
176-
permissions: 384
177-
owner: 0
178-
group: 0
179-
content: |
180-
STARTMODE='hotplug'
181-
BOOTPROTO='none'
182-
encoding: ""
183-
ownerstring: ""
184-
```
185-
186-
:::note
187-
188-
If you didn't select an interface during installation, you must add an entry to initialize the interface. Please check the `/etc/sysconfig/network/ifcfg-ens6` file creation in the above example. The file name should be `/etc/sysconfig/network/ifcfg-<interface-name>`.
189-
190-
:::

‎docs/logging/harvester-logging.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ In Harvester's practice, the `Logging`, `Audit` and `Event` shares one architect
4747
The Harvester logging infrastructure allows you to aggregate Harvester logs into an external service such as [Graylog](https://www.graylog.org), [Elasticsearch](https://www.elastic.co/elasticsearch/), [Splunk](https://www.splunk.com/), [Grafana Loki](https://grafana.com/oss/loki/) and others.
4848

4949
### Collected Logs
50+
5051
See below for a list logs that are collected:
5152
- Logs from all cluster `Pods`
5253
- Kernel logs from each `node`
@@ -55,7 +56,7 @@ See below for a list logs that are collected:
5556
- `rke2-agent`
5657
- `rancherd`
5758
- `rancher-system-agent`
58-
- `wicked`
59+
- `NetworkManager`
5960
- `iscsid`
6061

6162
:::note

‎docs/networking/best-practice.md‎

Lines changed: 29 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -40,35 +40,38 @@ You can follow the steps below and run them in each node step by step.
4040

4141
Before any action is taken, it is important to collect the current network information and status.
4242

43-
- Harvester network configuration: By default, Harvester creates a bond interface named `mgmt-bo` for the management network and one new bond interface for each cluster network. Harvester saves network configuration details in the file `/oem/90_custom.yaml`.
43+
- Harvester network configuration: By default, Harvester creates a bond interface named `mgmt-bo` for the management network. On top of that is a bridge interface named `mgmt-br`, which may optionally use a VLAN. There is also one new bond interface for each cluster network. The current connection details can be views using the `nmcli` tool.
4444

45-
Example: A NIC named `ens3` was added to the `mgmt-bo` bond interface.
45+
Example:
4646

4747
```
48-
- path: /etc/sysconfig/network/ifcfg-mgmt-bo
49-
permissions: 384
50-
owner: 0
51-
group: 0
52-
content: |+
53-
STARTMODE='onboot'
54-
BONDING_MASTER='yes'
55-
BOOTPROTO='none'
56-
POST_UP_SCRIPT="wicked:setup_bond.sh"
57-
BONDING_SLAVE_0='ens3'
58-
BONDING_MODULE_OPTS='miimon=100 mode=active-backup '
59-
DHCLIENT_SET_DEFAULT_ROUTE='no'
60-
encoding: ""
61-
ownerstring: ""
62-
63-
- path: /etc/sysconfig/network/ifcfg-ens3
64-
permissions: 384
65-
owner: 0
66-
group: 0
67-
content: |
68-
STARTMODE='hotplug'
69-
BOOTPROTO='none'
70-
encoding: ""
71-
ownerstring: ""
48+
$ nmcli
49+
50+
mgmt-br.2017: connected to vlan-mgmt
51+
"mgmt-br.2017"
52+
vlan, 5C:B9:01:89:C2:F5, sw, mtu 1500
53+
ip4 default
54+
inet4 10.115.55.20/21
55+
route4 10.115.48.0/21 metric 400
56+
route4 default via 10.115.55.254 metric 400
57+
58+
...
59+
60+
mgmt-bo: connected to bond-mgmt
61+
"mgmt-bo"
62+
bond, 5C:B9:01:89:C2:F5, sw, mtu 1500
63+
master mgmt-br
64+
65+
mgmt-br: connected to bridge-mgmt
66+
"mgmt-br"
67+
bridge, 5C:B9:01:89:C2:F5, sw, mtu 1500
68+
69+
eno50: connected to bond-slave-eno50
70+
"Intel 82599ES SFI/SFP+"
71+
ethernet (ixgbe), 5C:B9:01:89:C2:F5, hw, sriov, mtu 1500
72+
master mgmt-bo
73+
74+
...
7275
```
7376
7477
- Physical NICs: You can use the command `ip link` to retrieve related information, including the state of each NIC and the corresponding master (if applicable).

0 commit comments

Comments
 (0)