Home Lab OpenShift using Agent Based Installer

Configuration

Install the tools. Here’s the setup for a Fedora/RHEL box. 

mkdir ~/bin

wget -O /tmp/ocp/openshift-client-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz
tar -xvf /tmp/ocp/openshift-client-linux.tar.gz -C ~/bin

wget -O /tmp/ocp/openshift-install-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-install-linux.tar.gz
tar -xvf /tmp/ocp/openshift-install-linux.tar.gz -C ~/bin

sudo dnf install /usr/bin/nmstatectl -y

Install the Cluster

One of the things about the OpenShift agent based installer is that when you provide it configs, it actually destroys them. The way I get around this, if I am iterating in my installs, is to do the following. 

rm -rf install
mkdir install
cp agent-config.yaml install-config.yaml install
openshift-install agent create image --dir=install --log-level=debug

Then once you boot the machines with the iso created, then run the following commands to complete the install. 

openshift-install agent wait-for bootstrap-complete --dir=install --log-level=debug

...and when that is complete...

openshift-install agent wait-for install-complete --dir=install --log-level=debug

Config Files

Here’s the configs for my homelab setup, which is a bare-metal 3 node cluster on some Intel NUCs. 

apiVersion: v1
baseDomain: lab.snimmo.com
metadata:
  name: ocp
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.3.0.0/24
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  baremetal:
    apiVIP: 10.3.0.9
    ingressVIP: 10.3.0.10
pullSecret: '<redacted>'
sshKey: 'ssh-ed25519 AAAA....ZcOs'

And the agent based install. 

apiVersion: v1alpha1
kind: AgentConfig
metadata:
  name: ocp
rendezvousIP: 10.3.0.11
hosts:
  - hostname: nuc1
    role: master
    interfaces:
      - name: enp57s0u1c2
        macAddress: F8:E4:3B:BE:2E:16
    rootDeviceHints:
      deviceName: "/dev/sda"
    networkConfig:
      interfaces:
        - name: enp57s0u1c2
          type: ethernet
          state: up
          mac-address: F8:E4:3B:BE:2E:16
          ipv4:
            enabled: true
            address:
              - ip: 10.3.0.11
                prefix-length: 24
            dhcp: false
      dns-resolver:
        config:
          server:
            - 10.3.0.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.3.0.1
            next-hop-interface: enp57s0u1c2
            table-id: 254
  - hostname: nuc2
    role: master
    interfaces:
      - name: eno1
        macAddress: 1C:69:7A:61:5E:AE
    rootDeviceHints:
      deviceName: "/dev/sda"
    networkConfig:
      interfaces:
        - name: eno1
          type: ethernet
          state: up
          mac-address: 1C:69:7A:61:5E:AE
          ipv4:
            enabled: true
            address:
              - ip: 10.3.0.12
                prefix-length: 24
            dhcp: false
      dns-resolver:
        config:
          server:
            - 10.3.0.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.3.0.1
            next-hop-interface: eno1
            table-id: 254
  - hostname: nuc3
    role: master
    interfaces:
      - name: eno1
        macAddress: 1C:69:7A:61:64:F4
    rootDeviceHints:
      deviceName: "/dev/sda"
    networkConfig:
      interfaces:
        - name: eno1
          type: ethernet
          state: up
          mac-address: 1C:69:7A:61:64:F4
          ipv4:
            enabled: true
            address:
              - ip: 10.3.0.13
                prefix-length: 24
            dhcp: false
      dns-resolver:
        config:
          server:
            - 10.3.0.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.3.0.1
            next-hop-interface: eno1
            table-id: 254

Alternate Install – Bond with Single NIC

In the following agent config file, I wanted to create a bit more of an “enterprisey” setup, so I created a bond interface, which is active-passive, but with only a single ethernet interface backing it up because my NUCs don’t have dual NICs. More for documentation and to see if it would work. 

apiVersion: v1alpha1
kind: AgentConfig
metadata:
  name: ocp
rendezvousIP: 10.3.0.11
hosts:
  - hostname: nuc1
    role: master
    interfaces:
      - name: enp57s0u1c2
        macAddress: F8:E4:3B:BE:2E:16
    rootDeviceHints:
      deviceName: "/dev/sda"
    networkConfig:
      interfaces:
        - name: enp57s0u1c2
          type: ethernet
          state: up
          mac-address: F8:E4:3B:BE:2E:16
        - name: bond0
          type: bond
          state: up
          ipv4:
            enabled: true
            address:
              - ip: 10.3.0.11
                prefix-length: 24
            dhcp: false
          link-aggregation:
            mode: active-backup
            port:
              - enp57s0u1c2
            options:
              miimon: '100'
              primary: enp57s0u1c2
      dns-resolver:
        config:
          server:
            - 10.3.0.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.3.0.1
            next-hop-interface: bond0
            table-id: 254
  - hostname: nuc2
    role: master
    interfaces:
      - name: eno1
        macAddress: 1C:69:7A:61:5E:AE
    rootDeviceHints:
      deviceName: "/dev/sda"
    networkConfig:
      interfaces:
        - name: eno1
          type: ethernet
          state: up
          mac-address: 1C:69:7A:61:5E:AE
        - name: bond0
          type: bond
          state: up
          ipv4:
            enabled: true
            address:
              - ip: 10.3.0.12
                prefix-length: 24
            dhcp: false
          link-aggregation:
            mode: active-backup
            port:
              - eno1
            options:
              miimon: '100'
              primary: eno1
      dns-resolver:
        config:
          server:
            - 10.3.0.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.3.0.1
            next-hop-interface: bond0
            table-id: 254
  - hostname: nuc3
    role: master
    interfaces:
      - name: eno1
        macAddress: 1C:69:7A:61:64:F4
    rootDeviceHints:
      deviceName: "/dev/sda"
    networkConfig:
      interfaces:
        - name: eno1
          type: ethernet
          state: up
          mac-address: 1C:69:7A:61:64:F4
        - name: bond0
          type: bond
          state: up
          ipv4:
            enabled: true
            address:
              - ip: 10.3.0.13
                prefix-length: 24
            dhcp: false
          link-aggregation:
            mode: active-backup
            port:
              - eno1
            options:
              miimon: '100'
              primary: eno1
      dns-resolver:
        config:
          server:
            - 10.3.0.2
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 10.3.0.1
            next-hop-interface: bond0
            table-id: 254

Here’s what the network interfaces look like on the host after this type of install. 

[core@nuc2 ~]$ nmcli c sh
NAME                  UUID                                  TYPE           DEVICE 
ovs-if-br-ex          fd87a8ba-8d32-4e76-bc14-505cf66ebc51  ovs-interface  br-ex  
br-ex                 b7786e2d-0da2-4903-909e-7e4ffa627402  ovs-bridge     br-ex  
eno1-slave-ovs-clone  dd89feca-823f-4d61-8866-109c8fd68f7e  ethernet       eno1   
ovs-if-phys0          6193d237-4d3f-43b4-9040-bb0675896215  bond           bond0  
ovs-port-br-ex        07fb551e-af0d-4e48-95cd-4874afa87ac7  ovs-port       br-ex  
ovs-port-phys0        d8e8f512-91af-4404-a6df-2c4a85e85115  ovs-port       bond0  
lo                    b9378fb3-1369-4ac7-ada1-1f62564637b1  loopback       lo     
bond0                 925b4a95-2de0-5b2d-bcf5-8b684a7e9cb4  bond           --     
eno1                  ff8b47e2-9b1c-5e04-9f8b-34671372837c  ethernet       --