Setting up a lightweight Kubernetes cluster with K3s and Fedora CoreOS.

10 min readMar 2, 2021

I have been playing around with my OpenStack cloud at home and figured I would take shot at setting up Kubernetes to put it through it’s paces. I’m interested in playing around with Kubeflow, Spark, Luigi, Jupyterhub to see how they work.

These days there are an overwhelming number of ways to install Kubernetes and OS’s to run it on. The “cloud” I am running is comprised of 2 desktops. Since I don’t have a huge amount of resources to work with I would like to keep things as minimal as possible.

In order to keep this a minimal setup I am working with Kubernetes K3s.

The basic process of getting it up and running only takes a few steps.

Master Node

[core@host-10-10-3-171 ~]$ sudo -i
[core@host-10-10-3-171 ~]$ rpm-ostree install [core@host-10-10-3-171 ~]$ https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm
[core@host-10-10-3-171 ~]$ systemctl reboot[core@host-10-10-3-171 ~]$ sudo -i
[core@host-10-10-3-171 ~]$ export K3S_KUBECONFIG_MODE="644"[core@host-10-10-3-171 ~]$ curl -sfL https://get.k3s.io | sh -

Worker Nodes

[core@host-10-10-3-171 ~]$ sudo -i
[core@host-10-10-3-171 ~]$ rpm-ostree install [core@host-10-10-3-171 ~]$ https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm
[core@host-10-10-3-171 ~]$ systemctl reboot[core@host-10-10-3-171 ~]$  export K3S_KUBECONFIG_MODE="644"
[core@host-10-10-3-171 ~]$  export K3S_URL="https://MasterIP:6443"
[core@host-10-10-3-171 ~]$  export K3S_TOKEN="******"[core@host-10-10-3-171 ~]$  curl -sfL https://get.k3s.io | sh -

For the OS I saw a lot of recommendations to stick to something like CentOS or Ubuntu since they are a well supported known quantity. In my case I would rather not run a full Linux OS for my nodes. I considered a few options. Flatcar looks cool and if I had been a CoreOS user I would certainly use it. I looked into Rancher OS, Alpine and a few others before deciding to mess around with Fedora CoreOS. So many of these cloud focused distributions seem to be ending development or getting acquired and changing course in a way that doesn’t instill confidence for long term support. Fedora CoreOS may or may not be a long term solution but seems like a good place to start for now.

Fedora CoreOS is an auto updating immutable operating system. This means that when a change is made we don’t modify the running OS we replace it with a new version.

Simple enough I download decompress and setup the image in OpenStack.

wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210201.3.0/x86_64/fedora-coreos-33.20210201.3.0-openstack.x86_64.qcow2.xzZIP=$(ls fedora-coreos-*.qcow2.xz)xz — decompress $ZIPFILE=${ZIP:0:-3}IMAGE=${FILE:0:-6}openstack image create --disk-format=qcow2 --min-disk=10 --min-ram=2 --file=$FILE $IMAGE

In order to configure Fedora CoreOS Ignition configuration files are loaded at boot time. To simplify the process Ignition files can be written in a more human readable yaml and then converted before use.

I found an excellent article by Leonardo Murillo at murillodigital.com which is linked in the references at the bottom of this page. Leonardo had a bit of a different use case. I was able to easily adapt his script suit my needs.

https://github.com/murillodigital/experiments/blob/master/k3s%2Bcoreos/ignition/k3s-autoinstall.fcc

Starting at line 68 I made the following changes for my master node.

- path: /usr/local/bin/run-k3s-installer
      mode: 0755
      contents:
        inline: |
          #!/usr/bin/env sh
          main() {
            export K3S_KUBECONFIG_MODE="644"
            curl -sfL https://get.k3s.io | sh -
            return 0

For the compute nodes I include the steps for the master node IP and authentication token.

- path: /usr/local/bin/run-k3s-installer
      mode: 0755
      contents:
        inline: |
          #!/usr/bin/env sh
          main() {
            export K3S_KUBECONFIG_MODE="644"
            #Get IP 10.10 from "nova list"
            export K3S_URL="https://MasterIP:6443"
            #Get token from master with "sudo cat /var/lib/rancher/k3s/server/node-token"
            export K3S_TOKEN="*********"
            curl -sfL https://get.k3s.io | sh -
            return 0

In practice I run the yaml script through the transplier.

stevex0r@gunstar:/etc/openstack/fedoracoreos$ docker run -i — rm quay.io/coreos/fcct:release — pretty — strict < fcc/k3s-autoinstall-master.fcc > ign/k3s-autoinstall-master.ign

I created the following script to launch my master.

stevex0r@gunstar:/etc/openstack/fedoracoreos$ cat launcher-master.sh 
CLUSTER_NAME=test
IMAGE=fedora-coreos-33.20210201.3.0-openstack.x86_64
OPENSTACK_NETWORK=kubernetes
OPENSTACK_KEYPAIR=smohr
OPENSTACK_FLAVOR=m2.small
OPENSTACK_INSTANCE_NAME=k3-$CLUSTER_NAME-master
OPENSTACK_VOLUME_SIZE=10
openstack server create            \
     --key-name=$OPENSTACK_KEYPAIR \
     --network=$OPENSTACK_NETWORK  \
     --flavor=$OPENSTACK_FLAVOR    \
     --image=$IMAGE                \
     --user-data ./ign/k3s-autoinstall-master.ign    \
     --boot-from-volume $OPENSTACK_VOLUME_SIZE  \
     $OPENSTACK_INSTANCE_NAME

Launch the master.

[root@gunstar fedoracoreos(keystone_admin)]# ./launcher-master.sh+-------------------------------------+--------------------------------------+| Field                               | Value                                |+-------------------------------------+--------------------------------------+| OS-DCF:diskConfig                   | MANUAL                               || OS-EXT-AZ:availability_zone         |                                      || OS-EXT-SRV-ATTR:host                | None                                 || OS-EXT-SRV-ATTR:hypervisor_hostname | None                                 || OS-EXT-SRV-ATTR:instance_name       |                                      || OS-EXT-STS:power_state              | NOSTATE                              || OS-EXT-STS:task_state               | scheduling                           || OS-EXT-STS:vm_state                 | building                             || OS-SRV-USG:launched_at              | None                                 || OS-SRV-USG:terminated_at            | None                                 || accessIPv4                          |                                      || accessIPv6                          |                                      || addresses                           |                                      || adminPass                           | ut3ck9vL25ci                         || config_drive                        |                                      || created                             | 2021-03-01T20:32:36Z                 || flavor                              | m2.small (6)                         || hostId                              |                                      || id                                  | 13b46a50-6d1c-4d12-a3e9-e084dc07e767 || image                               | N/A (booted from volume)             || key_name                            | smohr                                || name                                | k3-test-master                       || progress                            | 0                                    || project_id                          | 01d4cc12ae4641a7b91bdd8f0bd4c11e     || properties                          |                                      || security_groups                     | name='default'                       || status                              | BUILD                                || updated                             | 2021-03-01T20:32:36Z                 || user_id                             | 1b29256ea2d04eb29b701985908d7998     || volumes_attached                    |                                      |+-------------------------------------+--------------------------------------+

After a few minutes my master node is up and running.

[root@gunstar fedoracoreos(keystone_admin)]# nova list+--------------------------------------+----------------+--------+------------+-------------+------------------------+| ID                                   | Name           | Status | Task State | Power State | Networks               |+--------------------------------------+----------------+--------+------------+-------------+------------------------+| 13b46a50-6d1c-4d12-a3e9-e084dc07e767 | k3-test-master | ACTIVE | -          | Running     | kubernetes=10.10.3.106 |+--------------------------------------+----------------+--------+------------+-------------+------------------------+

After a few minutes my master node is up and running.

I assign a floating IP so that I can reach the master with SSH.

[root@gunstar fedoracoreos(keystone_admin)]# openstack server add floating ip k3-test-master 192.168.1.95

Get the token and the internal IP address of the master node.

[root@gunstar fedoracoreos(keystone_admin)]# ssh core@192.168.1.95 sudo cat /var/lib/rancher/k3s/server/node-tokenK10e58628aec6decf767c249cd53bd329e98eff2f6a5d5abd93d81da3da161167e9::server:f1658d45a5eb9a80a3db8c72603a6ee5[root@gunstar fedoracoreos(keystone_admin)]# nova list+ — — — — — — — — — — — — — — — — — — — + — — — — — — — — + — — — — + — — — — — — + — — — — — — -+ — — — — — — — — — — — — — — — — — — — +| ID | Name | Status | Task State | Power State | Networks |+ — — — — — — — — — — — — — — — — — — — + — — — — — — — — + — — — — + — — — — — — + — — — — — — -+ — — — — — — — — — — — — — — — — — — — +| 13b46a50–6d1c-4d12-a3e9-e084dc07e767 | k3-test-master | ACTIVE | — | Running | kubernetes=10.10.3.106, 192.168.1.95 |+ — — — — — — — — — — — — — — — — — — — + — — — — — — — — + — — — — + — — — — — — + — — — — — — -+ — — — — — — — — — — — — — — — — — — — +

I add the IP and token to my to my node ignition file.

- path: /usr/local/bin/run-k3s-installer
      mode: 0755
      contents:
        inline: |
          #!/usr/bin/env sh
          main() {
            export K3S_KUBECONFIG_MODE="644"
            export K3S_URL="https://10.10.3.106:6443"
            export K3S_TOKEN="K10e58628aec6decf767c249cd53bd329e98eff2f6a5d5abd93d81da3da161167e9::server:f1658d45a5eb9a80a3db8c72603a6ee5"curl -sfL https://get.k3s.io | sh -
            return 0

Run the ignition file through the transpiler.

stevex0r@gunstar:/etc/openstack/fedoracoreos$ docker run -i — rm quay.io/coreos/fcct:release — pretty — strict < fcc/k3s-autoinstall-node.fcc > ign/k3s-autoinstall-node.ign

My node launcher script uses the newly created ignition script and will run a for loop to launch 3 nodes.

[root@gunstar fedoracoreos(keystone_admin)]# cat launcher-node.sh
#!/bin/bash
for i in {1..3}
do
   CLUSTER_NAME=kubeflow
   IMAGE=fedora-coreos-33.20210201.3.0-openstack.x86_64
   OPENSTACK_NETWORK=kubernetes
   OPENSTACK_KEYPAIR=smohr
   OPENSTACK_FLAVOR=m2.small
   OPENSTACK_INSTANCE_NAME=k3-$CLUSTER_NAME-node${i}
   OPENSTACK_VOLUME_SIZE=10
   openstack server create            \
        --key-name=$OPENSTACK_KEYPAIR \
        --network=$OPENSTACK_NETWORK  \
        --flavor=$OPENSTACK_FLAVOR    \
        --image=$IMAGE                \
        --user-data ./ign/k3s-autoinstall-node.ign    \
        --boot-from-volume $OPENSTACK_VOLUME_SIZE       \
        $OPENSTACK_INSTANCE_NAME
done[root@gunstar fedoracoreos(keystone_admin)]# ./launcher-node.sh+ — — — — — — — — — — — — — — — — — — -+ — — — — — — — — — — — — — — — — — — — +| Field | Value |+ — — — — — — — — — — — — — — — — — — -+ — — — — — — — — — — — — — — — — — — — +| OS-DCF:diskConfig | MANUAL || OS-EXT-AZ:availability_zone | || OS-EXT-SRV-ATTR:host | None || OS-EXT-SRV-ATTR:hypervisor_hostname | None || OS-EXT-SRV-ATTR:instance_name | || OS-EXT-STS:power_state | NOSTATE || OS-EXT-STS:task_state | scheduling || OS-EXT-STS:vm_state | building || OS-SRV-USG:launched_at | None || OS-SRV-USG:terminated_at | None || accessIPv4 | || accessIPv6 | || addresses | || adminPass | buoaac3QqFyR || config_drive | || created | 2021–03–01T22:46:25Z || flavor | m2.small (6) || hostId | || id | 214c59ef-36f4–4cab-b590-d55bacc57b92 || image | N/A (booted from volume) || key_name | smohr || name | k3-kubeflow-node1 || progress | 0 || project_id | 01d4cc12ae4641a7b91bdd8f0bd4c11e || properties | || security_groups | name=’default’ || status | BUILD || updated | 2021–03–01T22:46:25Z || user_id | 1b29256ea2d04eb29b701985908d7998 || volumes_attached | |+ — — — — — — — — — — — — — — — — — — -+ — — — — — — — — — — — — — — — — — — — +

I now have a running Kubernetes cluster.

Lets log in and take a look around.

[root@gunstar stevex0r(keystone_admin)]# ssh core@192.168.1.95

List the nodes.

[core@host-10–10–3–106 ~]$ kubectl get nodesNAME STATUS ROLES AGE VERSIONhost-10–10–3–106 Ready control-plane,master 25h v1.20.4+k3s1host-10–10–3–161 Ready <none> 23h v1.20.4+k3s1host-10–10–3–34 Ready <none> 23h v1.20.4+k3s1host-10–10–3–3 Ready <none> 23h v1.20.4+k3s1

Lets run a Hello World service.

[core@host-10–10–3–106 ~]$ kubectl apply -f https://k8s.io/examples/service/access/hello-application.yamldeployment.apps/hello-world created

List the deployment

[core@host-10–10–3–106 ~]$ kubectl get deployments hello-worldNAME READY UP-TO-DATE AVAILABLE AGEhello-world 0/2 2 0 9s[core@host-10–10–3–106 ~]$ kubectl describe deployments hello-worldName: hello-worldNamespace: defaultCreationTimestamp: Tue, 02 Mar 2021 19:46:21 +0000Labels: <none>Annotations: deployment.kubernetes.io/revision: 1Selector: run=load-balancer-exampleReplicas: 2 desired | 2 updated | 2 total | 0 available | 2 unavailableStrategyType: RollingUpdateMinReadySeconds: 0RollingUpdateStrategy: 25% max unavailable, 25% max surgePod Template:Labels: run=load-balancer-exampleContainers:hello-world:Image: gcr.io/google-samples/node-hello:1.0Port: 8080/TCPHost Port: 0/TCPEnvironment: <none>Mounts: <none>Volumes: <none>Conditions:Type Status Reason — — — — — — — — Available False MinimumReplicasUnavailableProgressing True ReplicaSetUpdatedOldReplicaSets: <none>NewReplicaSet: hello-world-59966754c9 (2/2 replicas created)Events:Type Reason Age From Message — — — — — — — — — — — — -Normal ScalingReplicaSet 14s deployment-controller Scaled up replica set hello-world-59966754c9 to 2

Take a look at the 2 replicas

[core@host-10–10–3–106 ~]$ kubectl get replicasetsNAME DESIRED CURRENT READY AGEhello-world-59966754c9 2 2 0 26s[core@host-10–10–3–106 ~]$ kubectl describe replicasetsName: hello-world-59966754c9Namespace: defaultSelector: pod-template-hash=59966754c9,run=load-balancer-exampleLabels: pod-template-hash=59966754c9run=load-balancer-exampleAnnotations: deployment.kubernetes.io/desired-replicas: 2deployment.kubernetes.io/max-replicas: 3deployment.kubernetes.io/revision: 1Controlled By: Deployment/hello-worldReplicas: 2 current / 2 desiredPods Status: 0 Running / 2 Waiting / 0 Succeeded / 0 FailedPod Template:Labels: pod-template-hash=59966754c9run=load-balancer-exampleContainers:hello-world:Image: gcr.io/google-samples/node-hello:1.0Port: 8080/TCPHost Port: 0/TCPEnvironment: <none>Mounts: <none>Volumes: <none>Events:Type Reason Age From Message — — — — — — — — — — — — -Normal SuccessfulCreate 32s replicaset-controller Created pod: hello-world-59966754c9–4gcwrNormal SuccessfulCreate 32s replicaset-controller Created pod: hello-world-59966754c9-cn4cp

Expose the service so that we can reach it from our network.

[core@host-10–10–3–106 ~]$ kubectl expose deployment hello-world — type=NodePort — name=hw-serviceservice/hw-service exposed[core@host-10–10–3–106 ~]$ kubectl describe services hw-serviceName: hw-serviceNamespace: defaultLabels: <none>Annotations: <none>Selector: run=load-balancer-exampleType: NodePortIP Families: <none>IP: 10.43.101.25Port: <unset> 8080/TCPTargetPort: 8080/TCPNodePort: <unset> 30237/TCPEndpoints: <none>Session Affinity: NoneExternal Traffic Policy: ClusterEvents: <none>[core@host-10–10–3–106 ~]$ kubectl get pods — selector=”run=load-balancer-example” — output=wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEShello-world-59966754c9-cn4cp 0/1 ContainerCreating 0 96s <none> host-10–10–3–3 <none> <none>hello-world-59966754c9–4gcwr 0/1 ContainerCreating 0 96s <none> host-10–10–3–34 <none> <none>

Our service is reachable inside the Kubernetes network. In OpenStack we need to assign a floating IP address and open a security group to allow access from our network to the port on one of our replica worker nodes.

[root@gunstar stevex0r(keystone_admin)]# openstack server add floating ip k3-test-node2 192.168.1.93[root@gunstar stevex0r(keystone_admin)]#  openstack security group rule create helloworld — protocol tcp — dst-port 8080:8080 — remote-ip 0.0.0.0/0
[root@gunstar stevex0r(keystone_admin)]#  openstack server add security group k3-kubeflow-node2 helloworld

Open a browser and visit http://192.168.1.93:8080 to access the Hello World message.

Whats next?

Install Helm and have some fun!

[core@host-10-10-3-106 ~]$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
[core@host-10-10-3-106 yaml]$ helm search hub

References

https://docs.fedoraproject.org/id/fedora-coreos/provisioning-openstack/

https://devopstales.github.io/home/k3s-fcos/

https://k3s.io/

https://www.murillodigital.com/tech_talk/k3s_in_coreos/

https://kubernetes.io/docs/tasks/access-application-cluster/service-access-application-cluster/

https://stevex0r.medium.com/openstack-homelab-installation-75ad6d798994

Setting up a lightweight Kubernetes cluster with K3s and Fedora CoreOS.

Sign up to discover human stories that deepen your understanding of the world.

Free

Membership

Written by Steve Mohr

No responses yet