Jul 29, 2021 4 min read

Understanding the OCP Install Processes (Part 2)

Here is where I will attempt to gather all of the information that happens from the point-in-time when we run the “openshift-install create ignition-configs” and turn on the bootstrap server.

Here is what is configured in the bootstrap.ign file

Users

Core users with SSH keys

Storage

FileName	Description	Contents	Link
/etc/containers/registries.conf	Empty at Install Time
/etc/ignition-machine-config-encapsulated.json	Passes Desired FIPs state (on/off) and additional kernel arguments
/etc/motd	Tells us that this bootstrap node and provides command for running journalctl to check status
/etc/pki/ca-trust/source/anchors/ca.crt	Empty (perhaps for proxy)
/etc/profile.d/proxy.sh	Empty
/etc/systemd/system.conf.d/10-default-env-godebug.conf	??
/etc/systemd/system.conf.d/10-default-env.conf	??
/root/.docker/config.json	Pull secrets for quay.io/registrity.redhatio.io/etc
/usr/local/bin/approve-csr.sh	Automatically approves all CSR requests up to the point of the bootstrap being finished
/usr/local/bin/bootkube.sh	See below for description
/usr/local/bin/crio-configure.sh
/usr/local/bin/installer-gather.sh	Helps with troubleshooting failed cluster install
/usr/local/bin/installer-masters-gather.sh	Helps with troubleshooting failed masters
/usr/local/bin/kubelet-pause-image.sh	Need to get more info??
/usr/local/bin/release-image-download.sh	Download release-image Currently quay.io/openshift-release-dev/ocp-release@sha256:999a6a4bd731075e389ae601b373194c6cb2c7b4dadd1ad06ef607e86476b129
/usr/local/bin/release-image.sh	Gets release image digest and environment variables for pods
/usr/local/bin/report-progress.sh	This script gets executed as we wait-for bootstrap-complete
Static Objects in /opt/openshift	More info later

Systemd Units

UnitName	Description	Contents	Link
approve-csr.service
bootkube.service
chown-gatewayd-key.service
crio-configure.service
kubelet.service
progress.service
release-image.service
systemd-journal-gatewayd.service
systemd-journal-gatewayd.socket

Detailed description of bootstrap process:

<This is work in progress>

1. release-image-download service runs as one-shot service. This downloads the main OCP release image which contains info for all the containers that need to run.
2. bootkube.sh service runs

a. mkdir –parents /etc/kubernetes/{manifests,bootstrap-configs,bootstrap-manifests}

b. release-image.sh is a function (image_for) that gets the specific image ID for the following:

MACHINE_CONFIG_OPERATOR_IMAGE=$(image_for machine-config-operator)
MACHINE_CONFIG_OSCONTENT=$(image_for machine-os-content)
MACHINE_CONFIG_ETCD_IMAGE=$(image_for etcd)
MACHINE_CONFIG_INFRA_IMAGE=$(image_for pod)
CLUSTER_ETCD_OPERATOR_IMAGE=$(image_for cluster-etcd-operator)
CONFIG_OPERATOR_IMAGE=$(image_for cluster-config-operator)
KUBE_APISERVER_OPERATOR_IMAGE=$(image_for cluster-kube-apiserver-operator)
KUBE_CONTROLLER_MANAGER_OPERATOR_IMAGE=$(image_for cluster-kube-controller-manager-operator)
KUBE_SCHEDULER_OPERATOR_IMAGE=$(image_for cluster-kube-scheduler-operator)
INGRESS_OPERATOR_IMAGE=$(image_for cluster-ingress-operator)
CLOUD_CREDENTIAL_OPERATOR_IMAGE=$(image_for cloud-credential-operator)
OPENSHIFT_HYPERKUBE_IMAGE=$(image_for hyperkube)
OPENSHIFT_CLUSTER_POLICY_IMAGE=$(image_for cluster-policy-controller)
CLUSTER_BOOTSTRAP_IMAGE=$(image_for cluster-bootstrap)
KEEPALIVED_IMAGE=$(image_for keepalived-ipfailover || echo “no-keepalived-image”)
COREDNS_IMAGE=$(image_for coredns)
MDNS_PUBLISHER_IMAGE=$(image_for mdns-publisher)
HAPROXY_IMAGE=$(image_for haproxy-router)
BAREMETAL_RUNTIMECFG_IMAGE=$(image_for baremetal-runtimecfg)

c. move /opt/openshift/manifests,openshift to /etc/kubernetes directory. This is marked done in the /opt/openshift directory when openshift-manifests.done exists.

d. cvo-bootstrap

bootstrap
bootstrap/bootstrap-pod.yaml
manifests
manifests/0000_00_cluster-version-operator_00_namespace.yaml
manifests/0000_00_cluster-version-operator_01_clusteroperator.crd.yaml
manifests/0000_00_cluster-version-operator_01_clusterversion.crd.yaml
manifests/0000_00_cluster-version-operator_02_roles.yaml
manifests/0000_00_cluster-version-operator_03_deployment.yaml
manifests/0001_00_cluster-version-operator_03_service.yaml

e. etcd-operator started as static pod

bootstrap-manifests
bootstrap-manifests/secrets
bootstrap-manifests/secrets/etcd-all-serving
bootstrap-manifests/secrets/etcd-all-serving/etcd-serving-bootstrap.hub.ocp-poc-demo.com.crt
bootstrap-manifests/secrets/etcd-all-serving/etcd-serving-bootstrap.hub.ocp-poc-demo.com.key
bootstrap-manifests/secrets/etcd-all-peer
bootstrap-manifests/secrets/etcd-all-peer/etcd-peer-bootstrap.hub.ocp-poc-demo.com.crt
bootstrap-manifests/secrets/etcd-all-peer/etcd-peer-bootstrap.hub.ocp-poc-demo.com.key
bootstrap-manifests/etcd-member-pod.yaml
config
manifests
manifests/00_etcd-endpoints-cm.yaml
manifests/00_openshift-etcd-ns.yaml

cp etcd-bootstrap/manifests/* manifests/

cp etcd-bootstrap/bootstrap-manifests/etcd-member-pod.yaml /etc/kubernetes/manifests/
mkdir –parents /etc/kubernetes/static-pod-resources/etcd-member

cp tls/etcd-ca-bundle.crt /etc/kubernetes/static-pod-resources/etcd-member/ca.crt

cp –recursive etcd-bootstrap/bootstrap-manifests/secrets/etcd-all-serving /etc/kubernetes/static-pod-resources/etcd-member

cp –recursive etcd-bootstrap/bootstrap-manifests/secrets/etcd-all-peer /etc/kubernetes/static-pod-resources/etcd-member

f. cluster-config-operator

bootstrap-manifests
bootstrap-manifests/cloud-credential-operator-pod.yaml
manifests
manifests/cco-cloudcredential_v1_operator_config_custresdef.yaml
manifests/cco-cloudcredential_v1_credentialsrequest_crd.yaml
manifests/cco-namespace.yaml
manifests/cco-operator-config.yaml

g. kube-apiserver

bootstrap-manifests
bootstrap-manifests/kube-apiserver-pod.yaml
config
manifests
manifests/00_openshift-kube-apiserver-ns.yaml
manifests/secret-kube-apiserver-to-kubelet-signer.yaml
manifests/secret-aggregator-client-signer.yaml
manifests/00_openshift-kube-apiserver-operator-ns.yaml
manifests/configmap-csr-controller-ca.yaml
manifests/configmap-sa-token-signing-certs.yaml
manifests/secret-bound-sa-token-signing-key.yaml
manifests/secret-service-network-serving-signer.yaml
manifests/cluster-role-binding-kube-apiserver.yaml
manifests/cluster-role-kube-apiserver.yaml
manifests/secret-control-plane-client-signer.yaml
manifests/secret-loadbalancer-serving-signer.yaml
manifests/secret-localhost-serving-signer.yaml
manifests/configmap-admin-kubeconfig-client-ca.yaml
manifests/configmap-kubelet-bootstrap-kubeconfig-ca.yaml

h. kube-controller-manager

bootstrap-manifests
bootstrap-manifests/kube-controller-manager-pod.yaml
config
cpc-config
manifests
manifests/00_namespace-security-allocation-controller-clusterrole.yaml
manifests/00_namespace-security-allocation-controller-clusterrolebinding.yaml
manifests/00_openshift-kube-controller-manager-ns.yaml
manifests/00_openshift-kube-controller-manager-operator-ns.yaml
manifests/secret-csr-signer-signer.yaml
manifests/secret-initial-kube-controller-manager-service-account-private-key.yaml

i. kube-scheduler

bootstrap-manifests
bootstrap-manifests/kube-scheduler-pod.yaml
config
manifests
manifests/00_openshift-kube-scheduler-ns.yaml

j. ingress operator

cluster-ingress-00-custom-resource-definition.yaml
cluster-ingress-00-namespace.yaml

k. machine-config-operator

bootstrap
bootstrap/manifests
bootstrap/manifests/machineconfigcontroller-controllerconfig.yaml
bootstrap/manifests/master.machineconfigpool.yaml
bootstrap/manifests/worker.machineconfigpool.yaml
bootstrap/manifests/machineconfigcontroller-pull-secret
bootstrap/machineconfigoperator-bootstrap-pod.yaml
manifests
manifests/csr-bootstrap-role-binding.yaml
manifests/kube-apiserver-serving-ca-configmap.yaml

l. cloud credential operator

Keith Calligan