# 8. State Persistence # Introduction to Docker Storage In this lesson, we explore how storage works in Docker and how these foundational concepts apply to Kubernetes storage management. Understanding Docker storage is essential for grasping the more advanced storage mechanisms in Kubernetes. When working with Docker, two core storage concepts are essential to understand: * Storage Drivers * Volume Driver Plugins ![The image is a slide titled "Docker Storage," featuring two categories: "Storage Drivers" and "Volume Drivers."](https://kodekloud.com/kk-media/image/upload/v1752871315/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Introduction-to-Docker-Storage/frame_30.jpg align="left") In the upcoming section, we focus on storage drivers—a topic thoroughly covered in the [**Docker Training Course for the Absolute Beginner**](https://learn.kodekloud.com/user/courses/docker-training-course-for-the-absolute-beginner). If you are already familiar with this material, feel free to skip ahead or use this section as a refresher. **Note** After discussing storage drivers, we will explore volume drivers to complete your understanding of Docker storage. # Storage in Docker Welcome to this lesson on advanced Docker concepts. In this guide, we will explore Docker storage drivers, file systems, and how Docker manages local filesystem data for images, containers, and volumes. ## Docker Data Storage on the Host When Docker is installed, it organizes data within the `/var/lib/docker` directory. This folder contains several subdirectories such as `aufs`, `containers`, `images`, and `volumes`. Each subdirectory serves a specific role in Docker’s architecture: * **containers:** Stores all files related to running containers. * **images:** Contains stored images. * **volumes:** Holds data for persistent storage created by containers. ![The image shows a file system structure for Docker, highlighting directories like "aufs," "containers," "image," and "volumes" under "/var/lib/docker."](https://kodekloud.com/kk-media/image/upload/v1752871318/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Storage-in-Docker/frame_50.jpg align="left") ## Layered Architecture in Docker Images Docker images use a layered architecture where each instruction in a Dockerfile creates a new layer capturing only the changes from the previous one. Consider the following example Dockerfile: ```plaintext FROM Ubuntu RUN apt-get update && apt-get -y install python RUN pip install flask flask-mysql COPY . /opt/source-code ENTRYPOINT FLASK_APP=/opt/source-code/app.py flask run ``` You can build this image with the following command: ```plaintext docker build -t mmumshad/my-custom-app . ``` In this example: * The **base layer** is the Ubuntu operating system. * Subsequent layers add APT packages, Python packages, application source code, and finally, the entry point configuration. Each layer only stores the differences from its predecessor. For example, although the base Ubuntu image might be around 120 MB and the APT updates add an additional 300 MB, the remaining layers are much smaller. This strategy optimizes build times and minimizes disk space usage. ### Reusing Layers for Similar Applications **Tip** If subsequent applications share many common layers, Docker will reuse the unchanged layers from its cache, significantly speeding up builds. Consider a second application similar to the first, with the same base image and dependencies but a different source file and entry point: ```plaintext FROM Ubuntu RUN apt-get update && apt-get -y install python RUN pip install flask flask-mysql COPY app2.py /opt/source-code ENTRYPOINT FLASK_APP=/opt/source-code/app2.py flask run ``` Build this image using: ```plaintext docker build -t mmumshad/my-custom-app-2 . ``` Because the first three layers are identical across both applications, Docker reuses the cached layers and only builds the new layers corresponding to the changes. ## Understanding Image and Container Layers A Docker image consists of several read-only layers: 1. **Base Layer:** The Ubuntu operating system. 2. **Packages Layer:** APT packages installed on top of Ubuntu. 3. **Dependencies Layer:** Python packages such as Flask. 4. **Source Code Layer:** Your application code included in the image. 5. **Entry Point Layer:** The layer that sets the container’s entry point. When building the image: ```plaintext docker build -t mmumshad/my-custom-app Dockerfile ``` the resulting layers remain read-only. Running a container from this image creates a new writable layer on top, which stores changes such as logs, temporary files, or modifications made during runtime. This mechanism is known as copy-on-write. Even if you modify a read-only file from the image, Docker creates a separate copy in the writable layer before applying the changes. ```plaintext docker run mmumshad/my-custom-app ``` ![The image illustrates the "Copy-on-Write" concept, showing container and image layers with "Read Write" and "Read Only" permissions for files like "app.py" and "temp.txt".](https://kodekloud.com/kk-media/image/upload/v1752871319/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Storage-in-Docker/frame_410.jpg align="left") **Important** Remember, when you remove a container, its writable layer and any associated changes will be lost. The original image remains unchanged unless it is rebuilt. ## Persisting Data with Volumes For data persistence outside the container’s ephemeral writable layer—such as database storage—use Docker volumes. ### Creating and Using Volumes First, create a volume: ```plaintext docker volume create data_volume ``` Then, mount the created volume when launching a container. For example, to store MySQL data in the volume, run: ```plaintext docker run -v data_volume:/var/lib/mysql mysql ``` If a specified volume does not exist, Docker automatically creates it. To inspect the volumes, you can list the contents of `/var/lib/docker/volumes`. ### Bind Mounts Alternatively, if you prefer using external storage (for instance, storing database files in `/data/mysql` on your host), you can use bind mounts: ```plaintext docker run -v /data/mysql:/var/lib/mysql mysql ``` This technique maps a directory on the host to the container, enabling direct access to the host’s filesystem. ### Using the --mount Option The newer `--mount` flag provides a clearer and more explicit syntax. The equivalent bind mount example using `--mount` is: ```plaintext docker run \ --mount type=bind,source=/data/mysql,target=/var/lib/mysql \ mysql ``` This syntax explicitly defines each parameter (type, source, target) and is recommended for its clarity. ## Docker Storage Drivers Docker uses storage drivers to manage layered filesystems, the creation of writable layers, and copy-on-write operations. Common storage drivers include: * AUFS * ZFS * BTRFS * Device Mapper * Overlay * Overlay2 The default storage driver varies by operating system: for example, Ubuntu typically uses AUFS, whereas Fedora or CentOS may use Device Mapper if AUFS is unavailable. Each driver offers unique performance and stability characteristics, so choose one based on your application’s requirements. ![The image lists storage drivers: AUFS, ZFS, BTRFS, Device Mapper, Overlay, and Overlay2, with a whale graphic in the background.](https://kodekloud.com/kk-media/image/upload/v1752871320/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Storage-in-Docker/frame_730.jpg align="left") For more detailed information on these storage drivers, please refer to the documentation provided in the relevant links. ## Conclusion This lesson on Docker’s storage architecture covered the fundamentals of how Docker organizes data on the host, utilizes a layered image architecture, and manages persistent data with volumes and bind mounts. Understanding these concepts is crucial for optimizing Docker builds, ensuring efficient disk usage, and managing data persistence. Thank you for reading, and we look forward to sharing more advanced Docker topics in our next lesson. # Volume Driver Plugins in Docker Docker leverages storage drivers to manage images and containers, while volume driver plugins handle data persistence. Unlike storage drivers, volumes need to be explicitly created and are managed by these dedicated plugins. **Note** The default volume driver plugin, "local," creates volumes on the Docker host and stores data at `/var/lib/docker/volumes`. Several third-party volume driver plugins further expand Docker's storage capabilities by enabling volume creation on external storage solutions. Popular plugins include: * Azure File Storage * Convoy * DigitalOcean Block Storage * Blocker * Google Compute Persistent Disks * ClusterFS * NetApp * RexRay * Portworx * VMware vSphere storage **Note** Certain volume drivers support multiple storage providers. For instance, the RexRay storage driver can provision storage on various platforms such as AWS EBS, S3, EMC storage arrays like Isilon and ScaleIO, Google Persistent Disk, and OpenStack Cinder. When running a Docker container, you can specify a volume driver—such as RexRay for AWS EBS—to provision a cloud-based volume. This approach ensures that your data remains safe even after the container exits. Below is an example command that demonstrates how to run a Docker container with a specified volume driver: ```plaintext docker run -it \ --name mysql \ --volume-driver rexray/ebs \ --mount src=ebs-vol,target=/var/lib/mysql \ mysql ``` This command creates a container named "mysql" and attaches a volume provisioned from Amazon EBS, ensuring persistent data storage in the cloud. # Volumes in Kubernetes Hello, and welcome to this lesson on persistent volumes in Kubernetes. I’m Mumshad Mannambeth, and today we’ll explore key concepts from the [**Certified Kubernetes Application Developer**](https://learn.kodekloud.com/user/courses/certified-kubernetes-application-developer-ckad) course. ## Understanding Volumes Before diving into persistent volumes, let's revisit the concept of volumes, starting with Docker. Docker containers are inherently ephemeral—they exist solely to process data and are removed once their task is done, taking with them any data stored exclusively inside the container. To ensure data persistence, a volume is attached when the container is created. The container writes data to the volume, and even if the container is removed later, the data remains intact. Similarly, in Kubernetes, Pods are transient. When a Pod is created to process data and subsequently deleted, any data stored within it is lost unless it is saved externally. By attaching a volume to the Pod, you can ensure that the data written to the volume remains persistent even after the Pod is terminated. ## A Simple Example with a Single-Node Cluster Consider a simple implementation where a Pod generates a random number between 1 and 100 and writes it to `/opt/number.out`. Note that without a persistent volume, deleting the Pod also removes the generated number. ```plaintext apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh","-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"] ``` To retain the generated number, we attach a volume that leverages host storage. In this example, the host directory `/data` is used as the storage backend. When the volume is mounted inside the container at `/opt`, any file written there will be persisted on the host, even after the Pod is removed. Below is the updated Pod specification with the volume properly configured: ```plaintext apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh", "-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"] volumeMounts: - mountPath: /opt name: data-volume volumes: - name: data-volume hostPath: path: /data type: Directory ``` In this configuration, the file `/opt/number.out` is stored on the host's `/data` directory. As a result, the data persists independently of the Pod’s lifecycle. **Important** Remember: Using the hostPath option is effective for single-node clusters, but it is not ideal for multi-node clusters, where consistent storage across nodes is required. ## Storage Options for Volumes The previous example utilized the hostPath option, which directly uses a directory on the host for persistent storage. While this works well in a single-node implementation, multi-node clusters typically require a more robust solution because the same directory structure is not guaranteed across different nodes. Kubernetes supports various storage solutions beyond hostPath, including network-based options like NFS, GlusterFS, and Flocker, as well as block storage types such as Fibre Channel, CephFS, and ScaleIO. Moreover, public cloud providers offer native persistent storage options, for example: * [**AWS Elastic Block Store (EBS)**](https://aws.amazon.com/ebs/) * [**Azure Disk**](https://azure.microsoft.com/services/storage/) * [**Google Persistent Disk**](https://cloud.google.com/compute/docs/disks) For instance, to configure an AWS EBS volume instead of using hostPath, the Pod specification is modified as follows: ```plaintext volumes: - name: data-volume awsElasticBlockStore: volumeID: fsType: ext4 ``` With this configuration, the Pod utilizes AWS EBS for persistent storage, enabling reliable data management through the cloud provider’s block storage services. ## Conclusion This lesson reviewed how volumes in Kubernetes work and how to persist data using hostPath and other storage solutions. In the next segment, we will delve into persistent volumes and explore how Kubernetes manages long-term data persistence in various environments. # Persistent Volumes Welcome to this comprehensive guide on Persistent Volumes. My name is Mumshad Mannambeth, and in this article, we walk through the concept and practical implementation of persistent volumes in Kubernetes. In previous discussions, we explored the concept of volumes where storage was configured directly within a pod definition file. For example: ```plaintext volumes: - name: data-volume awsElasticBlockStore: volumeID: fsType: ext4 ``` In larger environments with numerous users and multiple pod deployments, manually configuring storage for each pod becomes both tedious and error-prone. Every time a storage configuration change is required, it would need to be updated on every pod individually. Persistent volumes solve this problem by centralizing storage management. Administrators can create a large pool of storage, and users can request specific portions from that pool using Persistent Volume Claims (PVCs). ![The image illustrates the concept of Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) in Kubernetes, showing their relationship and allocation process.](https://kodekloud.com/kk-media/image/upload/v1752871317/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Persistent-Volumes/frame_90.jpg align="left") Users select the storage they need from this pool by using Persistent Volume Claims, simplifying the management and scalability of storage in Kubernetes environments. ## Creating a Persistent Volume To get started, let’s create a Persistent Volume using a base template. The following example demonstrates updating the API version, setting the kind to PersistentVolume, and giving it a name (here, PV-01). Under the `spec` section, the access modes—which define how a volume is to be mounted (e.g., read-only or read-write)—are specified. Kubernetes offers several modes such as `ReadOnlyMany`, `ReadWriteOnce`, and `ReadWriteMany`. Additionally, you define the capacity to reserve the necessary storage, set to 1Gi in this example. Initially, we use the `hostPath` option to utilize storage from the node's local directory. **Warning** Using `hostPath` is intended for demonstration or testing purposes only and is not recommended for production environments. ```plaintext apiVersion: v1 kind: PersistentVolume metadata: name: pv-voll spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi hostPath: path: /tmp/d ``` To create the persistent volume, run the following command: ```plaintext kubectl create -f pv-definition.yaml ``` After creation, list the persistent volumes with: ```plaintext kubectl get persistentvolume ``` The expected output should appear similar to: ```plaintext NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-voll 1Gi RWO Retain Available 3m ``` ## Using a Cloud Storage Backend For production environments, it is advisable to use a cloud storage backend rather than `hostPath`. For instance, you can configure AWS Elastic Block Store as illustrated below, where you specify both the `volumeID` and the `fsType`: ```plaintext apiVersion: v1 kind: PersistentVolume metadata: name: pv-voll spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi awsElasticBlockStore: volumeID: fsType: ext4 ``` Apply the updated configuration with: ```plaintext kubectl create -f pv-definition.yaml ``` Then verify its status using: ```plaintext kubectl get persistentvolume ``` You should see an output similar to: ```plaintext NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-voll 1Gi RWO Retain Available 3m ``` ## Conclusion This article provided an overview of how persistent volumes in Kubernetes simplify storage management by centralizing configuration. In the next section, we will delve into Persistent Volume Claims and explore how they seamlessly allocate storage from the available persistent volumes. # Persistent Volume Claims Welcome to this lesson on Persistent Volume Claims (PVCs) in Kubernetes. In this article, we will walk through the process of creating a PVC, explain how Kubernetes binds PVCs to Persistent Volumes (PVs), and discuss reclaim policies for managing storage efficiently. Before you create a PVC, ensure that one or more persistent volumes are available in your cluster. Remember, persistent volumes and persistent volume claims are separate Kubernetes objects. Typically, an administrator creates a pool of persistent volumes, while a user creates PVCs to request and access that storage. When a PVC is created, Kubernetes searches for an appropriate PV that meets the requested capacity and matching properties such as access modes, volume modes, and storage class. Each PVC is bound to a single PV based on a best-match algorithm. The diagram below illustrates the relationship between PVCs and PVs: ![The image illustrates the concept of Persistent Volume Claims (PVC) and Persistent Volumes (PV) in Kubernetes, showing their relationship with color-coded blocks.](https://kodekloud.com/kk-media/image/upload/v1752871316/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Persistent-Volume-Claims/frame_30.jpg align="left") If multiple persistent volumes could satisfy a claim’s requirements, labels and selectors can be used to bind a claim to a specific volume. For example, consider the following configuration snippet that uses a selector with a label match: ```plaintext selector: matchLabels: name: my-p ``` On the PV side, you might define: ```plaintext labels: name: my-pv ``` **Note** Even if the PVC requests a smaller amount of storage, it may bind to a larger PV if all other criteria match and no better option is available. Once a PV is bound to a PVC, its remaining capacity cannot be used for other claims. If no suitable volumes are available, the PVC remains in a pending state. When new PVs are added that meet the claim’s requirements, Kubernetes binds the PVC automatically. ## Creating a Persistent Volume Claim Let's create a persistent volume claim using a YAML template. In this example, the API version is set to v1 and the kind is PersistentVolumeClaim. The claim is named "myclaim". Under the specification, we set the access mode to ReadWriteOnce and request 500Mi of storage. Below is the YAML definition for the PVC: ```plaintext apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi ``` To create the claim, run the following command: ```plaintext kubectl create -f pvc-definition.yaml ``` Once applied, you can view the status of your PVC with: ```plaintext kubectl get persistentvolumeclaim ``` The output might initially be similar to: ```plaintext NAME STATUS VOLUME CAPACITY ACCESS MODES myclaim Pending ``` At this point, Kubernetes is evaluating the available persistent volumes. Suppose you have already configured a PV as shown below: ```plaintext apiVersion: v1 kind: PersistentVolume metadata: name: pv-voll spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi awsElasticBlockStore: volumeID: fsType: ext4 ``` Since the PV's access mode matches and its capacity (1Gi) exceeds the PVC's request (500Mi), Kubernetes binds the PVC to this PV if no other closer match exists. You can verify the binding by executing: ```plaintext kubectl get persistentvolumeclaim ``` The PVC status should now show as "Bound". ## Deleting a Persistent Volume Claim and Reclaim Policies To delete a PVC, use the following command: ```plaintext kubectl delete persistentvolumeclaim myclaim ``` After deletion, it's important to understand the fate of the underlying PV based on its reclaim policy. The reclaim policy determines what happens to a PV when its associated PVC is deleted. ### Reclaim Policy Options The default reclaim policy is set to Retain, which means the PV remains available until manually deleted by an administrator: ```plaintext persistentVolumeReclaimPolicy: Retain ``` If you want Kubernetes to automatically delete the PV when the PVC is removed—thus freeing up the storage—you can change the reclaim policy accordingly. Alternatively, you might choose the Recycling policy, where the data on the volume is scrubbed before the volume is made available for reuse: ```plaintext persistentVolumeReclaimPolicy: Recycling ``` **Tip** To practice configuring and troubleshooting persistent volumes and claims in Kubernetes, review the above configurations and experiment with these commands on your cluster. ## Summary In this lesson, we covered: * **PVC and PV Concepts:** How PVCs bind to suitable PVs based on capacity and properties. * **Configuration Example:** Creating a PVC using a YAML file. * **PVC Creation Process:** Binding status and troubleshooting. * **Reclaim Policies:** How PVs behave after their associated PVC is deleted. # Solution Persistent Volumes and Persistent Volume Claims optional In this lab exercise, you will learn how to configure persistent storage for your Kubernetes workloads by deploying a Pod that writes log entries to a file, configuring a hostPath volume to persist logs, creating a Persistent Volume (PV) and Persistent Volume Claim (PVC), resolving binding issues due to access mode mismatches, updating the Pod to use the PVC, and finally exploring the reclaim policy behavior upon deletion. --- ## Viewing Application Logs Inside the Pod After deploying the web application Pod, you can check its status and view the logs generated by the application. The application writes its log entries to `/log/app.log` within the container. To view these logs, run the following command: ```plaintext kubectl exec webapp -- cat /log/app.log ``` This command displays the log entries generated by your application. If the Pod were deleted, the logs stored within the container would be lost since they reside on the default ephemeral volume provided by Kubernetes. To verify the Pod configuration, use: ```plaintext kubectl describe pod webapp ``` Notice that there is no additional volume configured to persist the logs. --- ## Configuring a HostPath Volume for Log Storage Currently, the `/log/app.log` file exists only within the container’s file system. To ensure log persistence even when the Pod is recreated, you need to configure a hostPath volume. This involves mounting a directory from the host system (for example, `/var/log/webapp`) into the container. 1. **Verify the Host Directory** First, ensure that the host directory exists and is empty: ```plaintext ls /var/log/webapp ``` 2. **Update the Pod Configuration** Next, edit the Pod configuration to add a volume named `log-volume` of type hostPath: ```plaintext apiVersion: v1 kind: Pod metadata: name: webapp namespace: default spec: containers: - name: event-simulator image: kodekloud/event-simulator imagePullPolicy: Always env: - name: LOG_HANDLERS value: file volumeMounts: - name: kube-api-access-lmntb mountPath: /var/run/secrets/kubernetes.io/serviceaccount readOnly: true - name: log-volume mountPath: /log dnsPolicy: ClusterFirst nodeName: controlplane restartPolicy: Always volumes: - name: kube-api-access-lmntb projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - name: log-volume hostPath: path: /var/log/webapp ``` 3. **Apply the Configuration** Save your changes. In some cases, direct edits to a running Pod may be rejected. If so, force the update by saving the configuration to a temporary file and executing: ```plaintext kubectl replace --force -f /tmp/kubectl-edit-XXXX.yaml ``` After the Pod is recreated with the new volume, verify that the logs are now being written to the host’s `/var/log/webapp/app.log` file. --- ## Creating a Persistent Volume (PV) To leverage persistent storage, create a Persistent Volume. Here is an example PV specification using a hostPath: ```plaintext apiVersion: v1 kind: PersistentVolume metadata: name: pv-log spec: capacity: storage: 100Mi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain hostPath: path: /pv/log ``` 1. Save the file as `pv.yaml`. 2. Create the PV using: ```plaintext kubectl create -f pv.yaml ``` 3. Verify that the PV is available: ```plaintext kubectl get pv ``` --- ## Creating a Persistent Volume Claim (PVC) Now, create a Persistent Volume Claim to request storage from the configured PV. Initially, your PVC might look like this: ```plaintext apiVersion: v1 kind: PersistentVolumeClaim metadata: name: claim-log-1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Mi ``` 1. Save the file as `pvc.yaml`. 2. Apply it with: ```plaintext kubectl create -f pvc.yaml ``` 3. Check the state of the PVC: ```plaintext kubectl get pvc ``` At this point, you might notice that the PVC is still in a **Pending** state due to an access mode mismatch. Although the PV has a capacity of 100Mi, the PVC requests `ReadWriteOnce` while the PV supports `ReadWriteMany`. **Resolving Access Mode Mismatch** To resolve the pending state, update the PVC to request `ReadWriteMany` access mode, ensuring that it correctly binds to the PV. Edit the `pvc.yaml` file accordingly and execute: ```plaintext kubectl replace --force -f pvc.yaml ``` Verify that the PVC and PV are now correctly bound: ```plaintext kubectl get pv kubectl get pvc ``` You should see output similar to: ```plaintext NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS AGE pv-log 100Mi RWX Retain Bound default/claim-log-1 5m5s ``` And: ```plaintext NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE claim-log-1 Bound pv-log 100Mi RWX 26s ``` --- ## Updating the Pod to Use the PVC Update the webapp Pod configuration to mount the storage from the PVC instead of relying on the hostPath directly. Here is the updated volume configuration snippet for the Pod: ```plaintext apiVersion: v1 kind: Pod metadata: name: webapp spec: containers: - name: event-simulator image: kodekloud/event-simulator imagePullPolicy: Always env: - name: LOG_HANDLERS value: file volumeMounts: - name: kube-api-access-lmntb mountPath: /var/run/secrets/kubernetes.io/serviceaccount readOnly: true - name: log-volume mountPath: /log dnsPolicy: ClusterFirst nodeName: controlplane restartPolicy: Always volumes: - name: kube-api-access-lmntb projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - name: log-volume persistentVolumeClaim: claimName: claim-log-1 ``` Replace the previous Pod definition by forcing a replacement if necessary: ```plaintext kubectl replace --force -f ``` After the update, verify that logs are now available on the host through the PV by checking `/pv/log/app.log`. --- ## Examining the Reclaim Policy and Cleanup Behavior The persistent volume `pv-log` is configured with a reclaim policy of **Retain**. This means that if the PVC is deleted, the PV is not automatically removed; it remains in the cluster with a status of `Released`. Check its status with: ```plaintext kubectl get pv pv-log ``` Expected output: ```plaintext NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS AGE pv-log 100Mi RWX Retain Bound default/claim-log-1 9m47s ``` **Deleting Resources** If you attempt to delete the PVC while it is still mounted by an active Pod, it may remain in a terminating state until the Pod is deleted. To fully remove the PVC and allow the PV to transition to a `Released` state, delete the associated Pod first. For example: 1. Delete the PVC: ```plaintext kubectl delete pvc claim-log-1 ``` 2. If the PVC appears stuck, delete the Pod: ```plaintext kubectl delete pod webapp ``` Finally, check the PV status to confirm it has transitioned to `Released`: ```plaintext kubectl get pv pv-log ``` # Storage Classes In this guide, we'll dive into storage classes in Kubernetes, a vital concept for managing dynamic storage provisioning. Earlier, we covered the creation of Persistent Volumes (PVs), Persistent Volume Claims (PVCs), and their integration within pod definitions. With static provisioning, you manually create and manage disks along with their corresponding PV definitions before deploying an application. Below is an example of static provisioning using Google Cloud Persistent Disk: ```plaintext # pv-definition.yaml apiVersion: v1 kind: PersistentVolume metadata: name: pv-vol1 spec: accessModes: - ReadWriteOnce capacity: storage: 500Mi gcePersistentDisk: pdName: pd-disk fsType: ext4 ``` ```plaintext # pvc-definition.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi ``` ```plaintext # pod-definition.yaml apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh", "-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"] volumeMounts: - mountPath: /opt name: data-volume volumes: - name: data-volume persistentVolumeClaim: claimName: myclaim ``` Before creating the corresponding PV, you must provision the disk on Google Cloud manually. For example, you would use the following command to create a disk: ```plaintext gcloud beta compute disks create \ --size 1GB \ --region us-east1 \ pd-disk ``` **Static Provisioning** Static provisioning involves manually creating and managing the storage disks and their PV definitions. This can become cumbersome for dynamic applications. --- ## Dynamic Provisioning with Storage Classes Storage classes simplify storage management by allowing you to automatically create and configure storage resources when a PVC is created. They define a provisioner (such as Google Cloud Persistent Disk) that automatically creates a new disk, dynamically provisions a PV, and binds it to a PVC based on the storage class specified. To implement dynamic provisioning, create a StorageClass object with the API version set to [storage.k8s.io/v1](http://storage.k8s.io/v1). For Google Cloud, set the provisioner to [kubernetes.io/gce-pd](http://kubernetes.io/gce-pd). Here is an example: ```plaintext # sc-definition.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: google-storage provisioner: kubernetes.io/gce-pd ``` When a PVC references a storage class by name, Kubernetes automatically creates and attaches the required storage. For instance: ```plaintext # pvc-definition.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce storageClassName: google-storage resources: requests: storage: 500Mi ``` ```plaintext # pod-definition.yaml apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh", "-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"] volumeMounts: - mountPath: /opt name: data-volume volumes: - name: data-volume persistentVolumeClaim: claimName: myclaim ``` In this setup, when the PVC is created, Kubernetes uses the specified StorageClass to automatically provision a new disk, create a PV, and bind it to the PVC. This eliminates the need for manual disk provisioning. --- ## Advanced Storage Classes Storage classes can be further customized with parameters specific to the underlying provisioner. For instance, with Google Cloud Persistent Disk, you can define the disk type and replication mode. Consider this example: ```plaintext # google-storage with parameters apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: google-storage provisioner: kubernetes.io/gce-pd parameters: type: pd-standard # Options: pd-standard or pd-ssd replication-type: none # Options: none or regional-pd ``` This customization allows enterprises to define different classes of service based on performance and availability requirements. Below are examples of multiple storage classes: | Storage Class | Disk Type | Replication Mode | | --- | --- | --- | | Silver | pd-standard | None | | Gold | pd-ssd | None | | Platinum | pd-ssd | Regional (High Availability) | Examples for each: ```plaintext # silver storage class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: silver provisioner: kubernetes.io/gce-pd parameters: type: pd-standard replication-type: none ``` ```plaintext # gold storage class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gold provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: none ``` ```plaintext # platinum storage class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: platinum provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: regional-pd ``` By utilizing these tailored storage classes in your PVC definitions, you ensure that the storage provisioned aligns precisely with your application’s performance requirements and budget considerations. **Dynamic Provisioning Benefits** Dynamic provisioning streamlines the process of deploying applications in Kubernetes by eliminating manual storage configuration. This leads to improved efficiency, reduced errors, and enhanced scalability. --- ## In Summary Storage classes in Kubernetes offer a powerful mechanism to manage storage dynamically. By abstracting the complexities of physical disk configurations, storage classes enable you to create, manage, and bind storage resources automatically as needed. Whether you opt for static provisioning or embrace the flexibility of dynamic provisioning, storage classes are integral to ensuring your applications have the right storage infrastructure. # Why Stateful Sets In this lesson, we will explore StatefulSets in Kubernetes, emphasizing their necessity compared to Deployments. To understand their importance, let's begin with a real-world scenario outside of Kubernetes. Imagine you have a physical server running a database server, such as MySQL. After installing MySQL and creating the initial database, your database is up and running, accessible to client applications. To ensure high availability, you deploy additional servers with MySQL installed. However, these new servers start with empty databases, and you must replicate data from the original master database to them. For illustration, we use MySQL replication. Although the details of MySQL replication are simplified here, the focus is on the high-level procedure for setting up dynamic replicas. In a common single-master multi-slave topology, all write operations are directed to the master server, while read operations can be served by any server. The process involves the following steps: 1. Deploy the master server and bring it online. 2. Clone the database from the master to the first slave. 3. Enable continuous replication from the master to the first slave. 4. Once the first slave is ready, clone its data to a second slave. 5. Enable continuous replication on the second slave, configuring it to replicate from the master. A critical aspect of this replication setup is configuring the slave servers with the master's hostname or address. This static reference is essential because if the master is replaced (for instance, due to a crash), its IP address might change if assigned dynamically. Therefore, a stable, static hostname is required. In typical deployments, pods are created with random names, which makes maintaining a static reference challenging. For example, if a master pod crashes and is recreated with a new name, the replication linkage is broken. ![The image illustrates a MySQL replication setup process, detailing steps to configure a master-slave database architecture with two slave servers replicating from a master.](https://kodekloud.com/kk-media/image/upload/v1752871322/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Why-Stateful-Sets/frame_230.jpg align="left") ## Transitioning to Kubernetes In Kubernetes, you might consider deploying each instance in your MySQL cluster (both master and slaves) as pods within a Deployment. Deployments are excellent for scaling and managing pods; however, they do not guarantee the startup order of the pods. In our replication setup, the master must be established before starting the slaves, and this sequential order is imperative. Consider the following configuration snippet: ```plaintext MASTER_HOST=mysql-master MASTER_HOST=mysql-master ``` In this scenario, the master must start before any of the slave nodes. For example, slave one must initialize and clone data from the master, and later, slave two should clone data from slave one. Unfortunately, Deployments create all pods concurrently, so ensuring the necessary startup order becomes problematic. Additionally, during the cloning and continuous replication phases, it is critical to clearly distinguish between the master and slave pods. The master requires a constant and static hostname instead of an IP address—which might change when a pod is recreated. With Deployments generating dynamically named pods, this requirement is not met. If the designated master pod crashes and is replaced by a new pod with a different name, your replication configuration will break. **Key Point** A stable and predictable pod naming convention is crucial for proper database replication in a stateful system. ## Introducing StatefulSets StatefulSets address these challenges with features that resemble Deployments—such as pod templating, scaling, rolling updates, and rollbacks—but with a critical advantage: they enforce sequential pod creation. This ensures that the first pod is deployed, running, and ready before the next pod is initialized. This behavior directly caters to our requirement where the master must always start first, followed by the slaves. StatefulSets assign a unique ordinal index to each pod, starting at zero, and each pod’s name is derived from the StatefulSet name combined with its ordinal index. For example, given a StatefulSet named mySQL: * The first pod is named mySQL-0. * The second pod is named mySQL-1. * The third pod is named mySQL-2. This predictable naming allows you to reliably designate mySQL-0 as the master pod. Slave pods can then be explicitly configured—for instance, mySQL-1 and mySQL-2 can be set up to clone data from the master, or further cascade replication among themselves. Even if the master pod fails and is recreated, it retains its stable name due to the sticky identity provided by StatefulSets. ![The image illustrates a step-by-step process for setting up MySQL stateful sets with master-slave replication, including cloning data and configuring master addresses.](https://kodekloud.com/kk-media/image/upload/v1752871323/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Why-Stateful-Sets/frame_550.jpg align="left") ## Conclusion StatefulSets offer significant advantages when deploying systems like a MySQL cluster on Kubernetes. They guarantee that pods are launched in a specific sequence and maintain a stable network identity essential for configuring master-slave replication. **Next Steps** This discussion sets the stage for a deeper exploration into Kubernetes topics such as creating StatefulSets, headless services, and configuring Persistent Volumes. # Stateful Sets Introduction In this lesson, we explore StatefulSets in Kubernetes and explain when to use them instead of Deployments. StatefulSets are ideal for applications that require: * Ordered startup and shutdown * Stable network identities * Consistent storage provisioning When your application instances need to start in a specific order or require persistent identities between restarts, opting for a StatefulSet is the right choice. **Key Difference from Deployments** A StatefulSet is similar to a Deployment in that you define it via a YAML file with a Pod template. The main differences are: * Change the kind from Deployment to StatefulSet (note the uppercase “S”). * Include the additional field 'serviceName' to specify a headless service. ## Converting a Deployment to a StatefulSet Consider the following example of a MySQL Deployment: ```plaintext apiVersion: apps/v1 kind: Deployment metadata: name: mysql labels: app: mysql spec: replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql ``` To convert this Deployment into a StatefulSet, modify the YAML file by updating the kind and adding the `serviceName` field: ```plaintext apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql labels: app: mysql spec: replicas: 3 selector: matchLabels: app: mysql serviceName: mysql-h template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql ``` When you create a StatefulSet using this file, Kubernetes will: * Deploy Pods one at a time in an ordered, graceful manner. * Assign a stable, unique DNS record to each Pod, allowing other applications to refer to them reliably. * Scale Pods sequentially, where each new Pod starts only after the previous one is ready. This ordered behavior is particularly beneficial for applications like MySQL databases, where preserving state and order is critical. ## Creating and Managing a StatefulSet You can use standard Kubernetes commands to create and scale your StatefulSet. For example: ```plaintext kubectl create -f statefulset-definition.yml kubectl scale statefulset mysql --replicas=5 # Output: statefulset.apps/mysql scaled ``` When scaling down, Kubernetes terminates the Pods in reverse order: the last Pod is removed first, followed by earlier ones. Likewise, when deleting the StatefulSet, Pods are terminated in reverse order. ## Customizing Pod Deployment Strategy By default, StatefulSets follow an ordered approach for both deployment and termination. However, you can override this behavior by setting the `podManagementPolicy` to `Parallel`. This instructs Kubernetes to deploy and terminate all Pods simultaneously while still providing them with stable network identities. Below is an example StatefulSet configuration that uses the `Parallel` pod management policy: ```plaintext apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql labels: app: mysql spec: replicas: 3 selector: matchLabels: app: mysql serviceName: mysql-h podManagementPolicy: Parallel template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql ``` You can manage this StatefulSet with the following commands: ```plaintext kubectl create -f statefulset-definition.yml kubectl scale statefulset mysql --replicas=5 kubectl scale statefulset mysql --replicas=3 kubectl delete statefulset mysql # Output: statefulset.apps/mysql deleted ``` This configuration allows Pods to be launched or terminated in parallel, which can be advantageous when the order of operations is not a strict requirement for your application. **Remember** While the `Parallel` management policy offers faster scaling, it does so at the cost of ordered deployment. Choose the appropriate policy based on your application's initialization and shutdown needs. # Headless Services In this article, we explore headless services in Kubernetes and how they solve routing challenges in a master-slave architecture, such as a MySQL cluster. When using a StatefulSet, each pod is deployed one at a time with a unique ordinal index (for example, mysql-0, mysql-1, mysql-2). This approach provides stable, unique names for each pod, making it easier to direct connections to specific nodes—an essential requirement for database clusters where write operations must be handled by a single master. ## The Problem with Traditional Services Under typical conditions, Kubernetes services function as load balancers. They provide a Cluster IP and a DNS name (like mysql.default.svc.cluster.local) that routes traffic evenly to all matching pods. This behavior is acceptable when the application only performs read operations in a MySQL cluster but becomes problematic for write operations. Write queries must be directed solely to the master pod, and load balancing across all pods could lead to data inconsistency or conflicts. ## How Headless Services Address the Issue Headless services eliminate the load balancing behavior by not assigning a Cluster IP. Instead, they generate DNS records for each individual pod in the following format: pod-name.headless-service-name.namespace.svc.cluster.local For example, if you create a headless service named "mysql-h", the master pod can be accessed via: mysql-0.mysql-h.default.svc.cluster.local This DNS entry consistently resolves to the master pod in the MySQL deployment. **Note** In a headless service, setting `clusterIP` to "None" is the only key difference from a standard service definition. ## Headless Service Definition Example Below is an example of how to define a headless service: ```plaintext apiVersion: v1 kind: Service metadata: name: mysql-h spec: ports: - port: 3306 selector: app: mysql clusterIP: None ``` ## Configuring Pods to Use Headless Services When deploying a pod under a headless service, include the optional fields `subdomain` and `hostname` in the pod specification. The `subdomain` must match the name of your headless service, ensuring that Kubernetes creates the correct DNS record. For example, to create a pod with only the subdomain specified: ```plaintext apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: mysql spec: containers: - name: mysql image: mysql subdomain: mysql-h ``` However, to generate a fully qualified DNS record that includes the pod name, you must set the `hostname` field as well: ```plaintext apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: mysql spec: containers: - name: mysql image: mysql subdomain: mysql-h hostname: mysql-pod ``` ## Visual Overview of a MySQL Cluster Setup The diagram below illustrates the setup of a MySQL database cluster with one master and two replicas. It highlights the network configuration, including IP addresses and hostnames: ![The image illustrates a MySQL database cluster setup with one master node and two replicas, showing IP addresses and hostnames for network configuration.](https://kodekloud.com/kk-media/image/upload/v1752871314/notes-assets/images/Certified-Kubernetes-Application-Developer-CKAD-Headless-Services/frame_150.jpg align="left") ## Deployments vs. StatefulSets When deploying pods using a Deployment, the pod template generally does not specify the `hostname` or `subdomain` fields. Consequently, the headless service is unable to create unique A records for each pod. If these fields are manually added to the pod template, every pod receives the same DNS record, resulting in an address like mysql-pod.mysql-h.default.svc.cluster.local, which is unsuitable for distinct addressing. ### Deployment Example with Identical DNS Records Below is an example where a Deployment assigns the same hostname and subdomain to all pods: ```plaintext apiVersion: v1 kind: Service metadata: name: mysql-h spec: ports: - port: 3306 selector: app: mysql clusterIP: None ``` ```plaintext apiVersion: apps/v1 kind: Deployment metadata: name: mysql-deployment labels: app: mysql spec: replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql subdomain: mysql-h hostname: mysql-pod ``` In this configuration, all pods share the same DNS record because they have identical hostnames and subdomains. ## Leveraging StatefulSets for Unique DNS Records StatefulSets are designed to overcome this limitation by automatically generating unique hostnames for each pod. You do this by simply referencing the headless service name in the StatefulSet specification using the `serviceName` field. Kubernetes then assigns a unique DNS record to each pod based on its ordinal index. ### StatefulSet Example with Headless Service Below is an example of a StatefulSet that utilizes a headless service: ```plaintext apiVersion: v1 kind: Service metadata: name: mysql-h spec: ports: - port: 3306 selector: app: mysql clusterIP: None ``` ```plaintext apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql-deployment labels: app: mysql spec: serviceName: mysql-h replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql ``` In this configuration, each pod is assigned a unique DNS record, for example: mysql-0.mysql-h.default.svc.cluster.local   mysql-1.mysql-h.default.svc.cluster.local   mysql-2.mysql-h.default.svc.cluster.local This setup meets the requirement for distinct addressing, ensuring that write requests intended for the master pod are routed correctly. **Key Takeaway** StatefulSets combined with headless services provide a robust solution for applications requiring individual pod addressing without load balancing interference. # Storage in StatefulSets In this article, we explore storage in StatefulSets and explain how persistent storage operates in Kubernetes. We begin with a review of persistent volumes (PV) and persistent volume claims (PVC) used with Pods, then dive into different provisioning methods and their use in StatefulSets. --- ## Persistent Volumes and Static Provisioning Static provisioning involves a three-step process: 1. Create a PersistentVolume. 2. Create a PersistentVolumeClaim. 3. Reference the PVC in your Pod specification. Below is an example of each configuration: **Tip** For static provisioning, ensure that the PV capacity and access modes match the requirements of your applications. ### PersistentVolume Definition ```plaintext # pv-definition.yaml apiVersion: v1 kind: PersistentVolume metadata: name: pv-vol1 spec: accessModes: - ReadWriteOnce capacity: storage: 500Mi gcePersistentDisk: pdName: pd-disk fsType: ext4 ``` ### PersistentVolumeClaim Definition ```plaintext # pvc-definition.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-volume spec: accessModes: - ReadWriteOnce storageClassName: google-storage resources: requests: storage: 500Mi ``` ### Pod Definition Referencing the PVC ```plaintext # pod-definition.yaml apiVersion: v1 kind: Pod metadata: name: mysql spec: containers: - image: mysql name: mysql volumeMounts: - mountPath: /var/lib/mysql name: data-volume volumes: - name: data-volume persistentVolumeClaim: claimName: data-volume ``` --- ## Dynamic Provisioning with StorageClasses Dynamic provisioning simplifies the process by automatically creating PVs when you define a PVC and reference a StorageClass. This eliminates the need to manually provision PVs. ### StorageClass Definition ```plaintext # sc-definition.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: google-storage provisioner: kubernetes.io/gce-pd ``` ### PVC Definition Using Dynamic Provisioning ```plaintext # pvc-definition.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-volume spec: accessModes: - ReadWriteOnce storageClassName: google-storage resources: requests: storage: 500Mi ``` ### Pod Definition Referencing the Dynamically Provisioned PVC ```plaintext # pod-definition.yaml apiVersion: v1 kind: Pod metadata: name: mysql spec: containers: - image: mysql name: mysql volumeMounts: - mountPath: /var/lib/mysql name: data-volume volumes: - name: data-volume persistentVolumeClaim: claimName: data-volume ``` --- ## Using StatefulSets with Shared Storage StatefulSets support scenarios where multiple replicas share the same volume. If you reference a common PVC within a StatefulSet, all replicas will attempt to access the same storage. This setup works if your underlying storage supports multi-reader or multi-writer capabilities. ### Shared Storage StatefulSet Example ```plaintext --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql labels: app: mysql spec: replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql volumeMounts: - mountPath: /var/lib/mysql name: data-volume volumes: - name: data-volume persistentVolumeClaim: claimName: data-volume ``` **Important** Ensure that your storage solution supports concurrent access if you plan to share the same volume across multiple Pods. --- ## Separate Volumes for Each Pod Using VolumeClaimTemplates For scenarios like MySQL replication where each Pod requires dedicated storage, a volume claim template allows Kubernetes to automatically create a unique PVC for each Pod in a StatefulSet. ### Step 1: Define the StorageClass ```plaintext # sc-definition.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: google-storage provisioner: kubernetes.io/gce-pd ``` ### Step 2: Create a StatefulSet with a VolumeClaimTemplate ```plaintext # statefulset-definition.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql labels: app: mysql spec: replicas: 3 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql volumeMounts: - mountPath: /var/lib/mysql name: data-volume volumeClaimTemplates: - metadata: name: data-volume spec: accessModes: - ReadWriteOnce storageClassName: google-storage resources: requests: storage: 500Mi ``` In this configuration, Kubernetes provisions a unique PVC for each Pod automatically based on the volume claim template. This ensures that every Pod receives its dedicated storage. Additionally, StatefulSets maintain stable storage even if a Pod is rescheduled; the associated PVC and underlying PV remain intact and are reattached to the new Pod instance.