skip to content
Ruban Selvarajah

Stuck in CrashLoopBackOff due to PersistentVolumeClaim

/ 2 min read

The CrashLoopBackOff state means that your pod is stuck in a infinite loop where it starts and crashes, then restarts and crashes again. It’s a useful feature for debugging but a bit annoying when the bug in your code lives in a persistent volume.

The following steps will allow you to temporarily disable this feature so that you can get rid of the offending code in peace.

Let’s 🚀

Downscale the Deployment

Stop using the PersistentVolumeClaim so that we can update our code

kubectl get deployments
kubectl scale deployment DEPLOYMENT_NAME --replicas=0
//Returns "deployment.extensions "DEPLOYMENT_NAME" scaled"

Create a Debugger Pod

1. Get the PVC identifier

kubectl get deployment -o jsonpath="{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}" DEPLOYMENT_NAME

2. Prep a debugger pod

kind: Pod
apiVersion: v1
name: pvc-debugger
- name: my-buggy-volume
- name: debugger
image: busybox
command: ["sleep", "3600"]
- mountPath: "/data"
name: my-buggy-volume

3. Start the debugger pod

kubectl create -f pvc-debugger-pod.yaml
kubectl exec -it pvc-debugger sh

Now that I’m inside the container I can explore the volume which is mounted at /data and fix the issue.

Bring the Pod Online

1. Delete the debugger pod

kubectl delete -f pvc-debugger-pod.yaml

2. Scale the deployment back up

kubectl scale deployment DEPLOYMENT_NAME --replicas=1
//Returns "deployment.extensions "DEPLOYMENT_NAME" scaled"

Aaaand my site is up again!