search by tags

for the user

adventures into the land of the command line

how to remove orphaned pods from k8s via etcd

a bunch of pods are stuck terminating but there are no docker containers running for them

kubectl get pods -n mongo
NAME                      READY     STATUS        RESTARTS   AGE
mongodb-5598cbbcc-5tjzr   0/1       Terminating   0          22h
mongodb-5598cbbcc-bnsm8   0/1       Terminating   0          22h
mongodb-5598cbbcc-gfxvl   0/1       Terminating   0          23h
mongodb-5598cbbcc-qkwkw   0/1       Terminating   0          22h

there must be something left over in etcd. let’s check it. firstly, use version 3 of the etcd api, because in version 2, ls does nothing:

etcdctl --ca-file /etc/ssl/etcd/ssl/ca.pem --cert-file /etc/ssl/etcd/ssl/member-master2.pem --key-file /etc/ssl/etcd/ssl/member-master2-key.pem --endpoint "https://127.0.0.1:2379" ls
... crickets

using the version 3 version of the command, instead of ls, get /

ETCDCTL_API=3 etcdctl --cacert="/etc/ssl/etcd/ssl/ca.pem" --cert="/etc/ssl/etcd/ssl/member-master2.pem" --key="/etc/ssl/etcd/ssl/member-master2-key.pem" get / --prefix --keys-only

this will show all the keys. we need to find the key for the pod we want to delete

ETCDCTL_API=3 etcdctl --cacert="/etc/ssl/etcd/ssl/ca.pem" --cert="/etc/ssl/etcd/ssl/member-master2.pem" --key="/etc/ssl/etcd/ssl/member-master2-key.pem" get /registry/pods/mongo --prefix --keys-only
/registry/pods/mongo/mongodb-5598cbbcc-5tjzr
/registry/pods/mongo/mongodb-5598cbbcc-bnsm8
/registry/pods/mongo/mongodb-5598cbbcc-gfxvl
/registry/pods/mongo/mongodb-5598cbbcc-qkwkw

check again that there is no docker container running for this pod first. if not then delete its reference from etcd:

ETCDCTL_API=3 etcdctl --cacert="/etc/ssl/etcd/ssl/ca.pem" --cert="/etc/ssl/etcd/ssl/member-master2.pem" --key="/etc/ssl/etcd/ssl/member-master2-key.pem" del /registry/pods/mongo/mongodb-5598cbbcc-5tjzr

and its gone

kubectl get pods -n mongo
NAME                                READY     STATUS        RESTARTS   AGE
mongodb-5598cbbcc-bnsm8   0/1       Terminating   0          22h
mongodb-5598cbbcc-gfxvl   0/1       Terminating   0          23h
mongodb-5598cbbcc-qkwkw   0/1       Terminating   0          22h