Delete orphaned File and Block volumes on IBM Cloud

Update Nov 2, 2021

UPDATE: A newer version of the script can be found here: https://gist.github.com/timroster/6862807a7dd30a78d39fa2f4212a895e

Original post

Credit for the initial script goes to Tim Robinson, so +10 points for him, but also -1 point for not sharing it anywhere.

So, when you’re using a managed Kubernetes service, like IBM Cloud Kubernetes Service (IKS), it’s not uncommon for volumes assocaited with the cluster to become orphaned. This happens when the cluster is deleted but the underlying block and file volumes, which back the PVCs associated with the cluster, are not deleted. This may be a permission issue or something trickier to debug.

Below is a script you can run to find out which volumes have been orphaned. A canonical version will be posted on GitHub but hopefully the one below works for you.

ibmcloud ks clusters | grep normal | awk '{ print $ 2}' > cluster.lst

for block_id in $(ibmcloud sl block volume-list --column id --column notes --output JSON | jq '.[].id');
do
  cluster=$(ibmcloud sl block volume-list --column id --column notes --output JSON | jq -r ".[]|select(.id==$block_id)|.notes" | jq -r .cluster);
  block_name=$(ibmcloud sl block volume-detail $block_id --output JSON | jq -r .username);
  grep $cluster cluster.lst > /dev/null;
  if [ $? -eq 1 ]; then
    echo "cluster $cluster not found for block ID $block_id / block name $block_name";
  fi;
done

for file_id in $(ibmcloud sl file volume-list --column id --column notes --output JSON | jq '.[].id');
do
  cluster=$(ibmcloud sl file volume-list --column id --column notes --output JSON | jq -r ".[]|select(.id==$file_id)|.notes" | jq -r .cluster);
  file_name=$(ibmcloud sl file volume-detail $file_id --output JSON | jq -r .username);
  grep $cluster cluster.lst > /dev/null;
  if [ $? -eq 1 ]; then
    echo "cluster $cluster not found for file ID $file_id / file name $file_name";
  fi;
done

Which would produces output similar to the following:

cluster bph131hf0sni9ds0f16g not found for block ID 128596084 / block name IBM02SEL2032340-6
cluster brfq0uit0598r1787d70 not found for block ID 154224764 / block name IBM02SEL2032340-125
cluster brfq0uit0598r1787d70 not found for block ID 154227992 / block name IBM02SEL2032340-126
...
cluster c0c5tr1d0mkel12h6prg not found for file ID 208458886 / file name IBM02SEV2032340_997
cluster c0ckgbad0b844iinp3p0 not found for file ID 208661744 / file name IBM02SEV2032340_998
cluster c0h8hnhd0pi89ssb3nj0 not found for file ID 210707122 / file name IBM02SEV2032340_1006

You can of course add in a ibmcloud sl file volume-delete and ibmcloud sl block volume-delete commands instead of the echo commands but I’d rather have the list so I can spot check the volumes through the IBM Cloud dashboard.

Thanks for reading.