Jenkins Runs out of Space

[patricksimonian]

It looks like it fails to clean up some log/build files after a build has suceeded or cancelled. That or log rotation hasn't been configured.

A temporary solution

oc rsh -n <project> <jenkins pod>
cd ~ && rm -rf ./builds/*

Why that solution is not a good idea

After debugging with Jeff, we found that removing these files just removed the pointers to them and there were still processes writing to the files. In essence, the files appear to be deleted but they are not.

Since the openshift pods do not have the lsof program installed it is hard to debug that issue within the pod.