Enable retry on network failures

[arjunsuresh]

We often see CM script runs failing due to netwok failures like this. It'll be good to add a retry mechanism for such failures to improve the user experience and reduce the failures of automatic runs.

[anandhu-eng]

Hi @arjunsuresh , this would be useful. Should this be kept on by default or should it be controlled through any env variable? I'm wondering if there is a case where user wants to turn it off

[arjunsuresh]

@anandhu-eng I think we should enable it by default and let users an ENV variable to turn it off for any reason. But first we need to list out the places where we need this. Below are some of them. We should probably try it on one, and if it works as expected move to the remaining places.

1. git clone
2. System util installation
3. pip package installation

[arjunsuresh]

The below failure is seen many times in our github actions. Trying the fix --dns 8.8.8.8 --dns 8.8.4.4 to docker run command.

2024-11-05T13:38:34.8624636Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd0_245.4-4ubuntu3.24_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8629875Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/a/argon2/libargon2-1_0~20171227-0.2_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8634850Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/l/lvm2/libdevmapper1.02.1_1.02.167-1ubuntu1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8639786Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/i/iptables/libip4tc2_1.8.4-3ubuntu2.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8642386Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/k/kmod/libkmod2_27-1ubuntu2.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8644257Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-timesyncd_245.4-4ubuntu3.24_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8646140Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd_245.4-4ubuntu3.24_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8648397Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/l/lvm2/dmsetup_1.02.167-1ubuntu1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8650427Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/gobject-introspection/libgirepository-1.0-1_1.64.1-1~ubuntu20.04.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8652805Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/gobject-introspection/gir1.2-glib-2.0_1.64.1-1~ubuntu20.04.1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8654823Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/d/dbus-python/python3-dbus_1.2.16-1build1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8656630Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/pygobject/python3-gi_3.36.0-1_amd64.deb  Could not resolve 'archive.ubuntu.com'
2024-11-05T13:38:34.8658027Z E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?