Sudden OSError: [Errno 28] No space left on device upon start up

[thenaturalist]

Hi there,

upon trying to start composer-local today, I suddenly ran into the following error:

2023-02-16T09:48:07.877041427Z + sudo chown airflow:airflow airflow
2023-02-16T09:48:08.071326885Z + [ -f /var/local/setup_python_command.sh ]
2023-02-16T09:48:08.071863885Z + pip3 install --upgrade -r composer_requirements.txt
2023-02-16T09:48:10.541523761Z Defaulting to user installation because normal site-packages is not writeable
2023-02-16T09:48:11.546277762Z ERROR: Exception:
2023-02-16T09:48:11.546309178Z Traceback (most recent call last):
2023-02-16T09:48:11.546316387Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
2023-02-16T09:48:11.546321012Z     status = self.run(options, args)
2023-02-16T09:48:11.546323095Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
2023-02-16T09:48:11.546325095Z     return func(self, options, args)
2023-02-16T09:48:11.546330178Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 282, in run
2023-02-16T09:48:11.546334845Z     wheel_cache = WheelCache(options.cache_dir, options.format_control)
2023-02-16T09:48:11.546341137Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cache.py", line 293, in __init__
2023-02-16T09:48:11.546346262Z     self._ephem_cache = EphemWheelCache(format_control)
2023-02-16T09:48:11.546352928Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cache.py", line 260, in __init__
2023-02-16T09:48:11.546359803Z     self._temp_dir = TempDirectory(
2023-02-16T09:48:11.546362678Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/utils/temp_dir.py", line 138, in __init__
2023-02-16T09:48:11.546364720Z     path = self._create(kind)
2023-02-16T09:48:11.546366428Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/utils/temp_dir.py", line 186, in _create
2023-02-16T09:48:11.546368303Z     tempfile.mkdtemp(prefix="pip-{}-".format(kind))
2023-02-16T09:48:11.546370220Z   File "/opt/python3.8/lib/python3.8/tempfile.py", line 358, in mkdtemp
2023-02-16T09:48:11.546372012Z     _os.mkdir(file, 0o700)
2023-02-16T09:48:11.546373845Z OSError: [Errno 28] No space left on device: '/tmp/pip-ephem-wheel-cache-d9vlgzcw'
2023-02-16T09:48:22.974223461Z + sudo chown airflow:airflow airflow
2023-02-16T09:48:23.134888378Z + [ -f /var/local/setup_python_command.sh ]
2023-02-16T09:48:23.135352545Z + pip3 install --upgrade -r composer_requirements.txt
2023-02-16T09:48:25.787094171Z Defaulting to user installation because normal site-packages is not writeable
2023-02-16T09:48:26.922569046Z ERROR: Exception:
2023-02-16T09:48:26.922626255Z Traceback (most recent call last):
2023-02-16T09:48:26.922631796Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
2023-02-16T09:48:26.922634713Z     status = self.run(options, args)
2023-02-16T09:48:26.922637963Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
2023-02-16T09:48:26.922648255Z     return func(self, options, args)
2023-02-16T09:48:26.922650130Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 282, in run
2023-02-16T09:48:26.922652130Z     wheel_cache = WheelCache(options.cache_dir, options.format_control)
2023-02-16T09:48:26.922654088Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cache.py", line 293, in __init__
2023-02-16T09:48:26.922655880Z     self._ephem_cache = EphemWheelCache(format_control)
2023-02-16T09:48:26.922657588Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/cache.py", line 260, in __init__
2023-02-16T09:48:26.922659421Z     self._temp_dir = TempDirectory(
2023-02-16T09:48:26.922661046Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/utils/temp_dir.py", line 138, in __init__
2023-02-16T09:48:26.922662963Z     path = self._create(kind)
2023-02-16T09:48:26.922664713Z   File "/opt/python3.8/lib/python3.8/site-packages/pip/_internal/utils/temp_dir.py", line 186, in _create
2023-02-16T09:48:26.922666630Z     tempfile.mkdtemp(prefix="pip-{}-".format(kind))
2023-02-16T09:48:26.922668463Z   File "/opt/python3.8/lib/python3.8/tempfile.py", line 358, in mkdtemp
2023-02-16T09:48:26.922670213Z     _os.mkdir(file, 0o700)
2023-02-16T09:48:26.922671838Z OSError: [Errno 28] No space left on device: '/tmp/pip-ephem-wheel-cache-dsp1l3wy'
╭─ Error ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Environment failed to start.                                                                                                   │
│                                                                                                                                │
│ To print debug messages please add --debug flag.                                                                               │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

No config change on my end, worked flawlessly before.
Not sure where the the disk size limit would suddenly come from.
Device disk definitely has enough space.

[dan0dbfe]

Do you have enough inodes on your disk? df -i

[piotrwrotniak]

Maybe it's worth playing with: dm.basesize https://docs.docker.com/engine/reference/commandline/dockerd/

Could you post your requirements list? (I think this shows it breaks during packages installation).
Please check if limiting the dependencies list helps as well.