inode filling up

[tresf]

Many items of our site go down when our own ./tmp directory fills up (1 million+ entries). Most notably, the forums.

Since this is the second time this has occurred, I'm documenting the steps we used last time, so we have a work-around until a permanent solution is determined.

sudo rm -rf /home/deploy/lmms.io/tmp
sudo mkdir /home/deploy/lmms.io/tmp
sudo chown www-data:www-data /home/deploy/lmms.io/tmp
sudo chmod 775 /home/deploy/lmms.io/tmp

... and optionally, if some services are still acting up, bounce the Fast CGI processor...

sudo /etc/init.d/php5-fpm restart

Note: Most would do ./tmp/* to leave the directory intact, but there's too many files for rm to process, so xargs would be needed instead. Since blowing away the directory could be faster than deleting a million files, I've provided the above solution. 😈 If the command errors at any point, try it a second time before giving up.

In the mean time, I've opened a ticket with our hosting provider (NetworkRedux) to see what's the preferred method for cleaning this up.

[tresf]

The hosting provider has installed a clean-up script which will run nightly. To see where it's located:

sudo crontab -l

The contents of the script:

#!/bin/sh
find [path]/[to]/lmms.io/tmp -maxdepth 1 -type f -name 'sess*' -mtime +12 -exec rm -f {} \;

They said this is caused by us not using the standard tmp location for our session files. Not sure how valid that is, but I'm closing this issue out as resolved for now. Feel free to reopen if the problem resurfaces or a better approach is needed.

[Umcaruje]

👍

[tresf]

This is still occurring. Reopening.

[tresf]

Support says:

We have added extra cron jobs to clear wiki session files

We'll see how well that works. Closing back out for now.

[Umcaruje]

This is still happening.

[tresf]

Re-notified support. We'll see what happens.

[socceroos]

What part of the code is putting items in the tmp directory? Is it a rudimentary cache?

[tresf]

No issues in 1 year. Closing.

[Umcaruje]

This happened again today, the forums are down, cleaning the tmp dirs rn

[tresf]

Support is claiming it's happening due to the lsp_data directory. This is bullshit, there's only about 12,000 files there. I think they setup the disk with too few inodes. 12,000 files shouldn't be a breaking point on a VPS in 2017.

[tresf]

Ok... I found something...

lsp_data/11880/ is a monster and I'm not sure what it is. It's taking up 100x the inodes of the rest of the lsp_data directory.

Just cd'ing to the directory takes almost 60 seconds. I'm not sure what created this or what its contents are.

I'm still investigating.

[tresf]

Ok... lsp_data/11880/ is mostly sessions variables. This directory appears to be misplaced, probably a bug caused by the upgrade.

Of the 1,301,401 files, 1,300,215 of them are named ./sess_....

 ~/lsp_data/11880 $ find . -name . -o -prune | head
.
./sess_thuucbafmnui84mdi6s11qbus0
./sess_ti11pl4hba7fe0jv2tku2n3l57
./sess_ti5cgc2vlps37rh967dghru6k1
./sess_ti7d4pq92ofhr9bm1fmvugl0q4
./sess_ti9c7e3pvhicqd66j8u3edt5a2
./sess_ti9gq8bk8r4tpvriid13ou7i27
./sess_ticg8blksfrtqs7kbchis1m2e4
./sess_ticgf7fknuhkbovsj7lsuknnj5
./sess_ticv4b8aacohckr7bu8797q2i5

I'm not sure what else is inside this folder, but we can purge these sessions for now and then debug why they're getting misplaced.

[Umcaruje]

This happened again today, but it affected the download page #295

I don't know what happened to the ~/lmms.io/tmp directory, but it doesn't exist anymore. I found a bunch of undeleted session files in ~/lmms.io/var/cache/lsp and I cd'd to it and cleaned them up with find -type f -name 'sess*' -mtime +12 -exec rm -f {} \;

Current status of the inodes:

df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             228K   423  228K    1% /dev
tmpfs            233K   564  233K    1% /run
/dev/xvda2       1.5M  1.1M  470K   70% /
tmpfs            233K     2  233K    1% /dev/shm
tmpfs            233K     4  233K    1% /run/lock
tmpfs            233K    16  233K    1% /sys/fs/cgroup
tmpfs            233K     4  233K    1% /run/user/1001

All of them were used before my cleaning.

[liushuyu]

This happened again today, but it affected the download page #295

I don't know what happened to the ~/lmms.io/tmp directory, but it doesn't exist anymore. I found a bunch of undeleted session files in ~/lmms.io/var/cache/lsp and I cd'd to it and cleaned them up with find -type f -name 'sess*' -mtime +12 -exec rm -f {} \;

The issue was I migrate the website to Symfony so that the temporary folder path changed.

Current status of the inodes:

df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             228K   423  228K    1% /dev
tmpfs            233K   564  233K    1% /run
/dev/xvda2       1.5M  1.1M  470K   70% /
tmpfs            233K     2  233K    1% /dev/shm
tmpfs            233K     4  233K    1% /run/lock
tmpfs            233K    16  233K    1% /sys/fs/cgroup
tmpfs            233K     4  233K    1% /run/user/1001

All of them were used before my cleaning.

I have modified the clean up script located at /opt/redux/scripts/php-session-cleanup.sh to clean up the new path. Let's continue to monitor the situation.

[lanye74]

#281 mentions that

The file drop function just making the files laying on the disk wasting inodes

Just something to keep an eye out for :P

[tresf]

We've migrated to new hosting, this issue hasn't happened, so I think we're good.