FTL crashes: FATAL: verify_shmem_pid(): Failed to open shared memory object "FTL-settings": No such file or directory

[Speed7811]

Versions

• Pi-hole: v5.17.1
• AdminLTE: v5.20.1
• FTL: v5.23

Platform

• OS and version: Ubuntu 22.04.3
• Platform: Hyper-V Host (Windows Server 2022)

Expected behavior

FTP should not be crashing

Actual behavior / bug

After a while the FTL service seems to be crashing and restarting. I can see it on the web interface because the counters are beginning from 0.

Here an example log:

[2023-09-19 07:34:08.575 1117M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 753.7KB used, 463.1MB total, FTL uses 741.7KB)
[2023-09-19 07:35:07.597 1117M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 757.8KB used, 463.1MB total, FTL uses 745.8KB)
[2023-09-19 07:36:28.407 1117M] Resizing "FTL-domains" from 12288 to (1024 * 24) == 24576 (/dev/shm: 761.9KB used, 463.1MB total, FTL uses 749.9KB)
[2023-09-19 07:36:45.004 1117M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 774.1KB used, 463.1MB total, FTL uses 762.2KB)
[2023-09-19 07:39:21.199 1117M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 778.2KB used, 463.1MB total, FTL uses 766.3KB)
[2023-09-19 07:42:34.011 1117M] Resizing "FTL-dns-cache" from 20480 to (1536 * 16) == 24576 (/dev/shm: 782.3KB used, 463.1MB total, FTL uses 770.4KB)
[2023-09-19 07:42:34.011 1117M] FATAL: verify_shmem_pid(): Failed to open shared memory object "FTL-settings": No such file or directory
[2023-09-19 07:42:39.230 1426M] Using log file /var/log/pihole/FTL.log
[2023-09-19 07:42:39.230 1426M] ########## FTL started on pihole02! ##########

• Its a clean installed Ubuntu 22.04 (Server minimized) on a Hyper-V Host.
• Only PI-Hole is installed
• All updates are installed
• Linux Azure package is installed
• Upstream Server is cloudflared DoG

Steps to reproduce

• Waiting with the open Web UI -> Dashboard. After around 20 minutes the counters are starting from 0

Debug Token

• URL: https://tricorder.pi-hole.net/Bd74trUR/

[DL6ER]

I see two issues here:

1. The restarting after the FATAL message, and
2. That your counters restart from 0 (they shouldn't)

Concerning the restart:
This isn't a crash but, actually, any FATAL message is expected to cause a restart because recovery isn't possible (this is what makes FATAL different from ERROR.

Failed to open shared memory object "FTL-settings": No such file or directory

manes that the file /dev/shm/FTL-settings got lost, there is no way FTL can continue working when this file is removed. Note that FTL will not immediately realize when the files are deleted but only when reallocating space as files can only be removed but not truly deleted on Linux (an "invisible" copy of them stays alive until the last process has closed the file handler).

You will have to check your operating system if you have anything wiping /dev/shm possibly on a regular basis. As said, the file might already be gone for minutes until FTL actually realizes this and forces a restart. As this is the first report of this kind in four months since the release of FTL v5.23, I don't think it is a fundamental Pi-hole issue but rather something caused by our particular installation.

Concerning the "counters restart from 0":
FTL should store queries in the long-term database and recompute statistics from there on restarts. Did you intentionally disable the database or did you set a (high enough) custom privacy level?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.

[pralor-bot]

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pihole-ftl-service-crashes-when-under-light-load/70084/1

[dgaglioni]

Probably facing the same issue:

Versions
Pi-hole version is v5.18.2 (Latest: v5.18.2)
web version is v5.21 (Latest: v5.21)
FTL version is v5.25.2 (Latest: v5.25.2)

Platform

OS and version: Ubuntu 20.04.4
Platform: KVM (Unraid)

Expected behavior

FTP should not be crashing

Actual behavior / bug

Jun 22 06:13:44 home-dns pihole-FTL[108579]: [2024-06-22 06:13:44.096 108579M] Resizing "FTL-queries" from 52985856 to (950272 * 56) == 53215232 (/dev/shm: 59.1MB used, 501.8MB total, FTL uses 59.0MB)
Jun 22 06:13:44 home-dns pihole-FTL[108579]: [2024-06-22 06:13:44.196 108579M] Resizing "FTL-domains" from 2605056 to (109056 * 24) == 2617344 (/dev/shm: 59.3MB used, 501.8MB total, FTL uses 59.3MB)
Jun 22 06:13:44 home-dns pihole-FTL[108579]: [2024-06-22 06:13:44.439 108579M] Resizing "FTL-queries" from 53215232 to (954368 * 56) == 53444608 (/dev/shm: 59.3MB used, 501.8MB total, FTL uses 59.3MB)
Jun 22 06:13:44 home-dns pihole-FTL[108579]: [2024-06-22 06:13:44.439 108579M] FATAL: verify_shmem_pid(): Failed to open shared memory object "FTL-settings": No such file or directory
Jun 22 06:13:44 home-dns systemd[1]: pihole-FTL.service: Main process exited, code=exited, status=1/FAILURE
Jun 22 06:13:44 home-dns systemd[1]: pihole-FTL.service: Failed with result 'exit-code'.
Jun 22 06:13:49 home-dns systemd[1]: pihole-FTL.service: Scheduled restart job, restart counter is at 3.
Jun 22 06:13:49 home-dns systemd[1]: Stopped Pi-hole FTL.
Jun 22 06:13:49 home-dns systemd[1]: Starting Pi-hole FTL...
Jun 22 06:13:49 home-dns systemd[1]: Started Pi-hole FTL.
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] Using log file /var/log/pihole/FTL.log
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] ########## FTL started on home-dns! ##########
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] FTL branch: master
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] FTL version: v5.25.2
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] FTL commit: 8943e260
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] FTL date: 2024-05-08 20:59:50 +0100
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] FTL user: pihole
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] Compiled for x86_64 (compiled on CI) using gcc (Debian 8.3.0-6) 8.3.0
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    SOCKET_LISTENING: only local
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    AAAA_QUERY_ANALYSIS: Show AAAA queries
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    MAXDBDAYS: max age for stored queries is 365 days
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    RESOLVE_IPV6: Resolve IPv6 addresses
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    RESOLVE_IPV4: Resolve IPv4 addresses
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    DBINTERVAL: saving to DB file every minute
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    DBFILE: Using /etc/pihole/pihole-FTL.db
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    MAXLOGAGE: Importing up to 24.0 hours of log data
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    PRIVACYLEVEL: Set to 0
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    IGNORE_LOCALHOST: Show queries from localhost
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    BLOCKINGMODE: Null IPs for blocked domains
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    DBIMPORT: Importing history from database
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    PIDFILE: Using /run/pihole-FTL.pid
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    SOCKETFILE: Using /run/pihole/FTL.sock
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    MACVENDORDB: Using /etc/pihole/macvendor.db
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    GRAVITYDB: Using /etc/pihole/gravity.db
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    PARSE_ARP_CACHE: Active
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    CNAME_DEEP_INSPECT: Active
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    DELAY_STARTUP: No delay requested.
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    NICE: Set process niceness to -10 (default)
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    MAXNETAGE: Removing IP addresses and host names from network table after 365 days
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    NAMES_FROM_NETDB: Enabled, trying to get names from network database
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    EDNS0_ECS: Overwrite client from ECS information
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    REFRESH_HOSTNAMES: Periodically refreshing IPv4 names
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    RATE_LIMIT: Rate-limiting client making more than 100000 queries in 60 seconds
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    LOCAL_IPV4: Automatic interface-dependent detection of address
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    LOCAL_IPV6: Automatic interface-dependent detection of address
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    BLOCK_IPV4: Automatic interface-dependent detection of address
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    BLOCK_IPV6: Automatic interface-dependent detection of address
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    SHOW_DNSSEC: Enabled, showing automatically generated DNSSEC queries
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    MOZILLA_CANARY: Enabled
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    PIHOLE_PTR: internal PTR generation enabled (pi.hole)
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    ADDR2LINE: Enabled
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    REPLY_WHEN_BUSY: Drop queries when the database is busy
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    BLOCK_TTL: 2 seconds
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    BLOCK_ICLOUD_PR: Enabled
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    CHECK_LOAD: Enabled
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    CHECK_SHMEM: Warning if shared-memory usage exceeds 90%
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M]    CHECK_DISK: Warning if certain disk usage exceeds 90%
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] Finished config file parsing
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] Creating mutex
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.560 108736M] Creating mutex
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.561 108736M] PID of FTL process: 108736
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.561 108736M] Database version is 12
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.561 108736M] Resizing "FTL-strings" from 40960 to (81920 * 1) == 81920 (/dev/shm: 712.7KB used, 501.8MB total, FTL uses 700.8KB)
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.561 108736M] Imported 0 alias-clients
Jun 22 06:13:49 home-dns pihole-FTL[108736]: [2024-06-22 06:13:49.561 108736M] Database successfully initialized

Steps to reproduce
Issue starts randomly at least once a day, it probably started when I activated the DHCP service.

Debug Token
URL: https://tricorder.pi-hole.net/X2bnSWN7/

[DL6ER]

Sorry for the delay in replying!

The only thing I can really do is to exactly repeat the comment above - you are right it appears to be exactly the same underlying issue: #1642 (comment)

Emphasis on this takeaway message:

You will have to check your operating system if you have anything wiping /dev/shm possibly on a regular basis. As said, the file might already be gone for minutes until FTL actually realizes this and forces a restart.

As both of you are using some (even if not the same) kind of virtualization - maybe they are doing something crazy to /dev/shm. However, as you are surely not the only ones running PI-hole in a (kvm) virtualized environment, it may be something particularity about your kvm's settings... Sorry that I have no "better" reply at this point in time.

Could you maybe confirm that the file /dev/shm/FTL-settings actually exists after FTL has been started (before it crashes)?

[dgaglioni]

@DL6ER Thank you for your inputs, I just tried to get the settings and only some strange characters appeared, below the outputs.

root@home-dns:/home/pihole# cat /dev/shm/FTL-settings
l▒^▒▒root@home-dns:/home/pihole#
root@home-dns:/home/pihole#

Is there a better way to get it?

[DL6ER]

This is actually fine, the file is not meant for human consumption as the content is encoded. But this proves the file is there. Now we have to find out how it can become missing causing FTL to terminate...