Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[nightly] Library scan doesn’t remove non-existent series #3378

Open
newyorker123 opened this issue Nov 17, 2024 · 12 comments
Open

[nightly] Library scan doesn’t remove non-existent series #3378

newyorker123 opened this issue Nov 17, 2024 · 12 comments
Labels
needs-triage Needs to be triaged by a developer and assigned a release

Comments

@newyorker123
Copy link

What happened?

To reproduce:

  1. Delete a series folder from the file system.
  2. Scan the library in Kavita.

The issue:

The series will not be automatically removed from Kavita. To delete the series in Kavita, you must either scan it from its detail page or perform a force scan on the library. A normal scan does not remove the series.

What did you expect?

Library scan can automatically remove non-existent series.

Kavita Version Number - If you don not see your version number listed, please update Kavita and see if your issue still persists.

Nightly Testing Branch

What operating system is Kavita being hosted from?

None

If the issue is being seen on Desktop, what OS are you running where you see the issue?

None

If the issue is being seen in the UI, what browsers are you seeing the problem on?

No response

If the issue is being seen on Mobile, what OS are you running where you see the issue?

None

If the issue is being seen on the Mobile UI, what browsers are you seeing the problem on?

No response

Relevant log output

No response

Additional Notes

No response

@newyorker123 newyorker123 added the needs-triage Needs to be triaged by a developer and assigned a release label Nov 17, 2024
@DieselTech DieselTech added support User is needing support on something not working but not confirmed to be a bug and removed needs-triage Needs to be triaged by a developer and assigned a release labels Nov 17, 2024
@DieselTech
Copy link
Collaborator

Kavita works by going off the filesystem events to detect changes. If your underlying filesystem doesn't tell kavita something has changed, then we don't know about it.

I just removed a folder and ran a scan on the latest nightly to test this:

[Kavita] [2024-11-16 20:05:50.229 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library Manga Step 5: Remove Deleted Series
[Kavita] [2024-11-16 20:05:50.231 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Removing series that were not found during the scan
[Kavita] [2024-11-16 20:05:50.455 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Found 1 series to remove: A Couple of Cuckoos
[Kavita] [2024-11-16 20:05:50.701 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Series removal process completed

@newyorker123
Copy link
Author

For some reason, Kavita isn’t detecting changes after I delete a folder.

Good to know it’s not a bug in the nightly release. I’ll see what I can do to fix this on my end. Thanks!

@majora2007
Copy link
Member

I also can't reproduce as I've been deleting a lot of things during my testing.

@majora2007 majora2007 added the can't reproduce Can't reproduce, need additional information from user label Nov 17, 2024
@newyorker123
Copy link
Author

After the latest stable release, Kavita still doesn't detect when a series folder is removed during a regular scan from the side nav.

I'm running my Kavita server with all files stored on a QNAP NAS. When I delete a file, the QNAP log center logs it as "Deleted." However, Kavita still fails to recognize the change.

Currently, my Kavita only detects removed files in the following scenarios:

  1. Scanning from the series detail page.
  2. Performing a force scan.
  3. Conducting a regular scan from the side nav, but only if a new series is added. In this case, Kavita also removes the deleted series.

Can anyone help me identify the problem? It's really frustrating. Thanks in advance!

@majora2007
Copy link
Member

Let's do some targeted testing. Delete your log file for the day, then perform the scan where something was deleted. Then attach the logs here.

@newyorker123
Copy link
Author

Thank you for your help.

I have two series folders, test1 and test2, in a library directory test. I delete test2 from the filesystem and perform a regular scan for the library. The sereis test2 still exists in kavita. Here is the log:

[Kavita] [2024-11-23 12:33:38.958 +00:00  13] [Information] API.Services.TaskScheduler Enqueuing library scan for: 16
[Kavita] [2024-11-23 12:33:38.989 +00:00  13] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP POST /api/library/scan?libraryId=16&force=false responded 200 in 77.1454 ms
[Kavita] [2024-11-23 12:33:39.098 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Beginning file scan on test
[Kavita] [2024-11-23 12:33:39.130 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 1: Scan & Parse Files
[Kavita] [2024-11-23 12:33:39.175 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 1.A: Process 1 folders
[Kavita] [2024-11-23 12:33:39.177 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.B: Scan files in /media/test
[Kavita] [2024-11-23 12:33:39.186 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Step 1.C Found 1 directories to process for /media/test
[Kavita] [2024-11-23 12:33:39.189 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ProcessFiles] Skipping /media/test/test1 as it hasn't changed since last scan
[Kavita] [2024-11-23 12:33:39.189 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.C: Process files in /media/test
[Kavita] [2024-11-23 12:33:39.192 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Skipped File Scan for /media/test/test1 as it hasn't changed
[Kavita] [2024-11-23 12:33:39.193 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.D: Merge any localized series with series /media/test
[Kavita] [2024-11-23 12:33:39.196 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.E: Group all parsed data into logical Series
[Kavita] [2024-11-23 12:33:39.201 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.F: Generate Sort Order for Series and Finalize
[Kavita] [2024-11-23 12:33:39.213 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 2: Process and Update Database
[Kavita] [2024-11-23 12:33:39.215 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Found 0 Series that need processing in 79 ms
[Kavita] [2024-11-23 12:33:39.216 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Finished file scan in 79 milliseconds. Updating database
[Kavita] [2024-11-23 12:33:39.240 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 3: Save Library
[Kavita] [2024-11-23 12:33:39.275 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Finished library scan of 0 series in 290 milliseconds for test. There were no changes
[Kavita] [2024-11-23 12:33:39.275 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 5: Remove Deleted Series
[Kavita] [2024-11-23 12:33:39.277 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Removing series that were not found during the scan
[Kavita] [2024-11-23 12:33:39.279 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Found 0 series to remove: 
[Kavita] [2024-11-23 12:33:39.280 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Series removal process completed
[Kavita] [2024-11-23 12:33:39.285 +00:00  43] [Debug] API.Controllers.LibraryController Caching libraries for library_zehao
[Kavita] [2024-11-23 12:33:39.362 +00:00  43] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP GET /api/library/libraries responded 200 in 138.8310 ms
[Kavita] [2024-11-23 12:33:39.405 +00:00  43] [Debug] API.Controllers.LibraryController Caching libraries for library_zehao
[Kavita] [2024-11-23 12:33:39.408 +00:00  43] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP GET /api/library/libraries responded 200 in 9.3204 ms
[Kavita] [2024-11-23 12:33:39.412 +00:00  41] [Debug] API.Controllers.LibraryController Caching libraries for library_zehao
[Kavita] [2024-11-23 12:33:39.415 +00:00  41] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP GET /api/library/libraries responded 200 in 3.8171 ms
[Kavita] [2024-11-23 12:33:39.543 +00:00  37] [Information] API.Services.Tasks.CleanupService Performing cleanup of Cache directories
[Kavita] [2024-11-23 12:33:39.543 +00:00  37] [Information] API.Services.Tasks.CleanupService Cache directory purged

@majora2007
Copy link
Member

Can you also show me whats in the test1 and test2 directories (before you removed test2)?

Were they the same series?

@newyorker123
Copy link
Author

No, they are different series, with one file in each folder. The second one is the series in test2, which should be removed.

image

@majora2007
Copy link
Member

Please run this python script before the deletion of the folder and attach the json that comes from it. This will just create a map of the folders for me to put into a unit test framework I built around this script.

Please also give me the library type you have setup.

import os
import json


def map_files(root_dir):
    files_map = []

    for dirpath, dirnames, filenames in os.walk(root_dir):
        # Skip directories that start with "."
        dirnames[:] = [d for d in dirnames if not d.startswith('.')]

        for filename in filenames:
            # Skip files that start with "."
            if not filename.startswith('.'):
                # Get the relative path of the file
                relative_path = os.path.relpath(os.path.join(dirpath, filename), root_dir)
                files_map.append(relative_path)

    # Export the map to a JSON file
    with open('files_map.json', 'w') as outfile:
        json.dump(files_map, outfile, indent=4)


if __name__ == "__main__":
    root_dir = os.getcwd()
    map_files(root_dir)
    print("File map generated and saved to files_map.json.")

@majora2007 majora2007 added needs-triage Needs to be triaged by a developer and assigned a release and removed can't reproduce Can't reproduce, need additional information from user support User is needing support on something not working but not confirmed to be a bug labels Nov 23, 2024
@newyorker123
Copy link
Author

I run the python script in the test library root folder, here is the json, but I don't think that provide much information

[
    "try.py",
    "test1\\test1.epub",
    "test2\\test2.epub"
]

I'm using Book type library for testing. But It also happens to every library type I use (Manga, light novel and Book).

@majora2007
Copy link
Member

Thanks, that's all I need from you. When I have time I will mock this up in the code and see if I can reproduce. On my system, deleting folders has not been an issue, hence why this was marked previously as can't reproduce.

Time is extremely limited for me, so it will be a bit before I can try to reproduce.

Also does a force scan not get the series deleted?

@newyorker123
Copy link
Author

Thank you for taking the time to look into this. I completely understand how busy you are with everything related to Kavita. If there’s anything else I can assist with or clarify, please don’t hesitate to let me know.

A force scan will delete the sereis. The series will be removed in the following scenarios:

  1. Scanning from the series detail page.
  2. Performing a force scan.
  3. Conducting a regular scan from the side nav, but only if a new series is added. In this case, Kavita also removes the deleted series.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-triage Needs to be triaged by a developer and assigned a release
Projects
None yet
Development

No branches or pull requests

3 participants