1.7.2 Release

This is more of a QOL update for the codebase and the bugs that were persisting inside of it. I took the time to clean things up, get everything working a little bit smoother, as well as fixing critical high-severity bugs that were bricking portions of the program. You can view this as more of a hotfix rather than a full fledge release, and that will be evident in the release notes, but I feel this was a needed update, especially with how messy and hard it was to work on the codebase the more things got added to it. There is only a Windows build right now, but I'm looking to do Linux build for this version (and future versions) too, just need the time to sit down and set everything up for that. That might be a minute, but be patient. Enjoy this update, and have a smoother experience from here on.

I also want to note that I moved to a different TLS crate for making connections since the default one was causing "Checking if the connection is secure..." issues. This may or may not have fixed the issue, but so far it has in my testing. This may also help with the VPN issue, but that will have to be tested by those who use VPNs.

What's Changed

General

• Fixed critical bug where is_deleted could not be found when downloading pools [#67, #60, #62].
• Blacklists can now use score:.
• Fixed bug where tags containing a space at the end would attempt to create invalid directories that resulted in crashes [#59, #58, #68].

Misc

• Rolled back workflow Ubuntu version to 18.4 as the latest causes issues with OpenSSL.
• Documentation has been expanded and given more detail [#69 ... #91].
• Tons of code cleanup that are listed in the pull request (#65) below.
• Turned Config and Login into singletons for easier use and access.
• Imports are now organized across all files.
• Grabber now has all the code related to grabbing posts broken off into multiple functions, increasing readability greatly.
• Converted the programming workflow of the entire project to Gitflow for a more structured software development cycle (also reducing bugs from leaking into the main branch).

Pull Requests

• Removed is_deleted from PoolEntry by @head-gardener in #61
• Crate toml updates. by @McSib in #63
• Update workflow for this branch and update crates. by @McSib in #64
• Codebase cleanup by @McSib in #65

New Contributors

• @head-gardener made their first contribution in #61

Full Changelog: 1.7.1...1.7.2

1.7.1 Release

Time has passed since my last update, hasn’t it? It has been a while, and life has taken months of my dev time on this project away. But I am back, and I have more things for the program to refine it even more. This will mostly contain bug fixes, cleaning up parts of the project that I have overlooked, and keeping the program nice and up to date. I am thinking about doing something new for the next update, but it will take time, so I will have to balance it carefully to ensure that I can get it finished. Beyond that, enjoy the update, and resume using the program, now with some fixes that got rid of some annoying bugs that has plagued a few users.

And yes, this version has a Linux build too. ;3

Release Notes:

• You can now set the naming convention of downloaded files (md5 or id). [#47]
• The search term will now shorten in the downloader progress bar if its length is greater than 25 characters.
• The directory named after the search term will now shorten on Windows if it crosses the MAX_PATH length a path can have (260 characters). [#53]
• Fixed an incredibly rare bug where the created_at field in API responses would be null.
• Removed an unused function that was overlooked.
• Updated the parser to report parsing errors.
• Fixed numerous security vulnerabilities tied to my dependencies (I recommend updating just for this reason).

Bug Fixes:

• Located possible cause for bug that crashes the downloader at the very start of parsing or downloading (read more about this below). [#48]
• Fixed bug where tag including a colon crashed the downloader. [#51] [#52]
• Fixed bug where the program suddenly crashed when grabbing a single explicit or questionable post while in safe mode. [#54]

Misc:

• Marked regex, smallvec, and tokio to their latest versions to fix security vulnerabilities.
• Updated indicatif to 0.16.2 indefinitely until 0.17.0 releases with no performance impacts.
• Updated dialoguer to 0.10.1.
• Updated log to 0.4.17.
• Updated simplelog to 0.12.0.
• Updated reqwest to 0.11.10.
• Updated serde to 1.0.137.
• Updated serde_json to 1.0.81.
• Enforced more intense field visibility rules for structs.
• Created an auto builder to determine if the build is passing once commits are pushed.
• Fixed many Clippy warnings.
• Added debug printing for grabbing tags so that they are traceable as well.
• Created a rustfmt.yml for rustfmt to format more to a unified style I work with (this may change in the future as I figure out new things to change with it).

Notice for users who use VPNs

I have had a recurring “bug” that has shown in my issues the last couple of months, and they tend to crop up right after a new release, so I am going to supply a new notice for those using VPNs to prevent this becoming issue spam. There are users who are experiencing crashes consistently when parsing, obtaining blacklist, or downloading. It is an issue that is consistent, and each person thus far have been using a VPN with no other noticeable cause linked. After a multitude of testing, I have concluded that users using VPNs will occasionally have either e621 directly or Cloudflare prompt for a captcha, or a test for whether you are a robot. Since my program does not support GUI, or no tangible way of handling that, it will crash immediately. I have looked for fixes to this issue and have yet to find anything. So, if you are using a VPN, be warned, this can happen. The current work around for this issue is switching locations in the VPN (if you have that feature) or disabling the VPN altogether (if you have that option). I understand it is annoying, and can be a pain, but this is all I can do until I come across a fix. Sorry for the inconvenience, and apologies if you are some of the users experiencing this issue.

What's Changed

• 1.7.1 Dev Branch by @McSib in #49

Full Changelog: 1.7.0...1.7.1

1.7.0 Redesign Hotfix 1 Release

In light of an issue that popped up on my board yesterday (#45), I decided to take the time to make a hotfix release that adjusts the logging system to be much much more informative. There were questions I asked the person in that issue board that shouldn't have needed to be asked. With these adjustments to the logger, this will no longer be an issue.

With all of this provided below, I hope you have a good day. And if you experience an issue in the future, just share your log file and your issue and that should provide more than enough information for what might be happening.

NOTE:

I uploaded a Linux version of e621_downloader! This version is compiled on Arch Linux, so please let me know if it works for you or not, I will see about getting it compiled for others as well. If you are a Debian user or someone who uses a distro that is Debian based, let me know if this runs for you at all. I want to find a way to give up-to-date and ready to use Linux compilations of my downloader to users without them needing to compile the program. So far, this is one way I found. Let me know how it work, and I hope you enjoy the downloader!

Hotfix Notes:

• The logger now can print low-level information about your system, OS family, architecture, as well as executable information and DLL information.
• The logger will now be able to print out downloader specific information, such as the downloader name, version, working directory, etc.
• The logger will also print out login information, like username, API key, and whether or not you wish to download favorites. I would like to note, your API key is not visible in the .log file, the key is replaced with * and is merely there to show that the information was provided from the login.json. This is in the event there is a bug related to login issues.
• The logger can now show image save paths and is allowed to share collection information, such as collection names (typically these are search terms), collection category (pools, sets, general searches, etc.), among others.

Misc:

• The release version setup has now been heavily altered to accommodate the crate version.

Pull Requests Accepted:

• 1.7.0 hotfix 1 by @McSib in #46

Full Changelog: 1.7.0...1.7.0-hotfix.1

1.7.0 Redesign Release

Well, the 1.7.0 release is out. There isn't much done with this release, at least on the overall feature changes. In this update, I wanted to focus on primarily changing the visuals. I got tired of the black-and-white console and decided that injecting some color would make a nice difference. I also wanted a better debug time with this program. So, to do that, I incorporated a logger. This logger will not only make things easier to read, but it will also export its print-outs into a nice .log file for me to be able to read. If you have a bug in the future, all you have to do now is explain what's generally happening and then supply your log file. I can get a much deeper understanding of everything and what's happening from there. So with that, I hope you enjoy this new release!

Note: You need this new version if you wish to log in through the downloader as a recent change to e621 causes past versions to crash.

Release Notes:

• A new logging system for the program with the ability to save the logs to a log file.
• All-new color and design changes to the program, making it much more pleasing to the eye.

Bug Fixes:

• Fixed an issue with the pool name number being off when saving pool images. [#41]
• Fixed a bug where the order at which pools were saved was incorrect and often out of order. [#43]
• Fixed a crashing bug where the program would completely crash if the user logged in. [#44]

Misc:

• Did a massive code clean up that made everything generally faster and more concise to read through.
• Removed comments that no longer applied to the project code.
• Updated base64-url from 1.4.8 to 1.4.13
• Updated reqwest from 0.11.0 to 0.11.9
• Updated serde from 1.0.118 to 1.0.136
• Updated serde_json from 1.0.61 to 1.0.79
• Added dialoguer at version 0.10.0
• Added console at version 0.15.0
• Added log at version 0.4.14
• Added simplelog at version 0.11.2

Known Issues:

• indicatif will no longer be updated for some time as it currently has a performance issue that causes a 98% performance decrease in active release runtime. Currently, the version of indicatif used in the project is the closest up-to-date version without this performance impact.

Full Changelog: 1.6.1...1.7.0

1.6.1 In The Future Hotfix

So while the main release has been out, there has been some bugs popping up here and there, and I've been waiting for most of them to show, and I feel it's finally shown enough for me to warrant a new release. This is just a basic hotfix with bug fixes, nothing new or impressive, so enjoy!

Release Notes:

Bug fixes:

• Fixed a bug where the validator had to be either a tag or alias, giving no option to emergency exit encase of typo or invalid tag [#39]
• Fixed bug where the pool name for the image file conflicted with OS file and folder name restrictions, causing unexpected behavior and downright crashing on some occasions [#37] [#38]
• Fixed bug where the program would crash after not being able to figure out category a tag was [#40]

Misc:

• Backpedaled to a previous form of the safe mode selection as the current was starting a separate thread/output and letting the program run beyond the prompt itself, opening a window for bugs in the futures.

1.6.0 In the Future Release

The 1.6.0 In the Future release is here, and there was a lot I had altered, the list will contain everything that was altered, so be prepared for a very nerdy list. Have fun~

Also, before I fully get to the release notes, I must clarify that there is now a filter with e621. If you are not logged into E6, a filter (almost like a global blacklist) is applied with the entry young -rating:s. This blacklist will nullify any posts that fall under it. If you wish to download any posts of this kind, you must log in and download it, otherwise, this filter will be there.

Release Notes:

• Downloader now works with the new API on e621 [#30]
• Blacklist system updated and reworked to provide much greater accuracy and performance [#34]
• Login system uses a much safer method to authenticate (the safer method was added with the new version of e621)
• The server now conforms to a new filtering system that nullifies the file URL to posts containing the tag young and not having the post rating set to safe [#31]
• Users no longer need their ID in the login.json file to log into their accounts (this is a specific note from those who used the indev release of 1.6.0)
• The flagging system for the blacklist on the downloader now supports negated ratings, users, and IDs.

Bug fixes:

• Fixed massive memory leak in the blacklist system [#33]
• Fixed issue where the program would crash if the user's blacklist is empty [#35]
• Fixed issue where deleted posts and filtered posts (part of the new API) would crash the downloader while it was grabbing and downloading posts [#31, #32]
• Fixed bug where the user was required to log in, rather than logging in being an option
• Fixed bug where the blacklist would completely ignore blacklisted users

Misc:

• Added new depenancy base64-url at version 1.2.0 for the new authentacation system
• Updated dialoguer from 0.5.0 to 0.6.2
• Updated indicatif from 0.13.0 to 0.15.0
• Updated reqwest from 0.10.1 to 0.10.6
• Updated serde from 1.0.104 to 1.0.112
• Updated serde_json from 1.0.44 to 1.0.55
• Updated dialoguer from 0.1.6 to 0.1.8

1.6.0 Indev 2

This update is a massive rewrite and fix of the blacklist system. The previous version contained a huge memory leak I had not known about and came to the conclusion that the system needed to be updated which was stated here [#34]. This build should be completely usable.

This build is still in debug mode and will be until it is fully stable and released.

I hope you enjoy this, hopefully, more functional build.

1.6.0 Indev

This is a source mock-up of what is currently working in my software. This will only contain a debug build of the current version. No release builds are allowed right now as I need errors to print in large. This program will run slower than the more optimized release build, but in turn, it will allow you to see what is currently going on. There are also changes to the software that may take a moment to get used to, but this will all be leveled out when the build is stabilized. Till then, enjoy this debug build~!

1.5.6 Pools Release

Release Notes

• Pools are now downloaded in its entirety instead of just downloading one page.
• Pools' posts are now numbered when downloading, so reading them is much easier.

Tech-savvy Release Notes

• Better formating on tags.txt.
• When a duplicate is skipped, it will now increment the file_size before continuing to the next post to download.
• Removed the "Favorites" category when downloading the user's favorites.
• Documented code I forgot to document.
• Updated reqwest from 0.9.24 to 0.10.1.
• Updated serde from 1.0.103 to 1.0.104
• The progress bar now fills the line it's on.
• Removed README.txt

1.5.5 Progress Release

Release Notes:

• Replaced the progress bar with a better one. This one can now show the total file size you need to download, and how far along you are.
• The prompt for when you want to enter safe mode is now handled through a better system.

Tech-savvy Release Notes:

• Fixed bug where the images became directories.
• Renamed function get_length_posts to get_file_size_from_posts.
• Made the download_set function much more readable.
• Reverses array when grabbing posts instead of reversing them right before downloading them.
• Added new progress bar that displays the size of bytes that are going to be downloaded, instead of how many images.
• Small variable rename in FlagWorker
• Fixed bug where the flags raised would pass the margin because the artist tags were being flagged twice.
• Grabber now manages the Blacklist completely, not relying on the WebConnector for the information anymore.
• Blacklist is no longer created and parsed every time a new page of posts are grabbed.