botify 1.6 LTS

botify 1 LTS

This long-term-support version of botify 1.6 marks the end of development for botify 1 while development shifts to botify 2 featuring a new web client and improved queue management.
As an LTS release this version will continue to receive bug fixes even after botify 2 is released.

update 1.6.LTS.21 28/02/2021

• Implemented rate limiting for command submissions
  • Commands can no longer be submitted more frequently than 4 times in 5 seconds with a timout of 1 second when acquiring the permission. If it cannot be acquired or if the execution queue is full the command is not executed and a 15 second timeout is raised.
• Updated dependencies
  • Updated several dependencies.

update 1.6.LTS.20 26/01/2021

• Handle audio event in separate thread
  • When using jda-nas all audio events are handled by the same thread that populates the native buffer, meaning when one playback finishes a track and loads the next track it blocks populating the native buffer for all playbacks. This is not an issue without jda-nas because in that case the events are handled by the audio connection thread of the specific guild.
• Check additional permissions when sending message
  • Check MESSAGE_EMBED_LINKS when sending embeds and MESSAGE_ATTACH_FILES when attaching files.
• YouTubeService#populateList: fix completing last item
  • The last item was never completed due to incorrect index comparison.

update 1.6.LTS.19 17/01/2021

• Change default launch option to running as jar
  • Use maven-shade-plugin to enable building shaded jar with all dependencies and adjust launch script to use jar as launch option and add HEAP_SIZE property to specify the -Xmx flag.
• Add property to adjust or disable native buffer duration
  • Add NATIVE_AUDIO_BUFFER property to enable adjusting the duration of the native buffer used by jda-nas. Also allows to disable jda-nas by setting the value to 0.
• Other improvements
  • Various minor improvements.

update 1.6.LTS.18 11/01/2021

• Use jda-nas to enable native audio streaming
  • jda-nas enables using native code to buffer and stream audio to reduce stuttering caused by garbage collection activity if supported by the current platform.

update 1.6.LTS.17 08/01/2021

• Improved queue widget
  • The queue widget now edits its message on reset rather than deleting and resending.
• Improved login command
  • The output of the login command was improved and the link is deleted after usage to signal that it cannot be used anymore.

update 1.6.LTS.16 31/10/2020

• Improved command parsing
  • Improve how arguments and the argument prefix is handled. If an argument prefix is followed by whitespace or is the last character of the command input it is treated as command input or as part of the argument value of the current inline argument definition.
  • Show better error message when encountering a malformed argument definition with an empty argument identifier.
  • Treat the character '=' as part of the argument value if already recording the value.
    • Previous encountering '=' would simply set isRecordingValue to true, which means it is essentially ignored if isRecordingValue is already true.
  • Throw an IllegalEscapeCharacterException if the CommandParser is in escaped mode when terminating, i.e. the last character was the escape character.
    • Previously the escape character would essentially do nothing in that case, which is more confusing to the user than the clear error message they now receive.
  • ArgumentPrefixProperty: validate that the argument prefix is not a reserved meta character as this could lead to unexpected behavior.
• other improvements and fixes

update 1.6.LTS.15 29/10/2020

• Improved Spotify to YouTube redirection
  • The algorithm to find the best YouTube video for a given Spotify track has been improved. Note that you might not notice an improvement right away as the old results are still stored and will get updated gradually over the coming month.
• Improved track loading error messages
  • Error messages shown when failing to load a track now reliably show the correct track when the queue has already skipped to the next track.
• Update YouTube API quota cost calculation
  • Update how the bot calculates YouTube API quota usage based on the changed quota costs.
• Various fixes, stability improvements and important dependency updates.

update 1.6.LTS.13 10/02/2020

• added last_used attribute to SpotifyRedirectIndex and adjust RefreshSpotifyRedirectIndicesTask to only regenerate indices that have been used within the last two weeks, else the index is deleted in order to reduce requests

update 1.6.LTS.11 08/01/2020

• added support for adding IPv6 blocks for IP rotation when executing certain requests

update 1.6.LTS.4 14/11/2019

• introduce automatic YouTube API quota management
  • calculate the approximate YouTube API quota cost for each request
  • when the quota usage reaches a certain percentage of the total
    daily quota limit specified in settings.properties, botify will
    switch to using lavaplayer to parse the html response instead of
    using the youtube API wherever possible
  • create CurrentYouTubeQuotaUsage entity with a single row that
    monitors the current quota usage
  • create cron job to reset the quota usage every day at midnight PST,
    which is when YouTube resets the quota
  • create startup task to reset the quota usage if it hasn't been
    updated or reset today, i.e. if the bot was offline when the cron job
    was supposed to run
• introduce JDA sharding
  • as the production system approaches 2500 guilds JDA requires to split
    the load across multiple shards with separate WebSocket connections
    or Discord will refuse to connect
  • use the ShardManager instead of a single JDA instance
  • the PropertiesLoadingService has been updated with extra convenience
    methods
• adjust some EmbedMessages to conform to the Discord design update

backend improvements

• added liquibase framework to run custom change sets on startup for schema changes that are not executed by hibernate hbm2ddl
• added entity validation through JPA annotations and created the EntityValidationInterceptor to validate entities during a transaction and send validation error messages to discord
• rework ISnowflakeMap to improve performance and thread safety
  • never iterate over the entry set but access the id map of the ISnowflakeEntrySet directly
• introduce VersionManager to streamline version management
• other minor fixes and improvements
  • YouTubeService#populateList: check if the PlaylistItems size does not exceed the amount of HollowYouTubeVideos generated for the playlist in case the playlist grew between checking its size and loading its videos to avoid and IndexOutOfBounds
  • CleanDbCommand: count guilds for which there is still a GuildContext cached as active; in case the guild returns before the GuildContext is cleared the GuildContext should not refer to a deleted GuildSpecification
  • ChartsCommand: skip Playables that are not available anymore
  • VoiceChannelListener#noOtherMembersLeft: ignore bots in the voice channel since leave and join events for bots are ignored

botify 1.6.5

final release of botify 1.6

The last release of botify 1.6 focuses on improving and optimising existing features and the codebase. This update prioritised lowering the usage of the YouTube API quota, improving the Spotify redirect algorithm and fixing remaining issues.

• persist Spotify redirection results
  • create SpotifyRedirectIndex to persist the resulting YouTube videos in order to speed up the redirection process and save YouTube API quota
  • create RefreshSpotifyRedirectIndicesTask to refresh SpotifyRedirectIndices that have not been updated in 2 weeks
• Spotify redirect algorithm: improve the calculation of the Levenshtein distance
  • try several different combinations of track name plus artist and featuring artists to find the lowest possible Levenshtein distance for each proposed YouTube video title compared to the Spotify track
• rework WidgetActions to be executed via the CommandInterceptorChain
  • extract new Command interface from the AbstractCommand class
  • AbstractWidgetActions now implement this interface and are executed by the CommandInterceptorChain, like normal text-based commands
  • WidgetActions are now configured as XML contributions and instantiated per execution with a CommandContext instead of when setting up the widget
  • since AbstractWidgetActions are now executed via the CommandInterceptor chain there will now be a CommandHistory entry for executed widget actions, that means guilds that interacted with the bot via widgets will count as active guilds and will therefore not be cleaned up by the PlaybackCleanupTask (e.g. if a guild was playing a really long track and paused just before the task starts using the NowPlayingWidget, the playback would get cleaned up instantly)
• refactor ClientQuestion and Widget management
  • moved management of active Widgets away from the CommandManager to a new class
    • there is one instance of this new WidgetManager per GuildContext
  • moved management of open ClientQuestionEvents away from the GuildContext to a new class
    • there is one instance of this new ClientQuestionEventManager per GuildContext
  • enabled synchronisation for those new managers to avoid concurrent modification when iterating over the ClientQuestionEvents / Widgets
• other minor improvements and fixes

botify 1.6.4

album covers, automatic Spotify track selection, automatic playback cleanup and other improvements

• when searching a Spotify track, botify now automatically picks the best result based on the likeness of the track names, track popularity and how popular each artist is in this guild
  • a selection of spotify tracks to choose from can now be fetched using the refactored "select" (formerly "limit") argument
  • the refactored "select" argument also does not necessarily require a value anymore but uses 10 (for YouTube) or 20 (for Spotify) as default value
• added album covers to the queue and "now playing..." widgets for Spotify tracks
• added a periodic task that resets playback settings and queues after an hour of inactivity to preserve resources and ensure long term stability
  • created framework to implement such tasks as cron jobs
• questions now get deleted when they are answered or otherwise destroyed either by timing out or being replaced by a different question and "still loading..." messages disappear once the reference command is finished
• other improvements and fixes
• backend improvements:
  • Launcher: make sure all guilds are setup within one session instead
    of opening and closing a new one for each by wrapping the loop
    inside one StaticSessionProvider#invokeWithSession call to make sure
    a thread session is setup and closed after all guilds are done
    • also setup guilds before running the startup tasks as some might
      require a guild context which meant guilds were set up twice,
      once implicitly when needed in the task and once explicitly when
      setting up the guilds later
  • VersionUpdateAlertTask: make sure all guilds are handled within one
    session instead of opening and closing one for each when sending an
    update alert to each guild
• added number of currently active guilds to the analytics command
• when loading playlists videos that were originally redirected Spotify tracks are now loaded and treated as Spotify tracks using the new BulkOperationService

botify 1.6.3

additional commands and properties and other improvements

• create SynchroniseCommand to synchronise a botify playlist with an external playlist
  • in essence, this is an extension to the AddCommand that clears the playlist first
• create EmptyCommand to clear a botify playlist of all items
• add DefaultTextChannelProperty to customise the default channel per guild
  • MessageService#acceptForGuild now sends messages targeted towards a guild with no channel specified to the customised defaultTextChannel if possible
  • create AmbiguousCommandException to interrupt the command and ask a ClientQuestion from outside the AbstractCommand class during a command execution
• enable automatic removal of success and common error messages
  • create property to define the amount of seconds after which temporary messages are deleted
    • a timeout of 0 means there is not timeout
  • success and common error messages are now temporary messages that
    are removed after the time specified by the tempMessageTimeout
• several fixes and improvements

botify 1.6.2.1

improve concurrent command handling and other improvements

• create abort command to interrupt current commands and cancel
  queued commands
• commands can now be set to ignore the queue and run even if the
  queue is full
  • applied to admin commands and the new abort command
• widgets execution threads now do not get added to the queue but
  always run
• add size limit to AudioQueue
• adjust add command to only use the invoke method when needed and not
  put the entire command into an invoke call so that several add
  commands entered by the same guild can load tracks at the same time
  but not alter the playlist at the same time

botify 1.6.2

command parser, expand ClientQuestEvents, argument prefix, JDA 4

major new features

• create CommandParser that enables escaping meta characters through
  an escape character "\" or using single or doubles quotes and makes
  using arguments smarter and less strict
  • inline arguments are now treated as regular arguments and can be
    used as such, when used as an inline character the value of the
    argument will be the input following the argument up to the next
    argument. The order in which inline arguments are used no longer
    matters. Meaning insert a $to b $at c could also be written
    insert a $at c $to b or even insert $to=b $at=c a.
  • argument values may now contain whitespace by putting the value in
    quotes like command $arg="some value"
  • enables using escape characters and quotes to escape meta characters
    e.g. play $spotify \$trackname or play $spotify "$trackname"
• enable selecting several options comma separated when asked a question
• add an option to select all options when asked a question by certain
  commands
• add property to customise argument prefix
  • create CommandParseListener for the new CommandParser to update
    argument prefixes in existing Presets when changed
• add monthly charts to charts command
• help command examples now use the custom prefixes

other technical improvements

• upgrade to JDA 4
• refactor PlayableFactory to create Playables for Spotify albums and
  playlists and YouTube playlists when they appear in the provided list
  of objects for any method that returns a List of Playables
  • this was done to simplify #withUserResponse implementations if the
    user uses the new feature to select several options
• create AudioManager#startPlayback and #startOrResumePlayback
  instead of relying on Playable#matches to automatically determine
  whether the current track is the track that should be played or if the
  skip or rewind commands / actions were used because in rare cases the
  Spotify id might be null in which case the method is unreliable
• add Playable#getDisplayNow and #getDurationNow to use in queue view
  handler instead of using a timeout to further reduce loading time
• make Playable methods throw an UnavailableResourceException instead of
  InterruptedException when trying to access a cancelled item because
  an InterruptedException does not quite make sense
• make the the permission all attribute ignore admin commands
• optimise ThreadExecutionQueue to synchronise #add and #runNext to make
  sure a thread can't be added at the same time as another one finishes
  and tries starting the next one, creating a window where potentially
  one thread to many could run at the same time
• move GuildManager#getNameForGuild and #getPrefixToGuild to the
  GuildContext so the GuildContext
• create AdditionalInformationException for exception classes that
  show addition hints, separating the long hint from the error message
• create session in CommandListener to load bot name and prefix with
  instead of creating a session in both methods

botify 1.6.1.4

improved Spotify redirect algorithm and added more javadoc

• the redirection of Spotify tracks to YouTube has been made smarter by generating a match score based on whether or not the artist appears in the channel title, the similarity of the track / video titles and the view count
• a lot of new javadoc has been written for previously undocumented classes
• added handling for rare cases when loading a Spotify playlist's tracks, getting the Track for a PlaylistTrack might result in null

botify 1.6.1.3

Final stability update for botify 1.6.1

Last updates:

• update lavaplayer dependency to fix loading some YouTube videos
• print java version on startup

botify 1.6.1.2

enable silent updates and improve some old code

• enable silent updates, meaning no update notification will be sent if there are no changes that might interest the user
  • the reboot, quit and cleandb commands can now also be used without informing the active guilds
• fix export command failing when an early flush is caused by the query for artists when creating a Song by persisting the Playlist first to avoid a TransientObjectException
• improve upload command and Playlist#asTrackList
  • Playlist#asTrackList was an old inefficient method that loaded
    each track separately, which could cause a TooManyRequestsException;
    now it loads the maximum of 50 tracks per request
  • remove the Thread#sleep
    • the TooManyRequestsException that would get thrown there was just
      a product of the bad Playlist#asTrackList method
• fix Playable#matches throwing a NullPointerException when the parameter is null by returning false

botify 1.6.1.1

additions and improvements to admin tools and fix remaining issues

• create CleanDbCommand and GarbageCollectCommand
  • the CleanDbCommand cleans all data associated to guilds the bot is no longer part of
  • the GarbageCollectCommand start a manual garbage collection to free up memory when convenient
• fixed appending the UpdateCommand output as file when too long
• fixed AudioManager#playTrack behaviour when unpausing the playback after the bot left the channel
• improve performance of MessageService#sendToActiveGuilds
  • instead of looping over each guild and firing up to two queries per guild, only create two queries in total to find all guilds that recently played a track or entered a command
• make sure that operations that use Botify#shutdownListeners re-register the listeners when an exception happens
• show properties with null values as "Not Set"