Merge pull request #7 from j54j6/dev

Added Output format profiles

.github/workflows/functionality_tests.yml

@@ -63,7 +63,7 @@ jobs:
         shell: bash
       - name: Import subscription
         run: |
-          python3 ./yt_manager.py import-subscriptions ./ytdownloader/subscriptions_export.json True
+          python3 ./yt_manager.py import-subscriptions ./ytdownloader/subscriptions_export.json --overwrite True
           EXIT_CODE=$?
           if [ $EXIT_CODE -eq 0 ]; then
             echo "Command succeeded with exit code 0"

README.md

@@ -1,5 +1,4 @@
 
-
 [![GPLv3 License](https://img.shields.io/badge/License-GPL%20v2-green.svg)](https://opensource.org/licenses/)
 
 [![CodeFactor](https://www.codefactor.io/repository/github/j54j6/youtubedl-downloader/badge)](https://www.codefactor.io/repository/github/j54j6/youtubedl-dowloader)
@@ -33,6 +32,7 @@ For Reference please look in the project.json section
 - Create and import Backups
 - Validate your FS
 - Show duplicates
+- Format profiles (globally and per custom download / subscription)
 
 # Currently supported Sites
 - Pinterest
@@ -68,22 +68,22 @@ Options:
 ```
 With the provided command only subscriptions created with the youtube or reddit scheme are shown.  If no filter is passed all subscriptions will be shown like below. If the scheme changes a divider is inserted
 ```
-+----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+
-| ID | Name           | Scheme  | Avail. Videos | Downloaded Videos |     Last checked    | url                                                  |
-+----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+
-| 1  | test           | reddit  |       29      |         0         | 2024-05-19 11:59:55 | https://reddit.com/                                  |
-+----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+
-| 2  | @AlexiBexi     | youtube |      828      |         0         | 2024-05-19 12:07:04 | https://www.youtube.com/@AlexiBexi/videos            |
-| 3  | @Lohntsichdas  | youtube |      181      |         0         | 2024-05-19 14:28:50 | https://www.youtube.com/@Lohntsichdas/videos         |
-| 4  | @Finanzfluss   | youtube |      635      |         0         | 2024-05-19 14:29:05 | https://www.youtube.com/@Finanzfluss/videos          |
-| 5  | @DoktorWhatson | youtube |      288      |         0         | 2024-05-19 14:29:25 | https://www.youtube.com/@DoktorWhatson/videos        |
-+----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+
++----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+---------------
+| ID | Name           | Scheme  | Avail. Videos | Downloaded Videos |     Last checked    | url                                                  |output-format |
++----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+--------------+
+| 1  | test           | reddit  |       29      |         0         | 2024-05-19 11:59:55 | https://reddit.com/                                  |m4a           |
++----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+--------------+
+| 2  | @AlexiBexi     | youtube |      828      |         0         | 2024-05-19 12:07:04 | https://www.youtube.com/@AlexiBexi/videos            |              |
+| 3  | @Lohntsichdas  | youtube |      181      |         0         | 2024-05-19 14:28:50 | https://www.youtube.com/@Lohntsichdas/videos         |              |
+| 4  | @Finanzfluss   | youtube |      635      |         0         | 2024-05-19 14:29:05 | https://www.youtube.com/@Finanzfluss/videos          |              |
+| 5  | @DoktorWhatson | youtube |      288      |         0         | 2024-05-19 14:29:25 | https://www.youtube.com/@DoktorWhatson/videos        |              |
++----+----------------+---------+---------------+-------------------+---------------------+------------------------------------------------------+--------------+
 ```
 
 ### Add Subscriptions
 To add a subscription the overview url of the channel/playlist is needed. For example:
 ```
-        yt_manager.py add-subscription https://www.youtube.com/@AlexiBexi/
+        yt_manager.py add-subscription https://www.youtube.com/@AlexiBexi/ --output-format <<format>>
 ```
 or if you want to add multiple
 ```
@@ -112,6 +112,46 @@ or
         yt_manager.py del-subscription https://www.youtube.com/@AlexiBexi
 ```
 
+# Format handling
+If you want to change the output format you can create profiles (or use the pre defined ones)... 
+Currently you can only edit / add profiles in the db or add them manually inside the "formats.json" file. TZhe program will import the data automatically
+
+You can define the output format in different ways:
+  ## General Info
+  In any case: If you define an output profile (for custom downlaods or subscriptions) they overrule all other settings!
+  ### Subscriptions:
+  If you create a new subscription you can pass the "--output-format" flag and pass a profile in which all data will be downloaded (list is possible)
+  ### Custom downlaods
+  For custom videos you can also pass the "--output-format" flag to define the output format
+
+  ### Nothing passed (Global)
+  If nothing is passed you need to use the global settings. Profiles can be enabled or disabled. If enabled
+  all filedownloads without explicit output format passed will be downloaded using ALL enabled profiles. 
+  You can have multiple profiles enabled at the same time!
+  ### Fallback
+  The fallback if something breaks is "best"... You can change it in the config.ini file!
+
+  ## Commands
+  ### Show defined profiles and states
+
+  This command allows you to list all available profiles and see if profiles are enabled or disabled
+  ```
+        yt_manager.py show-format-profiles
+  ```
+
+  ### Disable a profile (globally)
+  You can disable profiles for global use (not counted for subscriptions!)
+  ```
+        yt_manager.py disable-format-profile <<name>>
+  ```
+
+  ### Enable a profile (globally)
+  You can enable profiles by using this command. Important: Multiple profiles can be active. You can use the 
+  optional flag "only_active" to disable all other enabled profiles!
+  ```
+        yt_manager.py enable-format-profile <<name>> --only_active
+  ```
+
 ## Backup functionalities
 ### Export Subscriptions
 You can create a backup file of your subscriptions. The file will be saved in your base dir (defined in config scheme/db)
@@ -166,7 +206,7 @@ A batch mode is planned but not implemented yet!
 
 Example use:
 ```
-        yt_manager.py custom https://www.youtube.com/watch?v=gE_FuQuaKc0
+        yt_manager.py custom https://www.youtube.com/watch?v=gE_FuQuaKc0 --output-format <<format>>
 ```
 or if you want to download multiple links
 ```
@@ -363,6 +403,4 @@ subscription_data => metadata of the subscription. This field can be very very l
  - [PrettyTables](https://github.com/jazzband/prettytable)
  - [TLDExtract](https://github.com/john-kurkowski/tldextract)
  - [Validators](https://github.com/python-validators)
- - [Awesome Readme Templates](https://awesomeopensource.com/project/elangosundar/awesome-README-templates)
-
-
+ - [Awesome Readme Templates](https://awesomeopensource.com/project/elangosundar/awesome-README-templates)

config.ini

@@ -8,3 +8,4 @@ db_pass = password
 
 [other]
 timezone=Europe/Berlin
+fallback_format=best

database_manager.py

@@ -216,6 +216,41 @@ def prepare_sql_create_statement(name, scheme):
     query +=");"
     return query
 
+def prepare_sql_add_column_statement(table_name, column_name, options):
+    """ This function is used to add columns to an existing table based on a defined json scheme.
+        Check documentation for help
+
+        Return Values:
+            - None -> Error
+            - SQL Statement
+    """
+    query:str = f"ALTER TABLE {table_name} ADD COLUMN {column_name}"
+
+    #Iterate over all defined columns. Check for different optionas and add them to the query.
+
+    #For each column create a cache query based on SQL -> <<Name>> <<type>> <<options>>
+    if not "type" in options:
+        #PyLint C0301
+        logger.error("""Error while creating table! -
+                     Column %s does not include a valid \"type\" field!""", column_name)
+        return None
+    c_query:str = ""
+    c_query += " " + options["type"]
+    if "not_null" in options and options["not_null"] is True and "default" in options and options["default"] != "":
+        c_query += " NOT NULL"
+    if "primary_key" in options and options["primary_key"] is True:
+        logger.warning("Altering a table and changing the primary key is NOT supported!")
+    if "auto_increment" in options and options["auto_increment"] is True:
+        logger.warning("Altering a table and changing the primary key is NOT supported! -> Auto increment is also disabled...")
+    if "unique" in options and options["unique"] is True:
+        c_query += " UNIQUE"
+    if "default" in options:
+        c_query += " DEFAULT " + options["default"]
+    query += c_query + ", "
+    query = query[:-2]
+    query +=";"
+    return query
+
 def create_table(name:str, scheme:json):
     """This function can create a table bases on a defined JSON scheme
 
@@ -274,6 +309,52 @@ def create_table(name:str, scheme:json):
         logger.error("Error while creating table %s Error: %s", name, e)
         return False
 
+def check_scheme_match(table_name: str, scheme:json):
+    """ 
+    This function is used to check if a given (existing) table is matching a given scheme (it checks if all columns of the scheme actually existing inside the db table)
+    If there are missing columns they will be added (but nothing removed!)
+    """
+    if not db_init:
+        init = check_db()
+        if not init:
+            logger.error("Error while initializing DB")
+            return False
+    #Check if the table already exist. If so - SKIP
+    if not check_table_exist(table_name):
+        logger.warning("Table %s does not exist! - Can't check if the table matches a scheme...", table_name)
+        return False
+
+    #Fetch all rows of the table
+    cursor = cursor = ENGINE.cursor()
+    cursor.execute("SELECT * from " + table_name)
+    ENGINE.commit()
+
+    names = list(map(lambda x: x[0], cursor.description))
+
+    missing_columns:list = []
+
+    for needed_column in scheme:
+        if not needed_column in names:
+            print(needed_column)
+            print("Is missing")
+            missing_columns.append(needed_column)
+
+    if len(missing_columns) > 0:
+        logger.info("Table %s misses %i columns. Add missing columns...", table_name, len(missing_columns))
+        for missing_column in missing_columns:
+            try:
+                sql_statement = prepare_sql_add_column_statement(table_name, missing_column, scheme[missing_column])
+                cursor = cursor = ENGINE.cursor()
+                cursor.execute(sql_statement)
+                ENGINE.commit()
+                return True
+            except sqlite3.Error as e:
+                logger.error("Error while adding column %s to table %s Error: %s", missing_column, table_name, e)
+                return False
+    else:
+        logger.info("Table %s is up to date...", table_name)
+        return True
+
 def fetch_value(table:str, conditions:dict|list=None, data_filter:dict|list = None,
                 is_unique=False, extra_sql=None):
     """ Fetch a value from a database based on a json filter {""} """