Merge pull request #17 from jordanhandy/upload-all

Add commands with ability to mass upload assets

README.md

@@ -25,7 +25,10 @@ Released under [MIT](/LICENSE) by [@jordanhandy](https://github.com/jordanhandy)
 This plugin allows you to automatically upload images, video, audio and raw files pasted to Obsidian directly into your Cloudinary account (instead of stored locally).  Note:  There is functionality for media manipulation in this plugin using Cloudinary's custom parameters
 
 ## How it Works
+### Single File Upload
 ![Action GIF](https://res.cloudinary.com/dakfccuv5/image/upload/v1636859613/Nov-13-2021_22-11-40_bpei0d.gif)
+### Multi-file Upload
+![Multi File](https://res.cloudinary.com/dakfccuv5/video/upload/v1718021709/mass-note-upload_qnx5ar.mp4)
 ## Configuration
 1. Disable Obsidian Safe Mode
 2. Install the Plugin

docs/cloudinary-duplication.md

@@ -0,0 +1,24 @@
+---
+label: Cloudinary Duplication
+layout: default
+order: 600
+author: Jordan Handy
+icon: gear
+---
+## Duplication in Cloudinary
+Without proper configuration, it is highly likely that Cloudinary will not upload items as you expect.  I recommend reading [Cloudinary documentation on Upload Presets](https://support.cloudinary.com/hc/en-us/articles/360016481620-What-are-Upload-presets-and-how-to-use-them).
+
+This doc will explain some of the most important pieces of information.
+
+### Use filename or externally-defined public ID
+In your Upload Preset Settings under the Storage and Access menu, this option allows you to preserve filenames in Cloudinary.  Normally, when you upload an item to Cloudinary, it is renamed with a unique public ID.  If you would like to preserve filename, use this setting to allow for you to keep original filenames.  The Obsidian plugin is configured such that filenames will be used if this setting is enabled.
+![Externally defined public ID](https://res.cloudinary.com/dakfccuv5/image/upload/v1718108147/457f3fb2-f4c6-4235-8708-981ed138485d.png)
+
+## Unique Filename
+If **disabled**, the Unique Filename option will guard against duplication.  With this enabled, Cloudinary will append random characters to end string of every upload.  This means that even if you preserve file names, no two uploads of the same file will match and every upload will be unique.  This could lead to mass duplication.
+
+Keep this **disabled** so that if a file with the same filename already exists duplication is less-likely to occur 
+![Unique Filename](https://res.cloudinary.com/dakfccuv5/image/upload/v1718108164/46971520-e89b-4ae3-ad31-83a7790419d3.png)
+
+
+Continue to [Plugin Commands](plugin-commands.md)

docs/configuring-cloudinary.md

@@ -5,7 +5,7 @@ order: 800
 author: Jordan Handy
 icon: gear
 ---
-## Cloudinary Confiuration Steps
+## Cloudinary Configuration Steps
 1. Log in to Cloudinary and find your Cloud Name here
 ![Cloudinary Dashboard](assets/cloudinary-dash.png)
 2. Enable Unsigned Uploads
@@ -21,4 +21,10 @@ When the preset is created, it will have a "name" associated with it.  Use this
 
 !!! Note
 If you have a folder name already configured on Cloudinary under the settings for your specific upload preset (can be configured on Cloudinary itself), this folder setting will be ignored.
-!!!
+!!!
+
+## Cloudinary and Potential Duplicate Uploads
+I recommend double-checking all of your Cloudinary Upload Preset settings before you begin to use the plugin.
+[!ref Cloudinary Duplication](cloudinary-duplication.md)
+
+Continue to [Plugin Commands](plugin-commands.md)

docs/configuring-the-plugin.md

@@ -11,15 +11,16 @@ To configure the Plugin, you'll need the following information:
 - Cloudinary Cloud Name
 - Cloudinary Upload Preset
 - Cloudinary Folder Name
+- Configure the behaviour that triggers a Cloudinary upload
+  - Copy/paste from clipboard
+  - Drag and Drop
 - OPTIONAL: [Default Transformation Options](https://cloudinary.com/documentation/transformation_reference)
 - OPTIONAL: Configure which asset types you want to be uploaded to Cloudinary
   - Raw (non-media type files)
   - Audio
   - Video
   - Images
-- OPTIONAL: Configure the behaviour that triggers a Cloudinary upload
-  - Copy/paste from clipboard
-  - Drag and Drop
+- OPTIONAL: Configure specific folders that different media types should be placed into
 
 By default, only images are set to be uploaded to Cloudinary via a copy/paste from the clipboard, but this can be altered
 ![Cloudinary plugin configuration](https://res.cloudinary.com/dakfccuv5/image/upload/v1716510330/obsidian/wwhysme8vdrz6syd5skp.png)

docs/plugin-commands/backup-vault-assets.md

@@ -0,0 +1,14 @@
+---
+label: Backup Vault Assets to Cloudinary
+layout: default
+order: 700
+author: Jordan Handy
+icon: cloud
+---
+## Backup All Vault Assets to Cloudinary
+
+!!!warning
+This is an experimental feature.  Your mileage may vary as to the success of this command because of the issues already outlined in [uploading single note content](upload-single-note-cloudinary.md) and [upload all note contents](upload-all-notes-cloudinary.md).  This command attempts to read all "non-markdown" files in your vault and upload a copy of them to Cloudinary.  The "Backup Folder" option is where these files are stored.  The "Preserve File Paths" option attempts to maintain the same foldering structure you have in your vault, prepended with your specified backup folder as the root.
+!!!
+
+[See note on Cloudinary Duplication](../cloudinary-duplication.md)

docs/plugin-commands/plugin-commands.md

@@ -0,0 +1,15 @@
+---
+label: Plugin Commands
+layout: default
+order: 1000
+author: Jordan Handy
+icon: gear
+---
+## Plugin Commands
+
+The Cloudinary Uploader plugin has palette commands that can be run.  Explore them below:
+- [Upload single note assets to Cloudinary](upload-single-note-cloudinary.md)
+- [Upload all note assets to Cloudinary](upload-all-notes-cloudinary.md)
+- [Local asset file backup](backup-vault-assets.md)
+
+Continue to [Upload single note assets to Cloudinary](upload-single-note-cloudinary.md)

docs/plugin-commands/plugin-commands.yml

@@ -0,0 +1,4 @@
+label: Plugin Commands
+order: 700
+icon: gear
+expanded: true

docs/plugin-commands/upload-all-notes-cloudinary.md

@@ -0,0 +1,15 @@
+---
+label: Upload all note assets to Cloudinary
+layout: default
+order: 800
+author: Jordan Handy
+icon: cloud
+---
+## Upload All Note Assets to Cloudinary
+
+Within the command palette, run the "Upload all note files to Cloudinary" option.  This will take all local assets located in the vault and attempt to upload them to Cloudinary.
+
+See below for information:
+[!ref Upload single note files](upload-single-note-cloudinary.md)
+
+Continue to [Backup Vault Assets](backup-vault-assets.md)

docs/plugin-commands/upload-single-note-cloudinary.md

@@ -0,0 +1,27 @@
+---
+label: Upload single note assets to Cloudinary
+layout: default
+order: 900
+author: Jordan Handy
+icon: cloud
+---
+## Upload Single Note Assets to Cloudinary
+
+Within the command palette, run the "Upload files in current note to Cloudinary" option.  This will take all local assets located in the current note and attempt to upload them to Cloudinary.
+
+!!!warning
+It is **strongly** recommended to take a backup of your notes before you perform this action.
+As part of the action, your local media assets are **not deleted** when the upload to Cloudinary happens, so you can still reference them if you wanted to.  However, because of the nature of the upload, and the variability of syntax in how some users may choose to reference media, this may alter the formatting of your notes.  Use with caution.
+!!!
+
+The success / failure of this plugin largely depends on the following factors:
+1. Your Cloudinary plan -- Certain Cloudinary plans have quotas applied to their accounts.  Consult Cloudinary documentation to understand what your limits are
+2. The file types being uploaded -- Cloudinary only accepts certain file types to be uploaded to their servers.  If you try to upload an unsupported file type, the upload for that specific file will fail.  Consult [this documentation on media types](https://support.cloudinary.com/hc/en-us/articles/202520642-What-type-of-image-video-and-audio-formats-do-you-support) and [this documentation on raw types](https://support.cloudinary.com/hc/en-us/articles/202520572-Using-Cloudinary-for-files-other-than-images-and-videos) for more information.
+
+## Warning
+When you first use the option for mass note upload, you will be presented with a warning message explaining the potential dangers of your action.  By default, this warning message will be displayed **every time** you invoke the action.  If you would like to disable this message, toggle the "Hide command palette mass upload warning" option in plugin settings.
+## Demo
+View a demo of the command below:
+![Demo video - mass upload](https://res.cloudinary.com/dakfccuv5/video/upload/v1718021709/mass-note-upload_qnx5ar.mp4)
+
+Continue to [Uploading all note assets to Cloudinary](upload-all-notes-cloudinary.md)

manifest.json

@@ -5,5 +5,5 @@
   "author": "Jordan Handy",
   "isDesktopOnly": true,
   "minAppVersion": "0.11.0",
-  "version": "0.3.1"
+  "version": "1.0.0"
 }

package.json

@@ -14,12 +14,13 @@
     "eslint": "^7.30.0",
     "obsidian": "obsidianmd/obsidian-api",
     "obsidian-plugin-cli": "^0.4.5",
-    "typescript": "^4.1.5"
+    "typescript": "^4.1.5",
+    "retypeapp": "^1.11.0"
   },
   "dependencies": {
     "axios": "^0.21.1",
+    "cloudinary": "^2.2.0",
     "compressorjs": "^1.0.7",
-    "object-path": "^0.11.5",
-    "retypeapp": "^1.11.0"
+    "object-path": "^0.11.5"
   }
 }

src/commands/utils.ts

@@ -0,0 +1,189 @@
+/*--------- Utilities for Commands and async uploads -----------*/
+
+import { Notice, FileSystemAdapter, TFile } from 'obsidian';
+import path from 'path';
+import objectPath from 'object-path'
+import { NoteWarningModal } from '../note-warning-modal';
+import { v2 as cloudinary } from 'cloudinary';
+import { audioFormats, videoFormats, imageFormats } from '../formats';
+import CloudinaryUploader from '../main';
+
+// Generates a modal when command is invoked
+// References 'plugin' as we need access to settings on Cloudinary Uploader
+export function uploadNoteModal(file?: TFile, type: string, plugin: CloudinaryUploader): void {
+  new NoteWarningModal(plugin.app, type, (result): void => {
+    if (result == 'true') {
+      if (file) {
+        uploadCurrentNoteFiles(file, plugin); // If a file was passed and modal agreed
+        return;
+      } else {
+        // If no file passed, but assets were to be uploaded
+        if (type == 'asset') {
+          fetchMessages(plugin);
+          //uploadVault(plugin); // Upload vault function
+          return;
+        } else if (type == 'note') {  //! If no file passed, but 'notes' to be uploaded, this means all notes are requested.
+          const files = plugin.app.vault.getMarkdownFiles()
+          for (let file of files) {
+            uploadCurrentNoteFiles(file, plugin);
+          }
+
+        }
+      }
+    } else {
+      return;
+    }
+
+  }).open();
+}
+
+export async function uploadVault(plugin: CloudinaryUploader): Promise<string[][]> {
+  let successMessages = [];
+  let failureMessages = [];
+  //* Get all files in vault that are not
+  //* MD files, so they may be uploaded
+  const files = plugin.app.vault.getFiles()
+  new Notice("Upload of vault files started.  Depending on your vault size, this could take a long while.  Watch for error or success notices",0);
+  for (let file of files) {
+    if (file.extension != 'md') {
+      let filePath;
+      const adapter = plugin.app.vault.adapter;
+      if (adapter instanceof FileSystemAdapter) {
+        filePath = adapter.getFullPath(file.path);
+      }
+      await cloudinary.uploader.unsigned_upload(filePath, plugin.settings.uploadPreset, {
+        folder: plugin.settings.preserveBackupFilePath ? path.join(plugin.settings.backupFolder, path.dirname(file.path)) : plugin.settings.backupFolder,
+        resource_type: 'auto'
+      }).then(res=>{
+        // add to success messages array
+        successMessages.push('success')
+      },err=>{
+        // add to failure messages array
+        failureMessages.push(err.message);
+      });
+    }
+  }
+  // send messages
+  return [successMessages,failureMessages];
+}
+
+//* This function fetches messages from the mass upload job
+//* as this could take a while if the vault is larger
+export async function fetchMessages(plugin:CloudinaryUploader) : Promise<void>{
+  // After the upload action completes then
+  // retrieve data and display messages based on results.
+  uploadVault(plugin).then((data)=>{
+    if(data[0].length > 0 && data[1].length > 0){
+      new Notice("Cloudinary vault asset backup: There was some success in uploading vault files, as well as some errors.  Open Developer Tools for error information in Console",0);
+      for(let msg of data[1]){
+        console.warn(msg);
+      }
+    }else if(data[0].length >0){
+      new Notice("Cloudinary vault asset backup: This was successfully completed.  No errors to report",0);
+    }else if(data[1].length > 0){
+      new Notice("Cloudinary vault asset backup: This operation failed. Please try again",0);
+    }
+  });
+}
+export function uploadCurrentNoteFiles(file: TFile, plugin: CloudinaryUploader): void {
+  //! Read a cached version of the file, then:
+  /*
+  * Find a RegEx match for [[]] file refs
+  * Get file name and find full path of file
+  * Pass full path to Cloudinary for upload
+  * Based on answer returned, determine subfolder, then:
+  * Replace current strings with Cloudinary URLs
+  */
+  let data = plugin.app.vault.cachedRead(file).then(() => {
+    const found = data.match(/\!\[\[(?!https?:\/\/).*?\]\]/g);
+    if (found && found.length > 0) for (let find of found) {
+      let fileString = find.substring(3, find.length - 2);
+      let filePath;
+      const adapter = plugin.app.vault.adapter;
+      if (adapter instanceof FileSystemAdapter) {
+        filePath = adapter.getFullPath(fileString)
+        cloudinary.uploader.unsigned_upload(filePath, plugin.settings.uploadPreset, {
+          folder: setSubfolder(undefined, filePath, plugin),
+          resource_type: 'auto'
+        }).then(res => {
+          console.log(res);
+          let url = objectPath.get(res, 'secure_url');
+          let resType = objectPath.get(res, 'resource_type');
+          url = generateTransformParams(url, plugin);
+          let replaceMarkdownText = generateResourceUrl(resType, url);
+          data = data.replace(find, replaceMarkdownText);
+          plugin.app.vault.process(file, () => {
+            return data;
+          })
+          new Notice("Upload of note file was completed"); // Success
+        }, err => {
+          // Failure
+          new Notice("There was something wrong with your upload.  Please try again. " + file.name + '. ' + err.message, 0);
+        })
+      }
+    }
+  });
+}
+// Called to generate the output of the transformation parameters
+// that are set on uploads
+export function generateTransformParams(url: string, plugin: CloudinaryUploader): string {
+  if (plugin.settings.transformParams) {
+    const splitURL = url.split("/upload/", 2);
+    url = splitURL[0] += "/upload/" + plugin.settings.transformParams + "/" + splitURL[1];
+  }
+  if (plugin.settings.f_auto) {
+    const splitURL = url.split("/upload/", 2);
+    url = splitURL[0] += "/upload/f_auto/" + splitURL[1];
+    // leave standard of no transformations added
+  }
+  return url;
+}
+// Once upload completes, generate the resulting getMarkdown
+// that will be written to file
+export function generateResourceUrl(type: string, url: string): string {
+  if (type == 'audio' || isType(url, audioFormats)) {
+    return `<audio src="${url}" controls></audio>\n`;
+  } else if (type == 'video' || isType(url, videoFormats)) {
+    return `<video src="${url}" controls></video>\n`;
+  } else {
+    return `![](${url})`;
+  }
+}
+// Required as Cloudinary doesn't have an 'audio' resource type.
+// As we only know the file type after it's been uploaded (we don't know MIME type),
+// we check if audio was uploaded based on the most-commonly used audio formats
+export function isType(url: string, formats: string[]): boolean {
+  let foundTypeMatch = false;
+  for (let format of formats) {
+    if (url.endsWith(format)) {
+      foundTypeMatch = true;
+    }
+  }
+  return foundTypeMatch;
+}
+// Determine the subfolder to place the uploaded
+// file in, based on file type uploaded
+export function setSubfolder(file: File, resourceUrl: string, plugin: CloudinaryUploader): string {
+  if (file) {
+    if (file.type && file.type.startsWith("image")) {
+      return `${plugin.settings.folder}/${plugin.settings.imageSubfolder}`;
+    } else if (file.type.startsWith("audio")) {
+      return `${plugin.settings.folder}/${plugin.settings.audioSubfolder}`;
+    } else if (file.type.startsWith("video")) {
+      return `${plugin.settings.folder}/${plugin.settings.videoSubfolder}`;
+    } else {
+      return `${plugin.settings.folder}/${plugin.settings.rawSubfolder}`;
+    }
+  } else if (resourceUrl) {
+    if (isType(resourceUrl, imageFormats)) {
+      return `${plugin.settings.folder}/${plugin.settings.imageSubfolder}`;
+    } else if (isType(resourceUrl, audioFormats)) {
+      return `${plugin.settings.folder}/${plugin.settings.audioSubfolder}`;
+    } else if (isType(resourceUrl, videoFormats)) {
+      return `${plugin.settings.folder}/${plugin.settings.videoSubfolder}`;
+    } else {
+      return `${plugin.settings.folder}/${plugin.settings.rawSubfolder}`;
+    }
+  }
+
+}