-
-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #17 from jordanhandy/upload-all
Add commands with ability to mass upload assets
- Loading branch information
Showing
16 changed files
with
551 additions
and
54 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
--- | ||
label: Cloudinary Duplication | ||
layout: default | ||
order: 600 | ||
author: Jordan Handy | ||
icon: gear | ||
--- | ||
## Duplication in Cloudinary | ||
Without proper configuration, it is highly likely that Cloudinary will not upload items as you expect. I recommend reading [Cloudinary documentation on Upload Presets](https://support.cloudinary.com/hc/en-us/articles/360016481620-What-are-Upload-presets-and-how-to-use-them). | ||
|
||
This doc will explain some of the most important pieces of information. | ||
|
||
### Use filename or externally-defined public ID | ||
In your Upload Preset Settings under the Storage and Access menu, this option allows you to preserve filenames in Cloudinary. Normally, when you upload an item to Cloudinary, it is renamed with a unique public ID. If you would like to preserve filename, use this setting to allow for you to keep original filenames. The Obsidian plugin is configured such that filenames will be used if this setting is enabled. | ||
![Externally defined public ID](https://res.cloudinary.com/dakfccuv5/image/upload/v1718108147/457f3fb2-f4c6-4235-8708-981ed138485d.png) | ||
|
||
## Unique Filename | ||
If **disabled**, the Unique Filename option will guard against duplication. With this enabled, Cloudinary will append random characters to end string of every upload. This means that even if you preserve file names, no two uploads of the same file will match and every upload will be unique. This could lead to mass duplication. | ||
|
||
Keep this **disabled** so that if a file with the same filename already exists duplication is less-likely to occur | ||
![Unique Filename](https://res.cloudinary.com/dakfccuv5/image/upload/v1718108164/46971520-e89b-4ae3-ad31-83a7790419d3.png) | ||
|
||
|
||
Continue to [Plugin Commands](plugin-commands.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
--- | ||
label: Backup Vault Assets to Cloudinary | ||
layout: default | ||
order: 700 | ||
author: Jordan Handy | ||
icon: cloud | ||
--- | ||
## Backup All Vault Assets to Cloudinary | ||
|
||
!!!warning | ||
This is an experimental feature. Your mileage may vary as to the success of this command because of the issues already outlined in [uploading single note content](upload-single-note-cloudinary.md) and [upload all note contents](upload-all-notes-cloudinary.md). This command attempts to read all "non-markdown" files in your vault and upload a copy of them to Cloudinary. The "Backup Folder" option is where these files are stored. The "Preserve File Paths" option attempts to maintain the same foldering structure you have in your vault, prepended with your specified backup folder as the root. | ||
!!! | ||
|
||
[See note on Cloudinary Duplication](../cloudinary-duplication.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
--- | ||
label: Plugin Commands | ||
layout: default | ||
order: 1000 | ||
author: Jordan Handy | ||
icon: gear | ||
--- | ||
## Plugin Commands | ||
|
||
The Cloudinary Uploader plugin has palette commands that can be run. Explore them below: | ||
- [Upload single note assets to Cloudinary](upload-single-note-cloudinary.md) | ||
- [Upload all note assets to Cloudinary](upload-all-notes-cloudinary.md) | ||
- [Local asset file backup](backup-vault-assets.md) | ||
|
||
Continue to [Upload single note assets to Cloudinary](upload-single-note-cloudinary.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
label: Plugin Commands | ||
order: 700 | ||
icon: gear | ||
expanded: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
--- | ||
label: Upload all note assets to Cloudinary | ||
layout: default | ||
order: 800 | ||
author: Jordan Handy | ||
icon: cloud | ||
--- | ||
## Upload All Note Assets to Cloudinary | ||
|
||
Within the command palette, run the "Upload all note files to Cloudinary" option. This will take all local assets located in the vault and attempt to upload them to Cloudinary. | ||
|
||
See below for information: | ||
[!ref Upload single note files](upload-single-note-cloudinary.md) | ||
|
||
Continue to [Backup Vault Assets](backup-vault-assets.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
--- | ||
label: Upload single note assets to Cloudinary | ||
layout: default | ||
order: 900 | ||
author: Jordan Handy | ||
icon: cloud | ||
--- | ||
## Upload Single Note Assets to Cloudinary | ||
|
||
Within the command palette, run the "Upload files in current note to Cloudinary" option. This will take all local assets located in the current note and attempt to upload them to Cloudinary. | ||
|
||
!!!warning | ||
It is **strongly** recommended to take a backup of your notes before you perform this action. | ||
As part of the action, your local media assets are **not deleted** when the upload to Cloudinary happens, so you can still reference them if you wanted to. However, because of the nature of the upload, and the variability of syntax in how some users may choose to reference media, this may alter the formatting of your notes. Use with caution. | ||
!!! | ||
|
||
The success / failure of this plugin largely depends on the following factors: | ||
1. Your Cloudinary plan -- Certain Cloudinary plans have quotas applied to their accounts. Consult Cloudinary documentation to understand what your limits are | ||
2. The file types being uploaded -- Cloudinary only accepts certain file types to be uploaded to their servers. If you try to upload an unsupported file type, the upload for that specific file will fail. Consult [this documentation on media types](https://support.cloudinary.com/hc/en-us/articles/202520642-What-type-of-image-video-and-audio-formats-do-you-support) and [this documentation on raw types](https://support.cloudinary.com/hc/en-us/articles/202520572-Using-Cloudinary-for-files-other-than-images-and-videos) for more information. | ||
|
||
## Warning | ||
When you first use the option for mass note upload, you will be presented with a warning message explaining the potential dangers of your action. By default, this warning message will be displayed **every time** you invoke the action. If you would like to disable this message, toggle the "Hide command palette mass upload warning" option in plugin settings. | ||
## Demo | ||
View a demo of the command below: | ||
![Demo video - mass upload](https://res.cloudinary.com/dakfccuv5/video/upload/v1718021709/mass-note-upload_qnx5ar.mp4) | ||
|
||
Continue to [Uploading all note assets to Cloudinary](upload-all-notes-cloudinary.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,189 @@ | ||
/*--------- Utilities for Commands and async uploads -----------*/ | ||
|
||
import { Notice, FileSystemAdapter, TFile } from 'obsidian'; | ||
import path from 'path'; | ||
import objectPath from 'object-path' | ||
import { NoteWarningModal } from '../note-warning-modal'; | ||
import { v2 as cloudinary } from 'cloudinary'; | ||
import { audioFormats, videoFormats, imageFormats } from '../formats'; | ||
import CloudinaryUploader from '../main'; | ||
|
||
// Generates a modal when command is invoked | ||
// References 'plugin' as we need access to settings on Cloudinary Uploader | ||
export function uploadNoteModal(file?: TFile, type: string, plugin: CloudinaryUploader): void { | ||
new NoteWarningModal(plugin.app, type, (result): void => { | ||
if (result == 'true') { | ||
if (file) { | ||
uploadCurrentNoteFiles(file, plugin); // If a file was passed and modal agreed | ||
return; | ||
} else { | ||
// If no file passed, but assets were to be uploaded | ||
if (type == 'asset') { | ||
fetchMessages(plugin); | ||
//uploadVault(plugin); // Upload vault function | ||
return; | ||
} else if (type == 'note') { //! If no file passed, but 'notes' to be uploaded, this means all notes are requested. | ||
const files = plugin.app.vault.getMarkdownFiles() | ||
for (let file of files) { | ||
uploadCurrentNoteFiles(file, plugin); | ||
} | ||
|
||
} | ||
} | ||
} else { | ||
return; | ||
} | ||
|
||
}).open(); | ||
} | ||
|
||
export async function uploadVault(plugin: CloudinaryUploader): Promise<string[][]> { | ||
let successMessages = []; | ||
let failureMessages = []; | ||
//* Get all files in vault that are not | ||
//* MD files, so they may be uploaded | ||
const files = plugin.app.vault.getFiles() | ||
new Notice("Upload of vault files started. Depending on your vault size, this could take a long while. Watch for error or success notices",0); | ||
for (let file of files) { | ||
if (file.extension != 'md') { | ||
let filePath; | ||
const adapter = plugin.app.vault.adapter; | ||
if (adapter instanceof FileSystemAdapter) { | ||
filePath = adapter.getFullPath(file.path); | ||
} | ||
await cloudinary.uploader.unsigned_upload(filePath, plugin.settings.uploadPreset, { | ||
folder: plugin.settings.preserveBackupFilePath ? path.join(plugin.settings.backupFolder, path.dirname(file.path)) : plugin.settings.backupFolder, | ||
resource_type: 'auto' | ||
}).then(res=>{ | ||
// add to success messages array | ||
successMessages.push('success') | ||
},err=>{ | ||
// add to failure messages array | ||
failureMessages.push(err.message); | ||
}); | ||
} | ||
} | ||
// send messages | ||
return [successMessages,failureMessages]; | ||
} | ||
|
||
//* This function fetches messages from the mass upload job | ||
//* as this could take a while if the vault is larger | ||
export async function fetchMessages(plugin:CloudinaryUploader) : Promise<void>{ | ||
// After the upload action completes then | ||
// retrieve data and display messages based on results. | ||
uploadVault(plugin).then((data)=>{ | ||
if(data[0].length > 0 && data[1].length > 0){ | ||
new Notice("Cloudinary vault asset backup: There was some success in uploading vault files, as well as some errors. Open Developer Tools for error information in Console",0); | ||
for(let msg of data[1]){ | ||
console.warn(msg); | ||
} | ||
}else if(data[0].length >0){ | ||
new Notice("Cloudinary vault asset backup: This was successfully completed. No errors to report",0); | ||
}else if(data[1].length > 0){ | ||
new Notice("Cloudinary vault asset backup: This operation failed. Please try again",0); | ||
} | ||
}); | ||
} | ||
export function uploadCurrentNoteFiles(file: TFile, plugin: CloudinaryUploader): void { | ||
//! Read a cached version of the file, then: | ||
/* | ||
* Find a RegEx match for [[]] file refs | ||
* Get file name and find full path of file | ||
* Pass full path to Cloudinary for upload | ||
* Based on answer returned, determine subfolder, then: | ||
* Replace current strings with Cloudinary URLs | ||
*/ | ||
let data = plugin.app.vault.cachedRead(file).then(() => { | ||
const found = data.match(/\!\[\[(?!https?:\/\/).*?\]\]/g); | ||
if (found && found.length > 0) for (let find of found) { | ||
let fileString = find.substring(3, find.length - 2); | ||
let filePath; | ||
const adapter = plugin.app.vault.adapter; | ||
if (adapter instanceof FileSystemAdapter) { | ||
filePath = adapter.getFullPath(fileString) | ||
cloudinary.uploader.unsigned_upload(filePath, plugin.settings.uploadPreset, { | ||
folder: setSubfolder(undefined, filePath, plugin), | ||
resource_type: 'auto' | ||
}).then(res => { | ||
console.log(res); | ||
let url = objectPath.get(res, 'secure_url'); | ||
let resType = objectPath.get(res, 'resource_type'); | ||
url = generateTransformParams(url, plugin); | ||
let replaceMarkdownText = generateResourceUrl(resType, url); | ||
data = data.replace(find, replaceMarkdownText); | ||
plugin.app.vault.process(file, () => { | ||
return data; | ||
}) | ||
new Notice("Upload of note file was completed"); // Success | ||
}, err => { | ||
// Failure | ||
new Notice("There was something wrong with your upload. Please try again. " + file.name + '. ' + err.message, 0); | ||
}) | ||
} | ||
} | ||
}); | ||
} | ||
// Called to generate the output of the transformation parameters | ||
// that are set on uploads | ||
export function generateTransformParams(url: string, plugin: CloudinaryUploader): string { | ||
if (plugin.settings.transformParams) { | ||
const splitURL = url.split("/upload/", 2); | ||
url = splitURL[0] += "/upload/" + plugin.settings.transformParams + "/" + splitURL[1]; | ||
} | ||
if (plugin.settings.f_auto) { | ||
const splitURL = url.split("/upload/", 2); | ||
url = splitURL[0] += "/upload/f_auto/" + splitURL[1]; | ||
// leave standard of no transformations added | ||
} | ||
return url; | ||
} | ||
// Once upload completes, generate the resulting getMarkdown | ||
// that will be written to file | ||
export function generateResourceUrl(type: string, url: string): string { | ||
if (type == 'audio' || isType(url, audioFormats)) { | ||
return `<audio src="${url}" controls></audio>\n`; | ||
} else if (type == 'video' || isType(url, videoFormats)) { | ||
return `<video src="${url}" controls></video>\n`; | ||
} else { | ||
return `![](${url})`; | ||
} | ||
} | ||
// Required as Cloudinary doesn't have an 'audio' resource type. | ||
// As we only know the file type after it's been uploaded (we don't know MIME type), | ||
// we check if audio was uploaded based on the most-commonly used audio formats | ||
export function isType(url: string, formats: string[]): boolean { | ||
let foundTypeMatch = false; | ||
for (let format of formats) { | ||
if (url.endsWith(format)) { | ||
foundTypeMatch = true; | ||
} | ||
} | ||
return foundTypeMatch; | ||
} | ||
// Determine the subfolder to place the uploaded | ||
// file in, based on file type uploaded | ||
export function setSubfolder(file: File, resourceUrl: string, plugin: CloudinaryUploader): string { | ||
if (file) { | ||
if (file.type && file.type.startsWith("image")) { | ||
return `${plugin.settings.folder}/${plugin.settings.imageSubfolder}`; | ||
} else if (file.type.startsWith("audio")) { | ||
return `${plugin.settings.folder}/${plugin.settings.audioSubfolder}`; | ||
} else if (file.type.startsWith("video")) { | ||
return `${plugin.settings.folder}/${plugin.settings.videoSubfolder}`; | ||
} else { | ||
return `${plugin.settings.folder}/${plugin.settings.rawSubfolder}`; | ||
} | ||
} else if (resourceUrl) { | ||
if (isType(resourceUrl, imageFormats)) { | ||
return `${plugin.settings.folder}/${plugin.settings.imageSubfolder}`; | ||
} else if (isType(resourceUrl, audioFormats)) { | ||
return `${plugin.settings.folder}/${plugin.settings.audioSubfolder}`; | ||
} else if (isType(resourceUrl, videoFormats)) { | ||
return `${plugin.settings.folder}/${plugin.settings.videoSubfolder}`; | ||
} else { | ||
return `${plugin.settings.folder}/${plugin.settings.rawSubfolder}`; | ||
} | ||
} | ||
|
||
} |
Oops, something went wrong.