make changes for import/export module to work properly #4270

ETLaurent · 2023-08-15T09:55:55Z

PRO-4271

Summary

Brings some changes in core that are necessary for the new Import/Export module.
Must be tested with this PR.

Allows to insert attachment with a given ID (to import attachments with same IDs). As well as with docIds and archiveDocIds to preserve related docs.
Adds an update method to the attachment module in order to allow a user that choose to override a piece with attachment to delete the existing one and reinsert it (the document in DB and the associated file). It deletes and recreates to reuse existing code and avoid big changes here, since the insert method does a lot of things.
Adds an option to the http remote method to allow receiving the original response from node-fetch, it allows to manipulate the response as a stream before it has been consumed in the method by res.text() we need it to download images through streams and push them in the compressed file.

What are the specific steps to test this change?

See steps in this PR.

What kind of change does this PR introduce?

Make sure the PR fulfills these requirements:

It includes a) the existing issue ID being resolved, b) a convincing reason for adding this feature, or c) a clear description of the bug it resolves
The changelog is updated
Related documentation has been updated
Related tests have been updated

linear · 2023-08-15T09:55:57Z

PRO-4261 As an editor, I can download a ZIP file of my exported documents

When clicking on the download button of the export modal, after I selected the related docs that I want to include in the export, the process must be triggered.

image.png

Two new routes export must exist in the modules piece-type and page. Both routes will call a method export from the module doc-type .

This method should take the IDs of the documents the user want to export, as well as the related types that he wants to get. Only the types, not the IDs, since we decided that it would be too heavy for the export modal to get the amount of related docs when we batch a large amount of documents.

Generate files

This method will need to use the relate method from the module schema, this method fills documents with related ones.

Since we need to store documents in a flat EJSON array, it might be a good idea to add an option to this method in order to get a flat array of related documents instead of filling requested documents with their related docs, see how it could potentially be used with query builders.

We want one EJSON file for each db collection, for example aposDocs.json and aposAttachment.json, containing arrays of documents..

We want, for each document to export the draft and live versions (only draft if there is no live). The draft version must appear first to avoid situation where a live exists without draft which is forbidden.

Same thing for related documents, they must be written first in the EJSON file, otherwise the doc manager will yell when we'll import it.

We will create an attachment folder when actual attachments will be stored (images, files…).

Generate Zip

To generate the ZIP file we would prefer using the native node library.

Progress bar notification

This progress bar notification is triggered by wrapping your export function with the run method of the job module. Here is an example from the piece-type-importer module:

return self.apos.modules['apostrophecms/job'].run(
  req,
  (req, reporting, { notificationId }) => self.importRun(req, {
    progressNotifId: notificationId,
    filePath: file.path,
    reporting,
    totalPieces
  }),
  {}
);

The reporting object passed from the second argument function allows to set the total of the documents you want to import with the method setTotal, as well as to call the methods success or failure for every document that we tried to export or import.

Finally the ZIP file will be sent back to the client and the download triggered from the user's browser.

Permissions

For exporting documents, a user that has at least one per-type permission should be able to export anything. He won’t be able to see adminOnly documents, therefore should not be able to export these documents. It matches the way it works currently because in this case the user can see all drafts. In core, admin should be able to export adminOnly content, except for users.

For Reference

https://github.com/apostrophecms/tech-designs/blob/main/3.x/share-documents-across-sites/design.md

Acceptance Criteria

I'm an editor or admin on with the Download modal open, and my documents selected, and I hit "Download" and when I do, a ZIP file begins downloading.
The zip file downloads and I open it and I see the the draft and live versions (only draft if there is no live) in my unarchived folder.

… as an event in notif

modules/@apostrophecms/job/index.js

…ckend

modules/@apostrophecms/user/index.js

…ed attachment

ValJed · 2023-10-03T08:46:32Z

modules/@apostrophecms/attachment/index.js

@@ -432,7 +437,11 @@ module.exports = {
        }
        if (self.isSized(extension)) {
          // For images we correct automatically for common file extension mistakes
-          const result = await Promise.promisify(self.uploadfs.copyImageIn)(file.path, '/attachments/' + info._id + '-' + info.name, { sizes: self.imageSizes });
+          const result = await Promise.promisify(self.uploadfs.copyImageIn)(


only indentation.

boutell

Small things hopefully

boutell · 2023-10-06T12:09:36Z

modules/@apostrophecms/attachment/index.js

@@ -402,16 +401,21 @@ module.exports = {
            extensions: accepted.join(req.t('apostrophe:listJoiner'))
          }));
        }
+
+        if (options.attachmentId && await self.apos.attachment.db.findOne({ _id: options.attachmentId })) {
+          throw self.apos.error('invalid', 'duplicate');


This could hit a race condition and imposes an unnecessary delay, try/catch the insert and check for self.apos.doc.isUniqueError(err) to see if mongodb threw a unique key error or something else.

As discussed I kept the behavior as it is for now, we might create a ticket to update slightly the insert method in order to insert the doc before to copy to file to uploadfs.
At least here if a user use the option attachmentId (I don't see a lot of use case) they are protected from a mess (file created while the document has not been inserted).

boutell · 2023-10-06T12:13:49Z

modules/@apostrophecms/attachment/index.js

+        await self.db.deleteOne({ _id: existing._id });
+        await self.insert(req, file, {
+          attachmentId: attachment._id,
+          docIds: attachment.docIds,


Merge the old and new docIds and archivedDocIds so you have a complete list. Conversely, use $in queries with an { _id: 1 } projection to confirm they all actually exist on the receiving site and filter out those that don't. (I assume you're doing all of the attachments last after the docs so that this would give a complete result.)

I know we are pressed for time. There is also a much simpler alternative re: tracking docIds and archivedDocIds. Just use the recomputeAllDocReferences task programmatically at the end of the entire import process. It takes a little time, but and it will definitely get the right result without any tricky bookkeeping. It is 100% OK to do that for this first release at least, if you find it difficult to get the lists right per the technique above.

You can invoke a task internally but I'd just refactor its logic to a method for convenience (with no other changes).

Thanks for having spotted that. I now only fill these arrays with existing docIds and archivedDocIds and the new ones that exist in the current site based on if the document is archived or not.

boutell · 2023-10-06T12:21:41Z

modules/@apostrophecms/image/ui/apos/components/AposMediaManager.vue

@@ -56,6 +56,7 @@
            :labels="moduleLabels"
            :disable="relationshipErrors === 'min'"
            :displayed-items="items.length"
+            :checked="checked"


Huh, was this just an unrelated bug?

Not a bug, it has been added to be consistent with the other vue changes.
Basically the modal for exporting documents needed the checked IDs so we pass it to AposDocsManagerToolbar when used.

…ify they exist

boutell

You also need to merge in the docIds and archivedDocIds from existing because the attachment could be present in other documents on the receiving site.

ETLaurent · 2023-10-09T13:16:45Z

CHANGELOG.md

+### Add
+
+* Allows to insert attachments with a given ID, as well as with `docIds` and `atchiveDocIds` to preserve related docs.
+* Adds an `update` method to the attachment module, that update the mongoDB doc and the associated file.


Suggested change

* Adds an `update` method to the attachment module, that update the mongoDB doc and the associated file.

* Adds an `update` method to the attachment module, that updates the mongoDB doc and the associated file.

ETLaurent · 2023-10-09T13:17:01Z

CHANGELOG.md

+
+### Add
+
+* Allows to insert attachments with a given ID, as well as with `docIds` and `atchiveDocIds` to preserve related docs.


Suggested change

* Allows to insert attachments with a given ID, as well as with `docIds` and `atchiveDocIds` to preserve related docs.

* Allows to insert attachments with a given ID, as well as with `docIds` and `archiveDocIds` to preserve related docs.

… docs

ETLaurent · 2023-10-12T10:05:42Z

modules/@apostrophecms/attachment/index.js

+                ...existing.docIds,
+                ...existing.archivedDocIds,
+                ...attachment.docIds,
+                ...attachment.archivedDocIds


@boutell is this what you meant?

ETLaurent mentioned this pull request Aug 15, 2023

export docs and related docs apostrophecms/import-export#7

Merged

10 tasks

pass checked document ids to toolbar

30ab5da

ValJed force-pushed the pro-4261-export-docs branch from d5171df to 30ab5da Compare August 22, 2023 12:38

ETLaurent added 3 commits August 23, 2023 10:35

use dismissProgressNotification

5cbec97

pass notification options and sent results as an event in notif

c714926

remove notification options and use req.body.messages to sent results…

3b06f5e

… as an event in notif

ETLaurent commented Aug 23, 2023

View reviewed changes

modules/@apostrophecms/job/index.js Show resolved Hide resolved

ValJed and others added 2 commits August 28, 2023 15:54

adds the option originalResponse to the http remote method for the ba…

7c94ea6

…ckend

disable export for users

ccb5c18

ETLaurent commented Aug 29, 2023

View reviewed changes

modules/@apostrophecms/user/index.js Outdated Show resolved Hide resolved

ETLaurent changed the title ~~pass checked document ids to toolbar~~ make changes for import/export module to work properly Aug 29, 2023

ETLaurent and others added 8 commits August 30, 2023 16:41

importExport

eaf5179

adds new color variable

b376997

allows to pass existing ID to attachment insert method

fc15e25

when passing attachmentId check if this one exists already

bea905a

adds update method

f1eb181

returns promise in order to wait if needed

a560591

starts notification with percentage of 0 to avoid NaN

5346f26

removes an useless return satement

e9f7a0c

ValJed mentioned this pull request Sep 26, 2023

import docs from archive and handle reporting apostrophecms/import-export#21

Merged

ValJed requested review from haroun and removed request for haroun September 26, 2023 16:26

ValJed and others added 2 commits September 28, 2023 11:40

in attachment update method, make sure to delete the existing retriev…

66989c4

…ed attachment

Merge remote-tracking branch 'origin/main' into pro-4261-export-docs

2b62ecf

ValJed reviewed Oct 3, 2023

View reviewed changes

fixes lint

6d88220

ValJed requested a review from boutell October 3, 2023 12:42

ValJed marked this pull request as ready for review October 3, 2023 12:42

adds changelog

26df2d0

minor changes

b1b72c2

boutell requested changes Oct 6, 2023

View reviewed changes

ValJed added 2 commits October 6, 2023 17:40

compute new docIds and archivedDocIds when updating attachment to ver…

2c6f00a

…ify they exist

moves import export related options to the module itself

734e292

ValJed requested a review from boutell October 6, 2023 15:47

only gets existing related docs from IDs

90f9fb3

boutell requested changes Oct 6, 2023

View reviewed changes

ETLaurent commented Oct 9, 2023

View reviewed changes

ETLaurent added 4 commits October 11, 2023 11:52

Merge remote-tracking branch 'origin/main' into pro-4261-export-docs

92b8054

typos

526c519

typo

8af5ac0

use docIds and archivedDocIds from existing attachment to get related…

4ba007d

… docs

ETLaurent requested a review from boutell October 12, 2023 10:05

ETLaurent commented Oct 12, 2023

View reviewed changes

boutell approved these changes Oct 12, 2023

View reviewed changes

ETLaurent merged commit 9ccb6be into main Oct 12, 2023
8 checks passed

ETLaurent deleted the pro-4261-export-docs branch October 12, 2023 12:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make changes for import/export module to work properly #4270

make changes for import/export module to work properly #4270

ETLaurent commented Aug 15, 2023 •

edited by ValJed

Loading

linear bot commented Aug 15, 2023

Generate files

Generate Zip

Progress bar notification

Permissions

ValJed Oct 3, 2023

boutell left a comment

boutell Oct 6, 2023

ValJed Oct 6, 2023

boutell Oct 6, 2023

boutell Oct 6, 2023

ValJed Oct 6, 2023 •

edited

Loading

boutell Oct 6, 2023

ValJed Oct 6, 2023

boutell left a comment

ETLaurent Oct 9, 2023

ETLaurent Oct 9, 2023

ETLaurent Oct 12, 2023

boutell Oct 12, 2023

	* Adds an `update` method to the attachment module, that update the mongoDB doc and the associated file.
	* Adds an `update` method to the attachment module, that updates the mongoDB doc and the associated file.


		### Add

		* Allows to insert attachments with a given ID, as well as with `docIds` and `atchiveDocIds` to preserve related docs.

make changes for import/export module to work properly #4270

make changes for import/export module to work properly #4270

Conversation

ETLaurent commented Aug 15, 2023 • edited by ValJed Loading

Summary

What are the specific steps to test this change?

What kind of change does this PR introduce?

Make sure the PR fulfills these requirements:

linear bot commented Aug 15, 2023

Generate files

Generate Zip

Progress bar notification

Permissions

Choose a reason for hiding this comment

boutell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValJed Oct 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boutell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ETLaurent commented Aug 15, 2023 •

edited by ValJed

Loading

ValJed Oct 6, 2023 •

edited

Loading