import docs from archive and handle reporting

[ETLaurent]

PRO-4271

What this PR does

• read from extracted export file
• import docs and attachments documents from it, using managers for docs, and using the insert method in attachments in order to create the attachments document in DB as well as uploading the associated file.
• For autopublish documents we only insert the draft version of the document and treat the published one the same way (inserted / duplicate / failed).
• send duplicated docs and attachments to the browser via the import-duplicates even triggered by a success notification that opens the modal and passes right data.
• Manages job and notification internally to have maximum control on it, the job and notification progress is shared between the first import process and the duplicate resolution one (if there is one).
• Adds event listeners to block user from leaving the page during the import process, as well as to call the modal to manage duplicates if there are duplicates docs.

selecting export file	importing	duplicated docs are sent to the browser

Run following mongo command to remove docs and re-import them, in order to test an import with no duplicates:

testbed> db.aposDocs.deleteMany({ type: /article|topic|data/ });

reporting:

Testing PR

Should be test with the branch of this PR in core.

Perform imports with unexisting documents with and without related documents it should import them properly.
Perform imports with existing documents with and without relatated documents it should update them properly as well as their related attachments.

[linear]

PRO-4271 As an editor, I can import Apostrophe documents in my site

As an editor, I am able to import documents, related ones and related attachments to another site. These are contained in a ZIP file generated from a documents export in another site (or the current one).

The ZIP file itself must be imported as it is to be sure everything is correctly imported and its content unchanged.

When the import button from the import modal is triggered, the zip file must be sent to the server through a single POST request for this iteration (so will be limited in size).

Read from ZIP file

The backend will unzip the file using the same lib used to generate it, We would prefer using the node native lib.

You might look a way to stream the content of the ZIP file to avoid having to load everything in memory before to read it. It should be ok now since we don't handle big loads of data, but in the future we will.

Attachments

It must import attachments and documents from aposAttachments first. There is a problem here, we want to keep the IDs of the aposAttachments documents to avoid having to rewrite apoDocs relationships.

The insert method of apostrophecms/attachments writes the file and create the associated aposAttachments document,

you might need to make some modifications here to be able to write a file and to insert an existing attachment, you need at least, to keep the ID.

If the attachment already exist (by ID) it must not be created, and the file not uploaded. We'll consider here that it has been imported in the past.

Documents

Then we need to import aposDocs documents, they must be imported following the order provided during the export. We will need to keep track of imported documents IDs. this way, if two documents have a related one in common (or a related one is imported too) it won't be imported twice or, at least, treated like a duplicated one.

We will also need to keep track of duplicated documents IDs, to ask later to the user if we must override them.

You must use the proper manager for each piece depending its type, and to use the insert method of the concerned module. If the document already exists we must stack it somewhere as explained above, it might work since data load won't be huge for now. An other solution would be to store the duplicated documents into the job directly but would involve more work and doesn't seem necessary right now.

the import process must be wrapped by the run method of the job module, it'll allow to manage the notification progress bar.

See this ticket for more information about the notification progress bar implementation.

Singletons

If the piece is from a module that has the option singleton: true (there is only one piece allowed eg. apostrophecms/global, apostrophecms-pro/palette) it overrides by default the existing one, without asking the user.

Pages

For this iteration we won't manage the download of all the exported page children. We still want to check the location of the page in the tree, if this one exists then we want to import the page at the same location (at least for the parent).

If the path doesn't exists, simply put the page as a Home page child.

Permissions

Since we will use the official insert/update Apostrophe methods, if they fail due to permissions the import should stop with an appropriate error.

Acceptance Criteria

• When clicking on the import button of the modal the process starts and I can see the notification progress bar.
• Attachments and documents are successfully imported with the same IDs and relations keep working.
• Duplicated documents must not be imported but kept in memory for the last step (asking the user for override).
• Pieces that are singletons automatically override the existing one (if there is one) during import.
• For pages, if the tree location exists we want to keep it, otherwise to locate the page as a home page child.