loadProjectFromDirectory file sync, import & export sync

[samuelstroschein]

Context

@samuel.stroschein said in MESDK-176:

Update: saveProjectToDirectory is not needed until we need roundtrip capabilities from dev tools <-> web apps. As long as the dev tools are single player, loadProjectFromDirectory can set up file watching and auto in- and export.

cc @felix.haeberle

Task

• add file syncing between host <-> lix
• add watching for changes in toBeImportedFiles()
• add auto import/export

[samuelstroschein]

@martin.lysk1 can you take this over?

progress is in branch https://github.com/opral/monorepo/tree/mesdk-200-loadprojectfromdirectory

Understanding File Syncing in Inline SDK

[martin-lysk]

sure i am on it

[martin-lysk]

Quick question about syncing from the inlang lix to the host:
Process is like this:

1. create a project in memory
2. import everything from the inlang folder passed in start watching - on change on
3. on change on files in lix, copy the files to the host.

Regarding 3. some followup questions.

How shall we distinguish between files we want to sync to the host (like settings) and those we don't want to pass to the host, like the inlang.db i guess.

Option 1 - hardcode a block list and just skip the inlang.db
Option 2 - keep track of files imported via copyFilesFromDiskRecursive and sync only those that we imported to the host again.

Lets go for Option 1 @samuel.stroschein?

How shall we notice a change in a file

Option 1 - should we add a callback to the lix api - on file change, create a trigger in lix?Option 2 - should we create a trigger using the lix db object in inlang

I prefere Option 1 but 2 is faster

Both will break with https://linear.app/opral/issue/LIXDK-102/re-visit-simplifying-the-change-queue-implementation#comment-65eb3485

[samuelstroschein]

How shall we distinguish between files we want to sync to the host (like settings) and those we don't want to pass to the host, like the inlang.db i guess.

Yes, hardcode ignored files.

How shall we notice a change in a file

You can use pollQuery which only emits an event if the underlying query result changes.

(pollQuery is simulating an observable for sqlite. Granted, not real time but good enough. Lix won't need callbacks and manually setting up triggers is also not needed)

[samuelstroschein]

@martin.lysk1 I completed MESDK-176. During the implementation the following observations emerged that you should consider for this issue:

1. Don't forget to use absolutePath and withAbsolutePaths when you interact with fs stuff that plugins use. See Fix cwd bug with plugin imports by making paths absolute #174
2. the saveProjectToDirectory tests might get flaky if loadProjectFromDirectory establishes watchers. To mitigate the problem I propose to add a watch: boolean flag to loadProjectFromDirectory that turns the sync on or off.

EDIT: yeah auto watching in loadProjectFromDirectory will likely lead to unexpected behavior see MESDK-196

[samuelstroschein]

After a sync with @martin.lysk1 we concluded:

• using lix LIXDK-126 to solve the sync issues makes sense
• because branching is WIP, martin will work on LIXDK-147 and LIXDK-158 in the meantime

cc @felix.haeberle the sync to the fs will be delayed until LIXDK-126 exists. i deem this to be OK given that sherlock doesn't need to change when this feature ships.

[martin-lysk]

Quick update from my side:

Blocked by MESDK-186 compare https://discord.com/channels/897438559458430986/1287684940418514964/1287700196381167656

[samuelstroschein]

marking as on hold because martin is focusing on a csv hosted demo for the onsite