lix host prototype spec

[samuelstroschein]

Context

The workflows between sync and file-based collab shouldn't differ.

Yet, we had discussions on data structures and workflows that differ in a hosted lix scenario. It would likely help to drive consensus and confidence in the data structures if a mini host prototype exists that showcases workflows in a synced scenario.

Proposal

Build a 1 day prototype that enables fink and the csv protoype to sync to a host.

• No UI, no auth, just a mini prototype.

[samuelstroschein]

Example loading a lix from the host

• file can be temp stored on the server/even in memory
• sending the lix binary around is good enough

// fetching the initial file from the host
const response = await fetch("https://lix-host.onrender.com/project.inlang")

// load the project 
const project = await loadProjectInMemory({ blob: await response.blob() })

// utility function that takes the lix and syncs it to the given host
sync({ lix: project.lix, host: "https://lix-host.onrender.com/project.inlang" })

Pseudocode sync implementation on the client

• receives events from host on changes that send the lix binary that is used to merge on the client
• sends lix file on changes to server

export async function sync(args: { lix: Lix, host: string }){
  // 1. server -> client change handling via merging from server to client
  longPollHostForChanges(args)
  // 2. client -> server change handling via merging from client to server

}

async function longPollHostForChanges(args){
  // receiving an update from the server (which sends the new binary lix file)
  const response = await fetch(args.host + "?long-poll")
  const sourceLix = openLixInMemory({ blob: await reponse.blob() })
  
  // merging into target (the client's lix) 
  await merge({ target: args.lix, source: sourceLix })

  // repeat
  await longPollHostForChanges(args)
}

async function sendFileOnChangeToHost(args) {
  // set up a manual trigger for every table fo interest
  setupChangeTriggers(["conflict", "change", "file"], () => {
    await args.lix.settled()
    await fetch(args.host + "?long-poll", {
      method: "POST",
      body: await args.lix.toBlob()
    })
  })
}

Things needed to make this work

• merging from A to B must be the same as B to A

@jan.johannes @martin.lysk1 am i missing something?

[NilsJacobsen]

Why can the merging not happen on a client and we use a s3 to just override data?

[samuelstroschein]

Guys, this is not about live collaboration. This is just syncing. No need to show who opened a file and what not in this test. Just send the server the lix file on changes (which the server merges) and merge receiving lix files on the client.

• the server does not need to generate file names. this can be hardcoded in this prototype.
• i don't see a need for server generated, user id and what not
• no need to see who opens a file

We just need a sync button which, if checked, sync to the server.

@nils.jacobsen Why can the merging not happen on a client and we use a s3 to just override data?

Because you overwrite data and thereby lose merging on the server.

The client merges data from the server. The server merges data from the client. That simple. Two clients syncing with a server will always merge on the server. Otherwise, Client A and Client B need to connect to each other to merge.

@jan.johannes the goal is to replace the file sending with the same behaviour with a server / host.

Yes. Have syncing that avoids sending files around.

[samuelstroschein]

BTW this is blocked by LIXDK-126. We should probably discuss and introduce branching first

[samuelstroschein]

Yes exactly!

1. A user connects to the host (that already has the file)
2. Client establishes long polling to server to receive updates
3. Client sends lix file to server on changes

With those three steps, we should be able to have syncing between multiple clients. Conflicts don't have to be resolved. If a conflict occurs, the conflict will be shown to the user. Our current architecture of lix should support this use case out of the box. If not, something went wrong and we have to investigate.