SEO Compatible Rendering

[Archmonger]

Current Situation

Currently, sites built in IDOM are not SEO compatible. This is a fairly common issue with JavaScript frameworks such as ReactJS.

Prior discussion: #486
Prior Issue: reactive-python/reactpy-django#93

Proposed Actions

To resolve this, there needs to be an initial HTTP render, followed by a JavaScript re-render.

This will likely be accomplished through Preact's hydrate API.

The best way of doing this requires some form of persistent storage of all hook states, due to the fact that ASGI websockets are a completely different stack than HTTP rendering. Hook states will need to be stored server side in order to prevent spoofing. We can either use a database or determine if multiprocessing.shared_memory can be used here.

1. Use the template tag to render the initial component as raw HTML
2. Serialize all the component's hook values (probably through dill.pickle) and
   • This might need exposing some new hook APIs in core that provides returns hook values for a given component instance.
3. Store serialized hook values within the database
   • Use the component's UUID as the database ID.
4. When the JavaScript client requests it, rehydrate the components hook values
5. Execute the ReactJS rendering
   • Do Not Unmount Old View On Reconnect #760 will need to be resolve prior to this
6. Delete the component hook values from the database, as they are no longer needed.

The database model might look like this:

class IdomHookState(Model):
    uuid = models.UUIDField(
        primary_key=True, default=uuid.uuid4, editable=False, unique=True
    )
    hook_attributes = models.TextField()

This design brings up a challenge of determining when to evict old hook states (for example, if a user did the initial render but for some reason never performed a websocket connection). We don't want to store everything forever, so some configurable age-based eviction strategy might be needed. Expired entries should be culled before each fetch of IdomHookState.hook_attributes. Expiration should be based on last access time, which is fairly simple to do in Django like such: last_accessed = models.DateTimeField(auto_now=True).

[Archmonger]

This is an "experimental feature" and belongs in v2 -> v3

[Archmonger]

In order to take advantage of why users want SSR (namely SEO), IDOM might need to be modified to use HTTP instead of websockets.

[rmorshea]

So my thinking on this is that only the initial page needs the be rendered in HTML. Subsequent updates could be communicated via websockets. Then again, React has a way to do this, so if we can do server-side rendering with a traditional React solution and communicate via sockets on localhost then that could be an option too. The latter is a bit more complicated with respect to the tooling, but less work with respect to actual implementation, so I'd probably lean towards the latter assuming it's possible.

[Archmonger]

I did some investigation and here's what I came up with.

The Ugly

• The currently accepted "best methods" for SSR within React seems to rely frameworks such as NextJS, Razzle, or NodeJS.
  • These frameworks are used to render only the initial page server-side.
  • As you mentioned, that's the preferred behavior.
• If we want native SSR that relies on existing JS frameworks, we might end up relying on transpiling.
  • I can't think of a good way to propagate renders from IDOM Server Side -> Any JS Framework Server Side (transpiled?) -> React Client Side
  • Unless we can figure out a cross-platform way to make that work, I don't recommend going down this path.

Suggested Approach

• My suggested solution is to do this within Python.
• In theory, it's currently possible to immediately develop SSR for Django IDOM due to template tags.
  • We would need to render the component's initial HTML as a string within the template tag and then pass that into body of the component div.
  • mountViewToElement's behavior of wiping the div on WS connect will need to change.
    • Component Initial HTML reactpy-django#27 (comment)
  • Enabling or disabling SSR would configurable through settings.py:IDOM_SSR: bool = False
• There are a few things to consider with this design...
  • To avoid wasteful "double rendering" (SSR render + WS render), we need a way to signal mountViewToElement that the initial render has already been server-side rendered.
    • Likely just a ssr boolean parameter within mountViewToElement.
  • Rehydrating hook values from the SSR render into the WS render will be difficult and come with big limitations/hurdles.
    • We can't really store these values client side due to having to protect against value spoofing.
    • Will likely need to serialize and store the component's hook attributes within a database.
    • This limits SSR capabilities to web frameworks with integrated database capabilities.
    • This limits hooks' initial values to those serializable by dill.pickle.
    • On web frameworks without databases, we can fallback to "double rendering" the component.
    • Alternatively to database storage, we can use redis. But I'm personally not a huge fan of IDOM forcing redis usage.
• This template tag system should be replicated to all other frameworks we support.
  • Embedding ReactPy via Template Tags #653

---

Resources

• https://blog.logrocket.com/why-you-should-render-react-on-the-server-side-a50507163b79/
• https://asperbrothers.com/blog/server-side-rendering-in-react/#:~:text=What%20is%20Server%2DSide%20Rendering,with%20all%20the%20static%20elements.
• https://www.freecodecamp.org/news/server-side-rendering-your-react-app-in-three-simple-steps-7a82b95db82e/

[Archmonger]

@rmorshea I've discovered react actually has an API for this.

https://beta.reactjs.org/reference/react-dom/client/hydrateRoot

[rmorshea]

We're technically using Preact right now, so we'd probably what we'd want to pay attention to its specific hydration details.

It would be interesting to see if there's anything special about pre-rendering. If not, then we could potentially just use vdom_to_html. If so, we might need a build step where we're require the user to have NodeJS installed so we can use preact build.

[Archmonger]

I don't think we should use that pre-rendering API.

If we do, then

1. We are making the assumption that IDOM knows where HTML static pages should be stored.
   • This would mandate users to define a HTML output directory via Options
2. Have users perform a build step
3. Would seems like like we'd be encroaching features that our backends do.

We may want to stare at the pre-rendering source and see if anything unique is being done, which we could implement as mutations. I doubt it though, so vdom_to_html seems like the right choice.

[Archmonger]

I've been doing some thinking on this. I think we might need to take a "zero rehydration" policy and always double render (HTTP followed by WS render).

Since the HTTP stack and websocket stack aren't always on the same process/thread, we can't store any data in-memory. Additionally, not all backends have have a built-in database framework... and we really don't want to build any database features directly into ReactPy.

So, to keep things consistent it's probably best to do the following:

• Template tag performs the Initial HTTP render using html_to_vdom
  • Exist solely to pre-populate the page.
  • This initial render is completely non-interactive, and the page will remain non-interactive until the re-rendering step below.
• ASAP re-render using websockets.
  • Developers will need to be mindful of a few details
    • There may be differences between the two renders, since hook values aren't stored + rehydrated.
    • use_scope/use_connection hooks will utilize a HTTP Carrier during this initial render.

If we really want to, we can enable rehydration for batteries-included frameworks such as Django. But in my opinion, having a the functional difference between the rehydration/non-rehydration between frameworks isn't a good idea. Even more so when considering that it would introduce more maintenance burdens.