Is there a way to reliably programmatically escape/ignore html in mdx?

[Stvad]

My situation is as follows:
I get input content as a set of "blocks" each of the blocks contains a flavored Markdown. A part of this special flavor is that html elements are not rendered. So those block can contain both valid and invalid html (I don't control content).

What I want to do is to assemble a set of blocks into a pages rendered by MDX. Which works well most of the time, but things fail parsing (and the build) when there is malformed html present in the source.

My question is - is there a reliable programmatic way to escape html within mdx so it will get parsed as a plain text. Or configure parser to ignore html elements in particular contexts somehow (by wrapping them in special element or something like that)?

https://maxschmitt.me/posts/escape-html-mdx/ is the only thing I found on the topic and it doesn't really provide a solution that works in all contexts (inline, multiline, and not disfiguring code inside code-fences)

Context: Building a static site with gatsby-plugin-mdx

[ChristianMurphy]

HTML and JSX are incompatible, mdx parses JSX.

There are a few options:

1. validate HTML tags to avoid invalid content
2. use https://github.com/remarkjs/remark#readme to parse HTML (mdx builds on remark, mdx replaces remark's HTML parser with a JSX parser)

[Stvad]

Background

I'd like to add a bit more of a background to hopefully explain why my requirements make sense (they do at least from the inside 😉)

I work on a project to make it easy to publish notes you've made inside https://roamresearch.com/ (can recommend checking it out if you haven't seen it before) as a static site (digital garden) - https://roam.garden/
So people originally write content in a different tool and mostly without concern for using it with my publishing service.
Then they export their content and import it to my service, at which point I try to build a static website from it.

This has the following implications:

• I can't validate content before the build time
• It'd be really bad customer experience for me to reject an input that is considered valid in the original system (or crash during the build which gatsby-plugin-mdx tends to do any time it catches a whiff of syntactially invalid mdx)

So my goals here are as follows:

• Never ever break the build
• For things that can't be properly fully parsed generate the best representation I can

How does data look and what I do with it

Fortunately the structure of the data I get is reasonably amenable to the second goal stated above. I get the notes in a JSON that represents a hierarchy of blocks, and each of the blocks is a semi-independent specially flavoured markdown document.
A simplified page example:

{
  "title": "test",
  "children": [
    {
      "string": "> I'm markdown",
      "children": [
        {
          "string": "*nested hierarchically*"
        }
      ]
    },
    {
      "string": "and I can have all kinds of weird things",
      "children": [
        {
          "string": "<%J:\n```javascript\nlet reg_parent_string = new RegExp('```clojure\\s(.+)```','s');\n```%>"
        }
      ]
    }
  ]
}

From this I'm going to generate an MDX document which would look something like the following:

# test

// there is no indents IRL as it interferes with parsing             
<Block content={
    <>

        > I'm markdown

    </>
}>
    <Block content={
        <>

            *nested hierarchically*

        </>
    }>

    </Block>
</Block>

<Block content={
    <>

        and I can have all kinds of weird things

    </>
}>
    <Block content={
        <>
            {/* An example of graceful fallback accomplished with double parsing strategy */}             
            <BlockContentFailedParsing content={`<%J:
\`\`\`javascript
let reg_parent_string = new RegExp('\`\`\`clojure\s(.+)\`\`\`','s');
\`\`\`%>`}/>

        </>
    }>
    </Block>
</Block>

See an example of the full export of the test page here: canary.txt and it in the rendered form https://vlad.roam.garden/?stackedPages=Garden-Lorem-Ipsum

As shown above - Blocks can have all kinds of weird things in them, but fortunately I can treat each block semi-independently and if there is something bad in a given block - I can gracefully and gradually degrade the rendering behavior just for that block while rendering other things on the page properly.
Which I currently do via double parsing approach discussed above.

The overall document is MDX but each individual block is Markdown (though I use MDX functionality like supplying custom components to override standard ones for them)

To achieve the stated graceful degradation goals - each given block can be treated as follows from the parsing standpoint (in the order of preference):

• Parse it as full MDX if possible
• If can't parse it as MDX because JSX parts seem invalid - escape anything that looks like JSX and try to parse it as Markdown (Original subject of this question, currently not done)
• If all attempts at parsing failed - display block as plain text (currently implemented with double parsing)

This degradation behavior is implemented at "code generation" time currently when I assemble the full MDX document that would later be parsed by gatsby-plugin-mdx

I hope this helps explain what and why I'm trying to do 🙂.

Potential alternative approaches

One thing I was wondering from your messages above whether it's possible to parse the internal content of the blocks as pure Markdown but then plug the results of that into the larger document/use it with MDXProvider somehow?
It sounds complicated and I'm not sure it fully possible or better then current approach 🤷‍♂️

[wooorm]

I don’t see MDX coming out of the API. Why are you trying to parse it as such?

[Stvad]

The overall document/page I generate is MDX as I'd like to use custom components and such and that is what finally gets transformed into a Gatsby node.

As noted above, each individual block is Markdown, but I want to use MDX functionality like supplying custom components to override standard ones for them.
So I'm using the functionality discussed in #628 to assemble MDX document with Markdown interleaved.

As I say in the "Potential alternative approaches" section - If I can assemble pipeline that would parse the Markdown blocks first independently (but allowing me to use the gatsby-remark-*/etc plugins) & then allow me to assemble it into the larger document & and use MDXProvider with all that - it can be an alternative. I'm not really sure how to do that and it seems like a more complicated approach from the outset vs the current one.
If you have any pointers in this direction - I'd love to explore them though!

[wooorm]

So I'm using the functionality discussed in #\628 to assemble MDX document with Markdown interleaved.

That issue is about authoring MDX. Here you’re asking about embedded markdown inside MDX, where the rules of MDX do not apply.

If I can assemble pipeline that would parse the Markdown blocks first independently […] & then allow me to assemble it into the larger document & and use MDXProvider with all that

☝️ Yep, this is what you’re looking for. You’ve found Y!

(but allowing me to use the gatsby-remark-*/etc plugins)

Can you elaborate what you mean by this?

I'm not really sure how to do that and it seems like a more complicated approach from the outset vs the current one.

It’s a bit more involved, but the problem with your current approach is that it can’t work.

[Stvad]

☝️ Yep, this is what you’re looking for. You’ve found Y!

Maybe, it feels more complicated than X though 😛, and X works. Y has potential to be more reliable though, so I'd really appreciate pointers on how one might go about setting up such a pipeline!

Can you elaborate what you mean by this?

Current final parsing & transformation is currently performed by gatsby-plugin-mdx and uses both raw remark-, rehype- plugins and gatsby-remark-* plugins (e.g. gatsby-remark-double-parenthesis-link or gatsby-remark-images). The hypothetical alternative mechanism should be able to do the same.

And then presumably allow me to assemble the final page in MDX (or raw JSX?) so I can use custom components and override the standard markdown components...

[EryouHao]

@Stvad Have you solved this problem, please? Do you have any solutions?

[wooorm]

There is no problem: html does not exist in mdx. JSX exists. But, the above question has a lengthy discussion. Your "is there a solution" does not add anything new to warrant a different answer.

[Stvad]

something like https://github.com/orgs/mdx-js/discussions/1477#discussioncomment-452585 is the thing I stuck with for my use-case. It's not ideal but kind-of works