Docs: Improve MDX new user experience

[karlhorky]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Problem

Let me start with saying MDX and the experience it enables (when properly set up + configured) is great! The simplicity and ease of Markdown with the power, flexibility and interop of React 😍

However, there are not-uncommon cases where users can fall off the happy path / miss the pit of success if going beyond the simple first examples - there's a long tail of complexity and high chance for breakage in moving beyond simple examples.

Let's take <video> support an MDX document in Next.js as an example:

1. User tries to add <video src="abc.mp4"> to an MDX document, knowing already that they can add HTML to MDX - it works! 👍
2. User would like to import the .mdx file in another location. "abc.mp4" cannot be resolved anymore.
3. User researches, finds rehype-mdx-import-media, using cmd-F, finds that it supports video[src] and tries configuring using the example config
4. Nothing seems to be happening with the <video>, user researches quickly and finds no issues or problems with the <video> syntax. (at this point, knowledgeable, experienced MDX users would have found the bullets under Usage, but during my user journey, I did not understand this on the first try). User instead tries out an image in Markdown syntax and it appears to be working.
5. User reflects and considers that there may be a problem with <video> and rehype-mdx-import-media, but since there are no issues directly related to this plugin, researches in related ecosystems rehype and MDX. User does not find something directly related to lack of <video> support, but eventually finds that there may be behavior of MDX which causes it to treat HTML elements differently than Markdown elements.
6. User researches how to add <video> in MDX syntax and finds the ![](abc.mp4) syntax (which, to be fair, contains the caveat "Depending on your markdown processor,"). This doesn't work out of the box.
7. User researches how to add ![](abc.mp4) syntax support and happens on to @chailotl/remark-videos, which does help - the error message has now changed to Module parse failed: Unexpected character '' (to be fair, this is entering Next.js + webpack territory now)
8. User researches what this could be, thinking this is also related to MDX/remark/rehype/etc but fails. User changes research techniques and finds that it may be related to Next.js and webpack loaders. After some time, finds a webpack loader which should be able to load .mp4 files.
9. User observes working video 👍

Another example would be related to a user trying to include nested MDX content on part of a Next.js page (not a full page with app/page.mdx) - this could lead to A) next-mdx-remote (not fully working with nested MDX), B) mdx-bundler (working with nested MDX but complicated) and then C) back to @next/mdx (which does work, but it is undocumented how to do it - await import(...)).

In my personal evaluation, after years on and off in the MDX ecosystem (having also invested more time and energy than I would say is average for a part-time user): this is not an uncommon experience when working with MDX - as soon as needs exceed the basics a bit, I've seen it be common to have a "rabbit hole"-type debugging experience, through complexity and incompatibilities on various levels.

In my personal evaluation, some of the first documentation problems I perceive relate to new user experience are:

A) Happy Path / Pit of Success

Lack of working, blessed, happy-path setup and integration suggestions for use cases beyond simple content

B) Transitive Dependency Web / Many Moving Parts

The complex web of transitive dependencies that extends from MDX to remark/rehype/recma to other unified packages like the unist-* packages makes it challenging for a senior engineer to understand, navigate and fix / work around incompatibilities - eg. type errors or runtime incompatibilities with remark/rehype/recma plugins and transitive dependencies

Many of these issues may be perceived by experienced developers or those working in the MDX and related ecosystems as "papercuts" or low-barrier issues, but for beginners, it can lead to hitting walls and even quitting. Even for me as an experienced developer, I often find myself spending a lot of time on something that appears to be a simple problem.

---

I don't intend this to be a condemnation or negative criticism of the ecosystem or packages or people involved - just trying to surface a report of the UX pitfalls for an (enthusiastic) user in the ecosystem.

Solution

A) Happy Path / Pit of Success

1. An ecosystem for modern, updated, maintained recipes / cheatsheets / examples - eg. similar to the Next.js examples - to allow both MDX maintainers and the community to contribute helpful, common configurations, setups and integrations (for the first example above, a "Next.js + MDX + imported video" recipe/example would be not much code and would save a lot of time)
2. Potentially also: Modern, updated, maintained end to end / integration tests with latest versions of integrating packages

B) Transitive Dependency Web / Many Moving Parts

1. Create docs content dedicated to common interop problems with transitive dependencies and solutions for these and rank highly on search engines with this content
   1. Potentially also: docs pages for other common papercuts
2. Additional combined UIs for the API docs for MDX, remark/rehype/recma, unified packages, eg:
   1. Allowing for seamless navigation among the docs "owners" without having to jump between multiple layers or websites (even just including the types of other libraries without having to click on links would be a great start)
   2. Search Add site search to the new MDX website #1776

---

If this issue or or parts of this issue are accepted, then I'm ok with this becoming the parent "umbrella issue" to track multiple initiatives.

Alternatives

Create additional tooling to avoid the issue, such as:

• CLI tools like mdx doctor which will find common issues like mismatching or duplicated transitive dependencies and other setup problems
• Additional editor-level tools to navigate and resolve issues
• Additional tools for integrations such as Next.js, to either offer features to smooth over the experience or to raise issues that users should resolve
• Additional chat-based UI for the docs for MDX, remark/rehype/recma, Unified packages, to allow for easier, faster navigation and search, when users don't use the exact wording or terms in the docs

But I think that these alternatives are complex and time-consuming to build and maintain, would not be able to effectively address all issues and would also require docs and outreach to make sure that users actually used them.

[ChristianMurphy]

Hey @karlhorky! 👋
Thanks for taking the time to share you experience and some ideas to improve the experience.
There is always interest in improving developer experience.

A few themes I'm hearing from you (correct me if I repeat these back inaccurately)

1. You were/are looking for a fairly custom solution. Not using an existing video player react component or video transcoding platform
2. You found various community plugins in varying degree of support/documentation/repair
3. The references/examples you used mainly came from next js

A few themes I'll share from my perspective before responding more in depth

1. Inherent and irreducible complexity. All problems have a degree of complexity which can't be reduced or abstracted away. Parsers/Compilers has a significant amount of theory and complexity surrounding them.
2. MDX is Stadium project. Unified/Micromark/Remark/MDX have millions of active developer users, and 3 active maintainers currently. (see more on project types here: https://blas.com/working-in-public/)
3. Incorrect documentation is worse than no documentation. Document needs to: work, reflect the latest version of dependencies, and reflect best practices. If it does not do all of these, it is a worse experience for the developer than trying to read the source itself.
4. Infinite solution space. MDX is a programming language with a content focus. It can do anything, people use it in wildly varied and creative ways from document editing, interactive experience creation, learning management system foundations, and much much more. The number of hypothetical uses are infinite, and even in practice have been growing exponentially since the project's inception.

User would like to import the .mdx file in another location. "abc.mp4" cannot be resolved anymore

Hindsight being 20/20 this feels like it could/should have been made an absolute path instead of a relative path so it would resolve the same anywhere?

Nothing seems to be happening with the

Not judging, just observing.
It is fairly common both in the issue tracker and in discussions for adopters to try stuff without reading the readme. Not understand the abstraction or what the plugin does. Then open an issue/discussion and suggest: more documentation.

More documentation won't help people that don't read documentation in the first place. 😅

as soon as needs exceed the basics a bit, I've seen it be common to have a "rabbit hole"-type debugging experience, through complexity

As you move from the higher level abstraction to lower level ones for more advanced use cases it does get more complex.
We offer some introductory materials to the concepts in https://unifiedjs.com/learn/
It really could be it's own book, there was even a start on one once upon a time https://github.com/unifiedjs/handbook but it was never finished.

and incompatibilities on various levels.

Could you expand what you mean by this?

The complex web of transitive dependencies that extends from MDX to remark/rehype/recma to other unified packages like the unist-* packages makes it challenging for a senior engineer to understand, navigate and fix / work around incompatibilities - eg. type errors or runtime incompatibilities with remark/rehype/recma plugins and transitive dependencies

Unified has a large API surface.
The types are the most effective way to see if things are compatible.
If they aren't issues are welcome!

If you need help typing your own custom plugins, we have recipes at https://unifiedjs.com/learn/ which offer some best practices for working with the typings, which maintain strong type guarantees.

Lack of working, blessed, happy-path setup and integration suggestions for use cases beyond simple content

If you have concrete suggestions on common use cases you see beyond content ideas are welcome!
The tricky part here isn't finding other uses (see infinite solution space above) but boiling that down to use cases many adopters have.

An ecosystem for modern, updated, maintained recipes

We have this! https://unifiedjs.com/learn/
And more at https://mdxjs.com/

examples - eg. similar to the Next.js examples

I land pretty strongly against example folders like this, they aren't maintained and tested the way they should be, and that can cause more confusion that it can help (see Incorrect documentation is worse than no documentation above)
For people that prefer to see code using the project. Every official Unified/Micromark/Remark/MDX project has 100% test coverages which offer a lot of highly tested examples.

allow both MDX maintainers and the community to contribute helpful, common configurations, setups and integrations

This is great in theory but doesn't happen in practice (see MDX is Stadium project above)

Even when contributions trickle through, they don't necessarily follow the best practices of the project.
Case and point, the official plugins list https://github.com/remarkjs/remark/pulls?q=is%3Apr+add+plugin+is%3Aclosed and the amount of back and forth it takes to get the typings working and basic tests in place.

first example above, a "Next.js + MDX + imported video" recipe/example would be not much code and would save a lot of time

This is a good example of why this is challenging.
I appreciate that you may have had a reason to use imports and self host a video.
That is not something I would recommend for a broader audience.
As a whole I'd recommend using a video hosting platform that handles transcoding, and a react component that offers an accessible interface around that video platform along with adaptive playback.
None of that needs customization at the MDX platform level.

Potentially also: Modern, updated, maintained end to end / integration tests with latest versions of integrating packages

I interpret this as meaning canary testing.
And I'm open to that, we already have a step like that for unified https://github.com/unifiedjs/unified/blob/242105bd6e18c61ca10f37d99529b89f1be37518/.github/workflows/main.yml#L25-L48 testing a few trusted downstream packages.

Create docs content dedicated to common interop problems with transitive dependencies and solutions for these

I still don't know what interop problems you are referring to.
Most of the ones I know of boil down to: please update your dependencies to the latest.
And the signal to know to do that is TypeScript is erroring.

rank highly on search engines with this content

We don't have a marketing team to go SEO chasing.
This is an open source project with a few maintainers (see MDX is Stadium project above)

Additional combined UIs for the API docs for MDX, remark/rehype/recma, unified packages

There is some linking between sites already.
Open to more linking.

even just including the types of other libraries without having to click on links would be a great start

I don't understand this comment.
Are you looking for hosted TSDoc?

CLI tools like mdx doctor which will find common issues like mismatching or duplicated transitive dependencies and other setup problems

Unified and all the umbrella projects have a large API surface, just versions doesn't communicate necessarily have enough information to communicate if they are compatible or not.
You may or may not care about a breaking change.

For that finer grained reporting we have TypeScript typings, which report when and where things mismatch.
I'm not sure making a second less capable version and less accurate version of is a good use of time.
Though open to other opinions.

Additional editor-level tools to navigate and resolve issues

That is what https://github.com/mdx-js/mdx-analyzer is here for, and it grows in capabilities regularly. (as you are aware regularly contributing issues)
If you have specific recommendation for the editor tools, feel free to share them!

Additional tools for integrations such as Next.js, to either offer features to smooth over the experience or to raise issues that users should resolve

This would be a great thing to raise with the Next team who manage the Next integrations

Additional chat-based UI for the docs for MDX, remark/rehype/recma, Unified packages, to allow for easier, faster navigation and search, when users don't use the exact wording or terms in the docs

Contributions to get intelligent search working again are welcome #1776

I don't think the team has the budget or interest in running a full LLM chat bot for the MDX docs (with July 2024 LLM cost and capabilities anyway).
If a project like https://githubnext.com/projects/copilot-for-docs/ comes around again we may consider enabling it.

[wooorm]

Thanks for sharing your thoughts!

Here are some responses to different parts. There’s a lot. Sometimes duplicated. So I haven’t responsed to everything. Let me know if there’s something I’m missing.

Let's take <video> support an MDX document in Next.js as an example:

Docs: Improve MDX new user experience

Given that the <video> issue was also raised in #2503, I do wonder whether this issue here is more about new users or people migrating. Info on things that worked before but no longer should be in a migration guide (this case, https://mdxjs.com/migrating/v2/#update-mdx-content), but not in the current guide for newcomers.

That MDX 2 release was big. I don’t see MDX change much ever again.

knowing already that they can add HTML to MDX - it works! 👍

I think this fundamental: MDX is not HTML. It isn’t markdown + HTML. If you want that, use markdown itself
MDX is JavaScript. It isn’t static. It’s a literal programming language. For when you find that you are writing a lot of tags like <em> and <a> often, you can use * and []() instead.
With this thinking, it makes more sense: would a <video src={whatGoesHere} /> work in a JavaScript file? That depends. It has a lot to do with where that JavaScript runs. Or how you configured your tools.
Then, videos on the web are also complex: only .mp4 is often not what you want.
Finally, MDX isn’t coupled to React (or even to HTML).
The problem this user faces has very little to do with MDX. It has much more to do with how else this user chooses to make a website.

Making websites is complex; compilers are complex.

User researches, finds rehype-mdx-import-media, using cmd-F, finds that it supports video[src] and tries configuring using the example config

user researches quickly

knowledgeable, experienced MDX users would have found the bullets under Usage

What’s that joke? PEBCAK 😅
At some point people, not just experienced people, are supposed to read. https://github.com/remcohaszing/rehype-mdx-import-media.
I think Christian mentions this too: we can’t solve this with more docs. People gotta read.
You have to see the example, which is a shortcut for turning A into B, and then think: is B what I want? Does that work for me? If not, this plugin isn’t for me. If yes, do I need a plugin or do I write it?

This leads to the following generated HTML:
(from your references issue remcohaszing/rehype-mdx-import-media#3)

After some time, finds a webpack loader which should be able to load .mp4 files.

Right, reiterating what I said above: how to use <video> in MDX with React through Webpack on Next.js, has very little to do with MDX, and more to do with how to use <video> with React through Webpack on Next.js.

Coming back to your earlier “User would like to import the .mdx file in another location. "abc.mp4" cannot be resolved anymore.” Answer that first. You don’t need all these plugins. Write it:

import myVideoSource from './some-video.mp4'

<video src={myVideoSource} controls />

Another example would be related to a user trying to include nested MDX content on part of a Next.js page

I don’t fully understand, can you explain?

A) Happy Path / Pit of Success

Lack of working, blessed, happy-path setup and integration suggestions for use cases beyond simple content

I don’t think these use cases are that simple or that they have that much to do with MDX.

An ecosystem for modern, updated, maintained recipes / cheatsheets / examples - eg. similar to the Next.js examples -
Potentially also: Modern, updated, maintained end to end / integration tests with latest versions of integrating packages

This is incredibly hard to keep up to date. I’m betting it costs Vercel tons, millions, to maintain that.
And, I’m hearing chatter about how RSC and Next.js and Webpack are too complex? So, does this really solve it for them?
MDX can be used with any JSX runtime, any frontent framework (react/preact/vue/svelte/solid/etc), any bundler, any, what you call it, next/remix/gatsby/astro.
I don’t think what you want is possible. The money or the time is missing. And the community can already help: they sometimes do (and it’s appreciated) but not that much.

Create docs content dedicated to common interop problems with transitive dependencies and solutions for these and rank highly on search engines with this content
Potentially also: docs pages for other common papercuts

Please raise these issues when you encounter them? I don’t know them.

Additional combined UIs for the API docs for MDX, remark/rehype/recma, unified packages, eg:
Allowing for seamless navigation among the docs "owners" without having to jump between multiple layers or websites (even just including the types of other libraries without having to click on links would be a great start)

unifiedjs.com has all the readmes. Still, it’s more than 500 readmes.
The code is >75 times the size as the source repo for Linux. It’s massive.

CLI tools like mdx doctor which will find common issues like mismatching or duplicated transitive dependencies and other setup problems

I think this makes more sense as a separate issue where you explain what is going on for you.
But, this feels like this is a problem of having multiple Reacts? npm ls react finds that?

Additional tools for integrations such as Next.js, to either offer features to smooth over the experience or to raise issues that users should resolve

This sounds like something on top of MDX: I think there’s lots of room above MDX for more site builder like tools.

If this issue or or parts of this issue are accepted, then I'm ok with this becoming the parent "umbrella issue" to track multiple initiatives.

I’m going to move this to a discussion. There’s lots of room for issues, but issues can be closed at some point.

[karlhorky]

Prefix 1: A voice for those without one

I should prefix my responses here with the fact that I'm working since years closely with beginners and mid-level engineers - teaching them how to break into tech and creating material for them (my material also started out terser and more intimidating, which I'm also trying to constantly improve). So my perspectives are based on what I observe as behaviors, tendencies, habits - what will work for new people to an ecosystem, and what will not. Of course, any project can decide that they cannot lower the barriers for any reason, including effort and investment.

My motivation is to use my privilege and experience as an experienced engineer to at least try to pave the cowpaths and make things easier for those who don't have a voice - those are not even opening issues because they don't get to that point.

Prefix 2: Easy to lose connection to beginners experience

I think in the position of experience, it's all too easy (I would argue, even natural) to lose a connection with the experience of those new to the scene.

What’s that joke? PEBCAK 😅

People gotta read.

I think it's reductive to put it on users and say "you need to just read"/PEBCAK. While I do agree with the general underlying rule, and I also almost daily give similar feedback gently (eg. "did you try reading the error message?"), I think that projects can also do better than this, and make the experience for new generations entering the scene better and simpler than it was for older generations. It does not always need to be about reading, and even when it's about reading, there is nuance.

If people aren't reading, then it's not always on the people - it's highly likely that it's unapproachable for some reason - docs are either buried, unclear, too jargony, too simplified, or too long. In interacting with the MDX/remark/rehype/Unified docs, the "buried", "jargony" and "too simplified" points have come up a number of times for me personally, although I'm able to re-read, research, invest time and move past these problems. If they're coming up for me, then that means for beginners, it's unapproachable.

[lacking documentation] is a worse experience for the developer than trying to read the source itself

This is maybe an example of what I mean with suggesting things not aligned with the experience of beginners. Code as documentation is a very high barrier, and will not be an option for most. Even experienced mid or senior level developers will not look at the source code for a source for docs.

I don't mean to say creating an approachable experience is easy. As I'm sure you're aware, creating consumable docs and software onboarding experiences is an extremely difficult balance: too much, and it's also unapproachable or unmaintainable. In my experience, I do however think that docs projects should tend towards over-explanation and explicitness instead of purity and simplicity.

Prefix 3: Cannot invest much time without acceptance

I'm unfortunately constrained in my time to write these long essays, even in the name of improving their experience. Since I'm feeling that my proposals here are not being considered fully / partially accepted (first feeling I get is more like mostly explanation of status quo + declining of the proposal), I feel like making a change here is an uphill battle that I cannot win. Again, this is just my first feeling and the temperature that I get from this thread. So taking all of that into consideration, I'll keep my responses shorter after this, and let this post speak for the words I do not write.

In total, I do hope that my thoughtful concerns from experience with beginners can come across and lead to some positive change, even with fewer words.

---

Custom Solutions

You were/are looking for a fairly custom solution. Not using an existing video player react component or video transcoding platform

I appreciate that you may have had a reason to use imports and self host a video.
That is not something I would recommend for a broader audience.
As a whole I'd recommend using a video hosting platform that handles transcoding, and a react component > that offers an accessible interface around that video platform along with adaptive playback.

I don't think a simple, short video is something that should be considered custom. <video> is accessible on its own, and producing a video is something that is supported on the level of the operating system, with no custom software. A video is less heavy than a .gif that many web developers are happy to put in a page.

This was actually the point that I had near the start of the original post above:

there's a long tail of complexity and high chance for breakage in moving beyond simple examples

I think there are many of these which are being called "custom solutions" which are actually pretty simple use cases of MDX.

Examples / Recipes

3. Incorrect documentation is worse than no documentation. Document needs to: work, reflect the latest version of dependencies, and reflect best practices. If it does not do all of these, it is a worse experience for the developer than trying to read the source itself.

I land pretty strongly against example folders like this, they aren't maintained and tested the way they should be, and that can cause more confusion that it can help (see Incorrect documentation is worse than no documentation above)

This is great in theory but doesn't happen in practice (see MDX is Stadium project above)

Ok, well if that's the opinion of the MDX project, then I think it's probably a main irreconcilable difference that will cause such beginner improvements as proposed to not go ahead.

I strongly disagree with the "Incorrect docs are worse than no docs" too (probably a controversial opinion). This is having seen how examples - also unmaintained examples (incl. 1st-party, partially-1st-party-maintained examples and 3rd-party examples and blog posts) - can help out beginners in countless ways. All code and learning material that is published has a high likelihood of being broken at some point and should be taken with a grain of salt.

New vs Migrating

I do wonder whether this issue here is more about new users or people migrating. Info on things that worked before but no longer should be in a migration guide (this case, mdxjs.com/migrating/v2), but not in the current guide for newcomers.

That MDX 2 release was big. I don’t see MDX change much ever again.

A bit confused at this one - the point was not about me knowing that MDX v1 had a feature and upgrading/migrating. It was about me (with the "new user hat" on), trying to use a feature which it felt that MDX should have. Since it was mentioned in some buried issues with a workaround which no longer works, the experience was bumpy.

I'm thinking about it from first principles: does a new MDX user think that, because it's possible to replace an MDX HTML element written like ![](), that it should also be able to replace an MDX HTML element written like <video>?

Irreducible Complexity / Infinite Solution Space

Inherent and irreducible complexity. All problems have a degree of complexity which can't be reduced or abstracted away. Parsers/Compilers has a significant amount of theory and complexity surrounding them.

I think this fundamental: MDX is not HTML. It isn’t markdown + HTML. If you want that, use markdown itself
MDX is JavaScript. It isn’t static. It’s a literal programming language. For when you find that you are writing a lot of tags like <em> and <a> often, you can use * and []() instead.
With this thinking, it makes more sense: would a <video src={whatGoesHere} /> work in a JavaScript file? That depends. It has a lot to do with where that JavaScript runs. Or how you configured your tools.
Then, videos on the web are also complex: only .mp4 is often not what you want.
Finally, MDX isn’t coupled to React (or even to HTML).
The problem this user faces has very little to do with MDX. It has much more to do with how else this user chooses to make a website.

Making websites is complex; compilers are complex.

Infinite solution space. MDX is a programming language with a content focus. It can do anything, people use it in wildly varied and creative ways from document editing, interactive experience creation, learning management system foundations, and much much more. The number of hypothetical uses are infinite, and even in practice have been growing exponentially since the project's inception.

Right, I can't disagree with anything here - the pieces are complex and the whole picture is complex. Same with metaframeworks like Expo, Next.js or any other package that consumes many other packages.

The point that I'm trying to make is that users shouldn't need to care about that at the beginning. Even when they are doing something that is a bit more custom than the simple first examples - they don't need to be met with the irreducible complexity of a parser/compiler ecosystem. It could be an easier experience - MDX could be more beginner friendly.

There should be some examples of going beyond the simple examples and achieve a bit more.

And, I’m hearing chatter about how RSC and Next.js and Webpack are too complex? So, does this really solve it for them?

• webpack being complex / JS fatigue was often about all of the configuration options, which Next.js has mostly abstracted away
• the latest complaints about Next.js + RSC being too complex are often from experienced developers with previous SPA React knowledge that they ned to re-learn. for beginners this is less of a concern
• the examples are very helpful in my experience - even if they are out of date or only community-maintained

Time/Money/Effort Required

2. MDX is Stadium project. Unified/Micromark/Remark/MDX have millions of active developer users, and 3 active maintainers currently. (see more on project types here: blas.com/working-in-public)

This is incredibly hard to keep up to date. I’m betting it costs Vercel tons, millions, to maintain that.

MDX can be used with any JSX runtime, any frontent framework (react/preact/vue/svelte/solid/etc), any bundler, any, what you call it, next/remix/gatsby/astro.
I don’t think what you want is possible. The money or the time is missing. And the community can already help: they sometimes do (and it’s appreciated) but not that much.

An alternative if the investment / effort is too high would be to let the examples be only partially maintained. But maybe this is untenable, for example if doesn't fit the quality goals of the MDX project.

But yeah, I'm assuming that is probably also a 2nd irreconcilable difference, if the time/money/effort is not there.

Existing Initiatives

Contributions to get intelligent search working again are welcome #1776

I don't think the team has the budget or interest in running a full LLM chat bot for the MDX docs (with July 2024 LLM cost and capabilities anyway).
If a project like githubnext.com/projects/copilot-for-docs comes around again we may consider enabling it.

Amazing, thanks for letting me know about these! I'll look forward to any movement with these.

As you move from the higher level abstraction to lower level ones for more advanced use cases it does get more complex.
We offer some introductory materials to the concepts in unifiedjs.com/learn
It really could be it's own book, there was even a start on one once upon a time unifiedjs/handbook but it was never finished.

Also really great, those look like high quality, interesting pieces. I'll keep them bookmarked for myself.

Not Specific Enough

If you have specific recommendation for the editor tools, feel free to share them!

Please raise these issues when you encounter them? I don’t know them.

There were a number of responses that I wasn't specific enough in my proposals or not illuminating in my examples. I tried to go into as much detail as I could in the amount of time it took me to write, edit and rewrite the essay above.

I probably was unable to fully communicate the depth of my experience and what I believe to be a good beginner experience in a specific, detailed way.

I'm sure that I will run into problems like this again in the MDX ecosystem, and maybe I will take the time to fully document them again.

For now, I'll stick with making small suggestions when I see them, and maybe will come back to this thread in future.

[ChristianMurphy]

Prefix 1: A voice for those without one
Prefix 2: Easy to lose connection to beginners experience

A friendly reminder you are talking to two accredited university instructors.
Who on top of teaching have spent years answering questions about these libraries for free for years.
Step all the way off your high horse, and have a conversation with peers. 🙂

I'm unfortunately constrained in my time to write these long essays, even in the name of improving their experience. Since I'm feeling that my proposals here are not being considered fully / partially accepted (first feeling I get is more like mostly explanation of status quo + declining of the proposal), I feel like making a change here is an uphill battle that I cannot win. Again, this is just my first feeling and the temperature that I get from this thread. So taking all of that into consideration, I'll keep my responses shorter after this, and let this post speak for the words I do not write.

Contributions are welcome!
This is the first time you've shared you have interest in helping beyond sharing a feature wish list.
It's worth discussing specific proposals before making a massive PR.

it's highly likely that it's unapproachable for some reason - docs are either buried, unclear, too jargony, too simplified, or too long.
The point that I'm trying to make is that users shouldn't need to care about that at the beginning.

There are high level packages like react-markdown and mdx which aim to abstract away a lot of the compiler concepts.
That doesn't seem to be where you are focusing here though, you are speaking more to the lower level parser and compiler customizations.

And as an instructor, you are likely familiar with the concept of competency pathways and skill progressions.
Concepts build on each other.
Parsers and compilers do indeed require some strong foundational knowledge of JS and programming concepts to use. (again see irreducible complexity above).

Which is why unified has guides, recipes, and a handbook.
There is a literal book worth of knowledge to understand, which at a university is usually taught in a dedicated 16 week course.

Happy to discuss improving those docs to better build concepts and ways to better link to help newcomers build that foundational knowledge.

Examples / Recipes

I land pretty strongly against example folders like this, they aren't maintained and tested the way they should be, and that can cause more confusion that it can help (see Incorrect documentation is worse than no documentation above)

Ok, well if that's the opinion of the MDX project, then I think it's probably a main irreconcilable difference that will cause such beginner improvements as proposed to not go ahead.

This particular implementation of the idea may be irreconcilable.
It also runs contrary to your strong stance on

Code as documentation is a very high barrier, and will not be an option for most.

But! Don't miss the links to the ways the project is offering examples and recipes which are curated and maintained

We have this! https://unifiedjs.com/learn/
And more at https://mdxjs.com/

Happy for these to continue to expand, and it feels like the commentary around the recipes helping new adopters understand them better aligns with your overall ideal.

I strongly disagree with the "Incorrect docs are worse than no docs" too (probably a controversial opinion). This is having seen how examples - also unmaintained examples (incl. 1st-party, partially-1st-party-maintained examples and 3rd-party examples and blog posts) - can help out beginners in countless ways.

It is controversial.
As a teacher and a maintainer, I mostly see these outdated examples lead beginners down a rabbit hole of incorrect assumptions and wind up with a massive XY problem which takes longer to resolve than reading the docs (and taking the time to understand it) or asking a question early on would have.

When to backtrack, revisit assumptions, and ask questions is a great topic for a CS/Engineering course.
It almost reads a bit like you'd like to unload that job on volunteer open source maintainers rather than doing it in your paid courses 😉

And, I’m hearing chatter about how RSC and Next.js and Webpack are too complex? So, does this really solve it for them?

the latest complaints about Next.js + RSC being too complex are often from experienced developers with previous SPA React knowledge that they need to re-learn. for beginners this is less of a concern

It's fine because beginners don't know how HTML, DOM, or React work so they aren't bothered by how Next/RSC break that model?
That is the least reassuring, reassurance I've read in a long time 😅

There were a number of responses that I wasn't specific enough in my proposals or not illuminating in my examples. I tried to go into as much detail as I could in the amount of time it took me to write, edit and rewrite the essay above.

I probably was unable to fully communicate the depth of my experience and what I believe to be a good beginner experience in a specific, detailed way.

Happy to also explore other avenues if essay writing isn't feeling effective.

For now, I'll stick with making small suggestions when I see them, and maybe will come back to this thread in future.

Nothing wrong with small suggestions either! 🙂
Often larger initiatives grow from trying out an idea in one spot and expanding it elsewhere after it works.

[karlhorky]

the latest complaints about Next.js + RSC being too complex are often from experienced developers with previous SPA React knowledge that they need to re-learn. for beginners this is less of a concern

It's fine because beginners don't know how HTML, DOM, or React work so they aren't bothered by how Next/RSC break that model? That is the least reassuring, reassurance I've read in a long time 😅

I think this one is getting a bit off-topic so I'll reply shortly here:

• it's not about breaking any model, and is not about HTML and DOM
• it's about seniors having previous knowledge (eg. getServerSideProps or SPA data fetching patterns) which they need to update, which is uncomfortable
• RSC is a better, simpler model, after putting in the work to get to the other side
• beginners don't need to put in the work to update existing knowledge because they are unburdened by this knowledge

[ChristianMurphy]

Ahhh, you and I have different ideas of experienced 😅
Yes, the Next API changes every few months, folks who learned that need to re-learn it.
I'm making a point back to the last cycle of hybrid frontend and backend rendering (GWT and friends) and the pitfalls of limited separation of concerns and remote procedure call communication.
And how this cycle isn't all that different in its trade offs, but I digress.

[karlhorky]

the Next API changes every few months, folks who learned that need to re-learn it

Interesting - haven't heard this feedback nor seen this happening, having been pretty closely involved with the features Next.js v9-v15 (caveat: we've had to patch Next.js for students 1 time + contributed features a few times to keep on the happy path, but that's mostly DX, not API changes).

The big change was RSC + App Router (some caching details changed in the last versions, but this didn't affect the way it was working for students).

Would be interested if there was something else I missed.

I'm making a point back to the last cycle of hybrid frontend and backend rendering (GWT and friends) and the pitfalls of limited separation of concerns and remote procedure call communication.

Hmm ok, not sure I'm understanding - is GWT the short form for Google Web Toolkit? With RPC are you referring to Server Actions?

[ChristianMurphy]

Would be interested if there was something else I missed.

You didn't miss anything at a technical level.
At a higher level though, it feels like there was some versioning/stability mislabeling, declaring unstable APIs stable within Next.

Hmm ok, not sure I'm understanding - is GWT the short form for Google Web Toolkit? With RPC are you referring to Server Actions?

Yes and yes

[karlhorky]

Prefix 1: A voice for those without one
Prefix 2: Easy to lose connection to beginners experience

A friendly reminder you are talking to two accredited university instructors.
Who on top of teaching have spent years answering questions about these libraries for free for years.
Step all the way off your high horse, and have a conversation with peers. 🙂

I'd like it if we could avoid making this personal or heated here (which I perceive from the "high horse" comment). I'm sorry if my words came across as personally attacking.

My intention and point (which I think, still stands):

1. There are people who will not voice concerns here and I'm trying to do it for them
2. It's easy for anyone experienced (including myself) to lose connection with the beginner experience
3. I may be able to offer a different perspective, working with people who have not been exposed to tech in the same way that university students or users who voice concerns here

I'm not trying to call my credentials into consideration here or say that I'm somehow immune to this - I'm trying to say that I may have something new and unconsidered to offer here.

@ChristianMurphy If you personally have the feeling that I cannot offer a new perspective because I don't have any knowledge or experience beyond what is already here, then I don't need to pursue these points.

Incorrect Docs Have Value

So I'll expand a bit on this here:

• Incorrect docs can lead people in the wrong direction (agreeing with you)
• Incorrect docs were often correct at one point in time
• Much of published code (not only docs, blog posts, tutorials, any other sources) is now broken in some way, but was correct at one point in time
  • This problem will not be solved with any initiatives or technology known today
• Students should learn early (within the first days of learning to program, not in a separate course):
  • seeing code in some place has very little guarantee it will work ("try it out! let's see")
  • code that is not working can almost always still offer value
  • how to adapt non-working code to their code and make it work

I strongly disagree with the "Incorrect docs are worse than no docs" too (probably a controversial opinion). This is having seen how examples - also unmaintained examples (incl. 1st-party, partially-1st-party-maintained examples and 3rd-party examples and blog posts) - can help out beginners in countless ways.

It is controversial. As a teacher and a maintainer, I mostly see these outdated examples lead beginners down a rabbit hole of incorrect assumptions and wind up with a massive XY problem which takes longer to resolve than reading the docs (and taking the time to understand it) or asking a question early on would have.

When to backtrack, revisit assumptions, and ask questions is a great topic for a CS/Engineering course. It almost reads a bit like you'd like to unload that job on volunteer open source maintainers rather than doing it in your paid courses 😉

So responding to this, I agree that outdated or broken code / docs can cause people going down wrong paths - and they should know this too, and be cautious and generally skeptical of code that they see (from the start).

The value I see in "more code examples, even if it becomes outdated or broken" is that it does offer some perspective / illumination of what approaches have worked in the past (and which may still work today).

These examples, blog posts and other material will happen to MDX in the wild anyway - the only difference that I'm proposing is to set up a space with some rules and some small moderation, to try to cultivate such examples in a structured way (probably even more high quality than the existing resources being produced in the wild). But in talking through it, it sounds like it would need a different approach to quality of materials than is acceptable to MDX.

There's also no "unloading" here either - the teaching you're describing (or at least, our version of it, which I described in the sub-bullets above) needs to happen before students encounter such docs.

Irreducible Complexity / Infinite Solution Space

And as an instructor, you are likely familiar with the concept of competency pathways and skill progressions.
Concepts build on each other.
Parsers and compilers do indeed require some strong foundational knowledge of JS and programming concepts to use. (again see irreducible complexity above).

I think this mostly goes in the direction of the "Irreducible Complexity / Infinite Solution Space" topic.

I've tried to explain my perspective on this:

The point that I'm trying to make is that users shouldn't need to care about that at the beginning. Even when they are doing something that is a bit more custom than the simple first examples - they don't need to be met with the irreducible complexity of a parser/compiler ecosystem. It could be an easier experience - MDX could be more beginner friendly.

I see it like how users who are not computer scientists can also use a computer and achieve many amazing things with the built-in programs and tools.

My point is that projects and libraries and the programming space in general should attempt to be like this experience more - smooth on-ramp, with many chunks and pieces that users can plug together. Going a bit beyond the basics shouldn't require compiler knowledge and the experience can be more helpful for beginners.

Although I think there may also be disagreement with what is "custom" and not. I don't think that a Next.js + MDX + <video> tag example is very custom - I think this is pretty basic still. But it already has some unusual, bumpy parts to the experience. And as I mentioned before, I think there are a lot of other use cases beyond the absolute basics which also have some unusual and bumpy parts (since you mentioned I should report specific issues, I'm glad that MDX in general is open to these improvements and I will start with suggestions to improve explicitness of replacing literal HTML elements in MDX and I may get back to this with further issues as I encounter them again in the future).

Going through these various use cases (even if it's mostly with community support) and documenting them and smoothing out the bumpy bits would be a great help for beginners.

Compiler / Parser Material for Experienced Users

There will always be more experienced users who are looking for compiler / parser knowledge and want to go deeper to https://unifiedjs.com/learn/ and https://github.com/unifiedjs/handbook - btw I think for these users, this ecosystem is very approachable. And from what I've seen, it's better than other compiler/parser ecosystems - more precise and correct and more simply explained.

[wooorm]

My intention and point (which I think, still stands):

1. There are people who will not voice concerns here and I'm trying to do it for them
2. It's easy for anyone experienced (including myself) to lose connection with the beginner experience
3. I may be able to offer a different perspective, working with people who have not been exposed to tech in the same way that university students or users who voice concerns here
   I'm not trying to call my credentials into consideration here or say that I'm somehow immune to this

I'm trying to say that I may have something new and unconsidered to offer here.

I appreciate the intent and there might definitely be concerns you know that we haven’t seen before. But I have seen a ton of some of what you are saying. There are lots of beginners asking questions.
I read several things in your OP, such as a) tough for beginners, b) concrete user story of video, c) complexity with peer dependencies, quickly in 2 sentences, not very SMART.
I think this is where the disconnect is: several different things in one, some very concrete some very vague, very small to very broad. Some we’ve seen already. Some presented as if we’ve never heard of them.

unloading

I think this is pretty crucial actually. Not this word, not maliciously intent. But priorities.
We don’t have time. We are dealing with lots of beginners already. The one thing we can’t handle is more beginners raising issues. The cost for us is high. There are infinite beginners using these tools. The cost for them to enter my inbox is zero. Every issue is an “issue” in my head. I literally can’t handle more.

Vercel is paying tons to have those samples, I don‘t have the resources.
If you want to make these samples, be my guest. I’ll give you access.

I see it like how users who are not computer scientists can also use a computer and achieve many amazing things with the built-in programs and tools.

I think I disagree here. I mean, it sounds nice. Everybody happy.
The cool things about computers, I believe, isn’t that you can use built-in programs and tools.
The cool thing is that you can make those built-in programs and tools.
Sure there’s space for people using programs and tools.
But this space right here is for people that want to make them complex programs and tools.

My point is that projects and libraries and the programming space in general should attempt to be like this experience more - smooth on-ramp, with many chunks and pieces that users can plug together. […]

Again this sounds pretty nice, everyone wants everyone to understand everything, but I disagree.
I don’t think a scientific paper needs to be understood by persons outside of that scientific field.
Yes there is space for people explaining the new research findings from that paper into normal words so that regular people can graps what’s going on.
But there’s also space for complex things to exist.
Not everything is for beginners.

[…] Going a bit beyond the basics shouldn't require compiler knowledge and the experience can be more helpful for beginners.

Bringing it back, how to do videos in Next.js is more for them; I don’t think the video user needs to know about compilers either.
How to center a div isn’t for the CSS spec. How to make an accordion isn’t for the HTML spec. How to do a video in Next.js + MDX + <video>, is I believe something for Next.js. Because it’s different for Remix + MDX + <video>. Because it’s the same as Next.js + <video>.

Lastly, of course things can be improved. But we’re having like 10 different conversations. It has to be made more concrete to go somewhere. I don’t know how to do everything has to be simpler.

And from what I've seen, it's better than other compiler/parser ecosystems - more precise and correct and more simply explained.

<3

[karlhorky]

Thanks for the response.

I appreciate all of the effort going into MDX and the ecosystem (incl. the effort you mentioned about beginners) and empathize with the points about overload + zero cost.

Sorry for the chaos / shotgun approach of this issue + discussion - I do see what you mean that the sizing / granularity / focus of the topics was not consistent and probably hard to deal with. I think I've been saving it up for a while (not reporting issues as they came up because they were all small and seemingly insignificant on their own), and tried to capture as much as I could remember at one point in time.

For me, ownership of such a thing belongs the most with the MDX projects, rather than multiple approaches documented in multiple consuming frameworks. But the Next.js MDX docs could also use some work, and I'll continue to push for that too.

About the resources/time issue, also understood. I think I am also too constrained to build up a complete MDX-owned examples / recipes area currently on my own. But I'll keep it in mind in the coming months.

Going forward:

1. I'll keep reporting incompatibilities / setup issues / docs polishing as I see them
2. I'll think more about the examples/recipes area
   1. I'll consider creating karlhorky/mdx-tricks as an MVP for some first content for this, similar to karlhorky/github-tricks
3. I'll continue pushing other external docs initiatives elsewhere eg. Next.js MDX docs

[wooorm]

Please raise small seemingly insignificant things! There is also space for a whole user journey, here in discussions. But I believe it then has to go through smaller SMART-ish issues.

For me, ownership of such a thing belongs the most with the MDX projects, rather than multiple approaches documented in multiple consuming frameworks.

I really doubt this video story can be solved with a guide in https://mdxjs.com/guides/, which applies to different jsx runtimes and different frameworks: how to deal with images is also very specific to the web framework (https://docs.astro.build/en/guides/images/, https://github.com/Josh-McFarlin/remix-image, https://nextjs.org/docs/pages/api-reference/components/image). I also doubt whether the user in question would’ve looked there. But I don’t know what article is in your head. Perhaps you can write a draft and then we can discuss whether it fits there or somewhere else. Maybe a guide that covers how each framework handles images would be good. And then also how each framework handles videos.

I think I am also too constrained to build up a complete MDX-owned examples / recipes area currently on my own.

karlhorky/mdx-tricks as an MVP

Not everything has to be completely done; there might be space here; depends what you want to write!