-
-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve regexes #666
base: main
Are you sure you want to change the base?
Improve regexes #666
Conversation
Thank you!
What do you recommend?
I had implemented this the same way before, but some bundlers had trouble tree-shaking it and including the regex in every bundle, whether used or not. So I repeated it. Due to compression it is probably also better in terms of bundle size. |
I recommend using it as written, with
Fair enough, if bundle size trumps readability. Another solution would be to use the |
Thanks you for your feedback! It may take some time to finish this PR as other things have a higher priority at the moment. |
I've done a lot more research on the emoji regex and published an improved/fixed version as |
Thank you very much! I will probably review and merge this PR next week. |
OK, updated. Passes tests and lint. |
In the last-added commit, I updated the comment about To clarify what the emoji regex in this PR matches:
On that last point, unfortunately, some common-sense and broadly-supported emoji are not officially in the "RGI" list. Even the Unicode org provides emoji-test.txt that mixes in non-RGI emoji strings to help identify real-world emoji. And some emoji are commonly used in an underqualified or overqualified way (by including or excluding certain invisible Unicode markers) that prevents them from being matched by The regex here allows overqualified and underqualified emoji using a general pattern that matches all Unicode sequences that follow the structure of valid emoji. |
Thank you for your research! Is the new emoji regex more strict or accurate than the old one? |
Yes. If by "the old one" you meant my own iterations, this is the final version based on research in depth, and I've now therefore also published it as its own library (emoji-regex-xs). If there are changes to emoji-regex-xs in the future (e.g., if new versions of Unicode modify the general patterns for emoji), it can easily be updated by anyone, by simply copying the pattern from future versions of emoji-regex-xs and wrapping it in On emoji-regex-xsThis library shares the API and 3,000+ tests with emoji-regex. emoji-regex is very large (13 KB uncompressed), but it is authoritative (its author helped add things like Problems with Valibot's current emoji regexIf by "the old one" you meant the regex currently used in the code that this PR replaces, then yes, this PR is both more strict and more accurate. Here is the current regex that is being replaced: /^[\p{Extended_Pictographic}\p{Emoji_Component}]+$/u This is extremely wrong. I'm assuming you picked it up from Zod, which uses the same thing, and I can find other people posting it online, which is where Zod probably got it from. It presumably has spread virally because:
There are two big problems with the regex that this PR replaces:
Regarding the first problem, I already mentioned some of its false positives in earlier comments, but here are some additional details (not comprehensive):
The emoji regex in this PR fixes all of these issues. |
Thanks again for your research and detailed answer! I thought it would be the best DX if |
I'm not familiar with Valibot's APIs. Could Certainly, with the new emoji regex, something like this could easily be done. You'd just need to change the But this seems like something for a follow-up issue. I'd prefer to land this PR as is and for new functionality to be added afterward. PS: The labels for this PR should include |
I agree.
Any ideas on how to implement this? Maybe we could add something like a |
Okay, I looked at valibot.dev/api/string/ and valibot.dev/api/emoji/ to understand a bit more what you're referring to. I agree that The term "character" is very overloaded so I'd advise against using A concrete example is the emoji 'π©π»βπ©π»βπ¦π»βπ¦π»'. Edit: This might not actually be a great example because it's only currently rendered as a single user-perceived character on Microsoft and Facebook platforms, but if you're not on Windows, imagine it rendered like this. // Code unit length
'π©π»βπ©π»βπ¦π»βπ¦π»'.length;
// β 19
// Each astral code point (above U+FFFF) is divided into high and low surrogates
// Code point length
[...'π©π»βπ©π»βπ¦π»βπ¦π»'].length;
// β 11
// These are: U+1F469 U+1F3FB U+200D U+1F469 U+1F3FB U+200D U+1F466 U+1F3FB U+200D U+1F466 U+1F3FB
// Grapheme length
// I don't think there's a native JS method to count this, but there are at least 4 graphemes:
// (U+1F469 U+1F3FB) (U+1F469 U+1F3FB) (U+1F466 U+1F3FB) (U+1F466 U+1F3FB)
// Not sure whether ZWJ (U+200D) also counts as a grapheme, is a non-grapheme, or gets included with graphemes
// Grapheme cluster length
[...new Intl.Segmenter().segment('π©π»βπ©π»βπ¦π»βπ¦π»')].length;
// β 1
Since I think |
Just realized that my example emoji 'π©π»βπ©π»βπ¦π»βπ¦π»' currently only renders as a single user-perceived character on Microsoft and Facebook platforms. Since other platforms don't currently include a unique emoji design for it, they might render it as four discrete emoji characters in a row (one for each of the graphemes in the grapheme cluster), possibly while still selecting it as a single character. |
I didn't audit all the regexes. Just the specific ones described below.
EMOJI_REGEX
0
, etc.),*
, bare U+200D (ZWJ), and some symbols likeπ
,β
,π³
, andβ
even when they're not followed by U+FE0F (none of which should match). So I fixed it.\p{Me}
, or if it's a more limited set like just U+20E3. The segment in question (\p{Emoji}\uFE0F\p{Me}?
) is used to match emoji like2οΈβ£
which is made up of 3 code points: U+32 U+FE0F U+20E3.HEXADECIMAL_REGEX
0h|0x
with0[hx]
.IPV4_REGEX
(?:(?:[1-9]|1\d|2[0-4])?\d|25[0-5])
with(?:2[0-4]\d|25[0-5]|1\d\d|[1-9]?\d)
. IMO it's easier to read without the nested grouping, and it's the same length. Then replaced\d\d
with\d{2}
to work around an eslint error that I don't agree with.IPV6_REGEX
IP_REGEX