Make E.String always ascii or utf-8, never utf-16

[pfgithub]

What does this PR do?

• Fixes a bug where ("0" + "1") == 0 was true
• Supports folding "\x00" + "1" to "\x001"
• Supports folding +"§" to NaN
• Supports folding unicode strings:
  • "æ" + "™" to "æ™"
  • "\uD800" + "\uDF34" to "𐌴".
• Fixes invalid utf-8 in a string did not correctly convert to the replacement character
• Fixes Syntax Error in String.raw #14213
• Fixes let abc\u0070 = 5 would define a variable called "p" rather than "abcp"
• Disallows `\8` because numeric escapes are not allowed in untagged template literals
• Fixes a bug where the utf-8 sequence (4)(cont)(cont)(cont)(ascii) would parse incorrectly, eg
  • 0b11110010 0b10101100 0b10110111 0b00101011 would parse as "\u{ACDEB}" rather than "���+" (or "�+")

Reworks E.String

Previously:

• For javascript, E.String was ascii or wtf-16, and constant folding "a"[0] and "a".charCodeAt(0) depended on not having utf-8.
• For JSON, E.String was always utf-8 with no rope segments. Lots of code accessed string.data directly.
• At least one call site in javascript forgot to collapse the rope string before using it (why ("0" + "1") == 0 gave the wrong answer)

Now:

• In javascript, E.String is wtf-8 or a wtf-8 rope string.
• In json, E.String is always wtf-8 and never a rope string
• It is now more difficult to access an uncollapsed rope segment by accident (.e_string.data is now .e_string._value_or_segment_value)

TODO:

benchmark is in path ~/Dev/Node/temp/generated/fdae05571e21265e7f76681435cbed48/tmp

performance:

• must remove strings.isValidUTF8 check. this can be done by making unicode special and then after seeing a unicode character once, next
  itertation it doesn't have to be special anymore and we can go back to strings.isValidUTF8
• text_.len < 64 doesn't have a significant effect
• try arraylist.toOwnedSlice() over allocator.dupe() in parseStringCharacters

code:

• remove fn decodeEscapeSequences

correctness:

• fix is_all_ascii is only set if a non-ascii byte is found but it also needs to be set if a non-acii codepoint is added in escape parsing
• implement the looser string parsing for tagged template literals
• add more tests
• compare benchnark bundle output before and after string parsing changes, make sure it is identical
• make sure json parsing works right and disallows things it shouldn't allow
• run parser tests (https://github.com/tc39/test262-parser-tests) and add them to automated testing
  • pass, todo, or skip all tests
  • use test.skip rather than not adding the test for skip tests
  • add so that CI runs it
  • make sure CI runs with --todo flag
• run json tests (https://github.com/nst/JSONTestSuite). re-emit and run again and make sure the result is identical.
  • pass, todo, or skip all tests
  • add so that CI runs it
  • make sure CI runs with --todo flag
• eliminate lexer.peek(); it's used exactly once and implemented wrong
• nextCodepointSlice() doesn't validate the codepoint - an invalid codepoint should return a 1-len slice rather than eg 3-len (ie c3 c3 should nextCodepointSlice to [c3] not [c3, c3]`)
• Should octal escapes in strings be allowed? In 1.1.34 they half-worked (\32 syntax error but \324 ok)
• make sure bundler_string_2.test.ts runs with --todo flag
• js_printer.zig writePreQuotedString \u{{{d}}} -> \u{{{X}}}
• remove "decodeWTF8RuneT", "wtf8ByteSequenceLength", "codepointSize"
• audit "NewCodePointIterator", "CodepointIterator", "UnsignedCodepointIterator" - these have been changed to not stop on a null byte, make sure the call sites expect that. consider removing these iterators and replacing them with a much simpler one.
  • If keeping, nextCodepointSlice has an incorrect implementation, nextCodepoint asserts utf-8 (not wtf-8), peek is wrong, and the api seems overcomplicated?
• compare "PackedCodepointIterator" vs regular codepoint iterator
• remove "std.unicode."
• compare strings.wtf8Sequence(), strings.encodeWTF8RuneT(), strings.unicode.encodeWtf8WithInvalid() and deduplicate
• identify and fix performance regression in 7dae15b and 65a8dcc
• See if there were any bugs around U+FFFD handling in previous bun versions and if they have been fixed (there is probably a case of ? -> ???)
  • Maybe in toml
• See if there were any bugs around null byte handling in previous bun versions (there is probably a case of null -> ?)
• Add a test for 0b11110010 0b10101100 0b10110111 0b00101011
• Compare performance on windows because it is probably worse due to the added assertions

TODO:

• Performance comparisons
  • 1.1.34 vs e595526: consistently 1.1.34 is ~1.06x faster with bundling a large json file
  • 1.1.34 vs ee75ab1: consistently v1.1.34 is ~1.05x faster (after e.string size reduction & merge of main)
  • 1.1.34 vs dd7b3ae: v1.1.34 is ~1.01x faster (after lazy isAllASCII)
• Pass tests
  • bundler_minify.test.ts
  • esbuild/extra.test.ts
  • transpiler.test.js (backtick string preservation)
  • bundler_edgecase.test.ts (backtick string preservation)
  • cd test && bun-debug install
• E.String size optimizations:
  • Make sure the union tag and is_from_template_string can share the same 8 bytes, rather than each taking 8 bytes

[robobun]

❌ @pfgithub, your commit 01ce93d has 24 failures in #7081:

test/js/web/fetch/body-stream.test.ts - timeout on 🐧 3.20 x64

test/regression/issue/09041.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/js/bun/http/bun-serve-static.test.ts - timeout on 🐧 3.20 x64-baseline

test/integration/next-pages/test/dev-server.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/integration/next-pages/test/dev-server.test.ts - 1 failing on 🐧 3.20 x64

test/js/node/test/parallel/worker-nested-uncaught.test.js - segmentation fault on 🐧 3.20 aarch64

test/js/node/test/parallel/worker-nested-uncaught.test.js - segmentation fault on 🐧 3.20 x64

test/v8/v8.test.ts - 22 failing on 🐧 3.20 aarch64

test/v8/v8.test.ts - 22 failing on 🐧 3.20 x64

test/v8/v8.test.ts - 22 failing on 🐧 3.20 x64-baseline

test/js/node/test/parallel/fs-watch-recursive-linux-parallel-remove.test.js - 1 failing on 🐧 22.04 x64-baseline

test/js/node/test/parallel/fs-watch-recursive-linux-parallel-remove.test.js - 1 failing on 🐧 22.04 x64

test/js/node/test/parallel/fs-watch-recursive-linux-parallel-remove.test.js - 1 failing on 🐧 12 x64

test/js/node/test/parallel/fs-watch-recursive-linux-parallel-remove.test.js - 1 failing on 🐧 3.20 aarch64

test/js/node/test/parallel/fs-watch-recursive-linux-parallel-remove.test.js - 1 failing on 🐧 3.20 x64-baseline

test/js/node/test/parallel/fs-watch-recursive-linux-parallel-remove.test.js - 1 failing on 🐧 3.20 x64

test/js/bun/http/serve.test.ts - SIGTRAP on 🐧 3.20 aarch64

test/js/bun/http/serve.test.ts - SIGILL on 🐧 3.20 x64-baseline

test/js/bun/http/serve.test.ts - SIGILL on 🐧 3.20 x64

test/js/node/child_process/child_process.test.ts - 1 failing on 🐧 3.20 aarch64

test/js/node/child_process/child_process.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/js/node/child_process/child_process.test.ts - 1 failing on 🐧 3.20 x64

test/cli/hot/hot.test.ts - 2 failing on 🐧 22.04 x64

test/cli/hot/hot.test.ts - timeout on 🐧 3.20 x64-baseline

test/cli/hot/hot.test.ts - timeout on 🐧 3.20 x64

test/bake/dev/bundle.test.ts - 1 failing on 🐧 3.20 aarch64

test/js/node/test/parallel/zlib.test.js - 1 failing on 🍎 13 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🍎 14 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🪟 2019 x64

test/regression/issue/03830.test.ts - 1 failing on 🪟 2019 x64-baseline

test/regression/issue/03830.test.ts - 1 failing on 🍎 13 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🐧 12 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🐧 22.04 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🐧 11 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🐧 20.04 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🐧 20.04 x64-baseline

test/regression/issue/03830.test.ts - 1 failing on 🐧 12 x64-baseline

test/regression/issue/03830.test.ts - 1 failing on 🐧 22.04 x64-baseline

test/regression/issue/03830.test.ts - 1 failing on 🐧 11 x64-baseline

test/regression/issue/03830.test.ts - 1 failing on 🐧 20.04 x64

test/regression/issue/03830.test.ts - 1 failing on 🐧 11 x64

test/regression/issue/03830.test.ts - 1 failing on 🐧 22.04 x64

test/regression/issue/03830.test.ts - 1 failing on 🐧 12 x64

test/regression/issue/03830.test.ts - 1 failing on 🐧 3.20 aarch64

test/regression/issue/03830.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/regression/issue/03830.test.ts - 1 failing on 🐧 3.20 x64

test/regression/issue/03830.test.ts - 1 failing on 🍎 14 x64

test/regression/issue/03830.test.ts - 1 failing on 🍎 13 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🍎 14 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🪟 2019 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🪟 2019 x64-baseline

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🍎 13 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 22.04 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 11 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 12 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 20.04 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 12 x64-baseline

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 20.04 x64-baseline

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 22.04 x64-baseline

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 3.20 aarch64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 11 x64-baseline

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 11 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 12 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 20.04 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 22.04 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 3.20 x64-baseline

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🐧 3.20 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🍎 14 x64

test/js/bun/resolve/toml/toml.test.js - 1 failing on 🍎 13 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🍎 14 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🍎 13 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🪟 2019 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🪟 2019 x64-baseline

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 12 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 20.04 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 11 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 22.04 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 22.04 x64-baseline

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 11 x64-baseline

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 12 x64-baseline

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 20.04 x64-baseline

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 20.04 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 11 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 22.04 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 12 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 3.20 aarch64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 3.20 x64-baseline

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🐧 3.20 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🍎 14 x64

test/js/bun/http/bun-serve-ssl.test.ts - 4 failing on 🍎 13 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🍎 14 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🍎 13 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🪟 2019 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🪟 2019 x64-baseline

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 11 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 22.04 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 20.04 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 12 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 22.04 x64-baseline

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 12 x64-baseline

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 11 x64-baseline

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 20.04 x64-baseline

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 12 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 11 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 20.04 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 3.20 aarch64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 22.04 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🐧 3.20 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🍎 14 x64

test/external/JSONTestSuite.with_todo.test.ts - 1 failing on 🍎 13 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🍎 14 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🍎 13 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🪟 2019 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🪟 2019 x64-baseline

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 11 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 12 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 22.04 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 20.04 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 11 x64-baseline

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 22.04 x64-baseline

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 20.04 x64-baseline

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 12 x64-baseline

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 3.20 aarch64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 11 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 22.04 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 12 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 20.04 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🐧 3.20 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🍎 14 x64

test/external/test262-parser-tests.with_todo.test.ts - 1 failing on 🍎 13 x64

test/cli/install/bun-install.test.ts - 3 failing on 🍎 14 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🍎 13 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🪟 2019 x64

test/cli/install/bun-install.test.ts - 3 failing on 🪟 2019 x64-baseline

test/cli/install/bun-install.test.ts - 3 failing on 🐧 12 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 11 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 22.04 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 20.04 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 20.04 x64-baseline

test/cli/install/bun-install.test.ts - 3 failing on 🐧 12 x64-baseline

test/cli/install/bun-install.test.ts - 3 failing on 🐧 22.04 x64-baseline

test/cli/install/bun-install.test.ts - 3 failing on 🐧 11 x64-baseline

test/cli/install/bun-install.test.ts - 3 failing on 🐧 3.20 aarch64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 20.04 x64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 11 x64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 12 x64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 22.04 x64

test/cli/install/bun-install.test.ts - 3 failing on 🐧 3.20 x64-baseline

test/cli/install/bun-install.test.ts - 3 failing on 🐧 3.20 x64

test/cli/install/bun-install.test.ts - 3 failing on 🍎 13 x64

test/cli/install/bun-install.test.ts - 3 failing on 🍎 14 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🍎 14 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🍎 13 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 11 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 22.04 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 12 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 20.04 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 12 x64-baseline

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 20.04 x64-baseline

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 11 x64-baseline

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 22.04 x64-baseline

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 3.20 aarch64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 11 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 12 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 20.04 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 22.04 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 3.20 x64-baseline

test/bundler/esbuild/default.test.ts - 1 failing on 🐧 3.20 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🍎 14 x64

test/bundler/esbuild/default.test.ts - 1 failing on 🍎 13 x64