ES Modules: please discuss

[jgonggrijp]

I suspect this is a feature many people would like, though perhaps not as many as were the case for Underscore. I personally think that there would be value in being able to import Backbone.Events, Backbone.Model, Backbone.Collection and perhaps the extend class emulator without importing the entirety of Backbone. Modularizing Backbone has been part of stage 2 of my maintenance takeover plan (#4244 (comment)).

@vtalas started an attempt in #4215, but seems to have abandoned the branch since. As I will discuss below, I can think of a few good reasons why he might have given up.

I decided to make my own attempt, partly because I wanted to focus on regenerating a backwards-compatible UMD bundle using Rollup first, and partly because I didn't like Node.js's idiosyncratic .mjs extensions. I was planning to go a similar route as I have done for Underscore (see jashkenas/underscore#2826, jashkenas/underscore#2849 and jashkenas/underscore#2914), with .js files in a modules directory that has a nested package.json with type: "module". However, I ran into serious difficulties. Unless some Rollup or ESM guru can help me out, I'm afraid we will have to face some hard choices.

Below is a link to my attempt-in-progress. The branch is not at all clean, so please do not base new work on top of it. Rollup would be invoked with npx rollup -c --bundleConfigAsCjs.
master...jgonggrijp:modules-2

The difficulties relate to the way the present, hand-written UMD bundle treats jQuery as an optional import.

• The factory assumes that if jQuery is available, it is available synchronously. For this reason, I cannot use dynamic import() to approximate the behavior (unless I optionally import jQuery and then call a factory function that builds Backbone around the provided import, but that would defeat the purpose of modularization). Even if I could somehow use import(), it begs the question whether Rollup would translate that to a UMD bundle that is backwards-compatible with the existing bundle.
• In the CommonJS case, the existing UMD wrapper wraps the require('jquery') call in a try/catch. As far as I can tell, there is no Rollup parameter or plugin that lets me replicate that behavior.
• In the browser globals case, the existing UMD wrapper attempts to find jQuery under various names with the expression root.jQuery || root.Zepto || root.ender || root.$. Again, there seems to be no way to reproduce this logic with Rollup. I can pass such a string to output.globals.jquery, but Rollup attempts to use the entire expression as a single property lookup in that case.

I have tried and considered several solutions, but none of them is (1) treeshakable, (2) backwards-compatible and (3) sourcemap-tracable to the source modules at the same time:

• I could use @rollup/plugin-alias to replace the jQuery import by a hand-written file that attempts to replicate the behavior in the existing UMD wrapper and then export { $ }. It would do require('jquery') in the AMD and CommonJS cases and the root.jQuery || root.Zepto || ... thing in the browser globals case. Alas, doing require('jquery') in the AMD case means a breaking change; it replaces async module loading by sync module loading, which in most cases will likely not even work. I cannot do define(['jquery'], ...) in the AMD case, because that would not produce the $ on time in order to export it.
• I looked into writing a custom Rollup plugin in order to modify the internal logic of the UMD wrapper, but it is unclear from the documentation whether any hook will actually allow me to do that. Even if there is a way, such a patch would have to update the sourcemap, something for which I currently lack the expertise.
• I could more bluntly patch the UMD wrapper with @rollup/plugin-replace or a similar plugin. However, this is an extremely fragile solution that would break the sourcemap as well. Also, it is again unclear from the documentation whether I can even modify the UMD wrapper in this way.
• I could make Rollup generate a bundle, trim off the header and footer entirely, and embed the remaining code in an completely hand-written UMD wrapper that looks exactly like the existing one. This would work and would be backwards-compatible, but again, there would be no sourcemap. It also feels like a rather questionable solution.

So unless somebody can point me to a golden solution, I'm afraid we will have to make at least one of three sacrifices:

1. Give up on treeshakability. This basically means giving up on ES modules entirely. While this is a valid option, it does not seem very future-facing.
2. Give up on backwards compatibility. Also a valid option, but I feel that now is not the right time for a 2.0 version bump, so this would likely delay modularization substantially. This option would also mean figuring out an entirely new approach to the optional jQuery dependency.
3. Give up on sourcemapping from the generated UMD bundle to the source modules. At first sight, this may seem like a minor sacrifice compared to the other two, especially given that Rollup keeps the bundle very readable. However, as I described above, every approach I can think of to make this sacrifice without the other two is rather iffy.

So, what do we do? Please share all the opinions, ideas and suggestions you may have!

[chkpnt]

Is UMD still a thing? IMHO ESM should get the primary focus nowadays.

[jgonggrijp]

Why do you think ESM should be a top priority?

UMD definitely is still a thing. Three main reasons:

• At lot of legacy software that a lot of people depend on, still assumes UMD. It will not go away any time soon.
• <script> tags still do not support ESM by default, you have to add the type=module attribute.
• Node.js ESM "support" only became "not experimental" in version 15 and it is still a bit of a pain to use. The package.json type field also still defaults to commonjs.

Even if this were not the case, though, the main group to worry about is the existing Backbone user base. Breaking the interface so I can straightforwardly bundle the thing with Rollup will break the interface for everyone, not just the people who were still using UMD without an ESM emulation layer on top.

I welcome pull requests so we can discuss some concrete options; maybe someone will be able to think of a better solution than I managed so far.

[chkpnt]

Okay, my bad. I should not conclude from my use case to others.

As far as I remember, when I migrated the backbonejs application which I'm maintaining in 2020 from no module system with grunt concat to a module system with Webpack, ESM were already preferred over UMD. But maybe I remembered it wrong.

[jgonggrijp]

Yes, ESM is the standard and it has been for a while. Then again, not everyone is catching up at the same rate.

[vilbergs]

Would using the "exports" field in package.json help, more specifically the "conditional exports"? I haven't looked into the details, but on the surface it looks like it can point to different files based on the import mechanism used.

https://nodejs.org/api/packages.html#conditional-exports

I have also seen some packages simply keeping the ESM build separate in an "es" or "esm" subdirectory. So the import would look something like import { ... } from 'backbone/esm'. For better or worse it kind of implies a pipeline for the ESM build.

[jgonggrijp]

It will be part of the solution. However, for backwards compatibility, a UMD bundle would still be necessary.

[kuwv]

ESM and components are already widely used in many frameworks. I'd argue that right now I need Backbone to provide easy DOM management with the above from Vanillajs.

I don't need legacy support.

I'd say put the 1.x in a separate release LTS branch move forward. I understand that it would effect some compatibility with ecosystem but this is a good change.

[jgonggrijp]

@kuwv Could you clarify what you are proposing to do on the 2.0 branch?

[kuwv]

Sure

• move backbone 1.x into a separate release branch for maintenance (cjs, amd, umd)
• bump backbone main to 2 alpha
• focus on usability for esm and cjs
• implement class support and deprecate 'extend'
• improve super initialization

Then maybe other improvements such as graphql can happen.

[jgonggrijp]

@kuwv Please forgive me for quote-sniping; there is a lot to unpack here. I'll go through both of your posts:

ESM and components are already widely used in many frameworks.

Why mention other frameworks? Why could Backbone not do something that is different from other frameworks?

(I agree that there is a value to ESM and components, I just wonder why you feel that other frameworks are relevant here.)

I'd argue that right now I need Backbone to provide easy DOM management

I presume you mean that Backbone is currently doing this, too, but ...

with the above

... you want it to be ESM-based and integrate more with Web Components, ...

from Vanillajs.

... preferably without depending on jQuery?

What is currently lacking when you want to use Backbone with Web Components, and how does it bear relevance to ESM?

What problem do you think will be solved for you, if Backbone no longer depends on jQuery?

I don't need legacy support.

I'll take your word for that. Lots of other people do, though. Even when I change the interface, I want to keep migration feasible for most of those people.

* implement class support and deprecate 'extend'

ES classes are an even more difficult problem than ES modules. I agreed with Jeremy to postpone such a transition until a really good solution manifests itself. You can read #4245 for background. ESM will definitely happen sooner than ES classes.

* improve super initialization

What do you mean by this?

[kuwv]

It's fine.

I don't need legacy support.

I'll take your word for that. Lots of other people do, though. Even when I change the interface, I want to keep migration feasible for most of those people.

Let me clarify. An AMD/UMD based app does not need ESM support, as an ESM app does not need to support either AMD/UMD.

Backbone can be split into two supported versions: active and maintenance through the use of a "git workflows" with releases.

https://about.gitlab.com/topics/version-control/what-is-gitlab-flow/ or https://www.endoflineblog.com/oneflow-a-git-branching-model-and-workflow

Also, if SEMVER is done correctly an Alpha version can be used to prototype these ES6+ features without locking the project into it. This would allow implementation of classes and modules much more interactively with the community IMO. Supporting two versions also removes any pressure for legacy users to make immediate changes.

https://survivejs.com/maintenance/packaging/publishing/

That's why I think it wouldn't be beneficial to try to pack all package types and features within one build. It's just not needed. This is why versioning exists.

Why mention other frameworks? Why could Backbone not do something that is different from other frameworks?

When evaluating tools or frameworks I look at it's fits within an ecosystem. There are two primary use cases for backbone which are: vanillajs apps and microfrontend components. Backbone can be used with other frameworks to build smaller UI components that can then be assembled in something like React or Vuejs frontend.

https://single-spa.js.org/docs/ecosystem-backbone/

With microfrontends ESM/Comonents are actually viable approach with microservices internally (non-www). Ideally these systems want multiple UI components from each API it consumes to assemble pages.

Currently, backbone doesn't provide much help to developers here.

[jgonggrijp]

@kuwv I understand the concepts of semver and support branches. What is still unclear to me, is what changes you envision on the future-facing branch. Backbone as-is cannot be straightforwardly ported to ESM, otherwise I would have already done it.

I'm asking you a technical question, not a project maintenance question. How would you approach the optional jQuery dependency in an ESM-first restructuring of Backbone?

[kuwv]

@jgonggrijp Fair enough. I still believe it's important to be discussed here as this discussion is specifically about the builds.

As for your question, I'm sure SystemJS can be used from Rollup. The System.import supports conditional imports if the module exists in the path with transpilation. As a universal module loader it would not only provide CommonJS and ESM but can additionally allow AMD for jQuery. I feel like that's mostly there but might need some additional consideration for users.

Although, I would prefer if underscore would just provide the view capability backbone requires. But, I understand that would be a significant investment of time and effort.

[jgonggrijp]

Ah, I didn't think of SystemJS. I'll look into that, thanks for the tip!

[kuwv]

It seems ESM support is already coming to jQuery:

jquery/jquery#4592
jquery/jquery#5255

Also, a stopgap was provided:

https://esm.sh/[email protected]

[jgonggrijp]

That is certainly interesting. Just to be clear though, the problem with jQuery is not that it doesn't support ESM yet, but that Backbone currently has a "creative" approach to depending on it.

[kuwv]

Yeah, I've been looking at that too. The thread suggested https://www.npmjs.com/package/@rollup/plugin-node-resolve

But, that would be systems side.

[VitorLuizC]

If that's a block, maybe the solution is just an initialization function that receives $. So the consumers will provide it, instead of this auto-resolution

import * as Backbone from 'backbone';
import jQuery from 'jquery';

Backbone.setJQuery(jQuery);

[jgonggrijp]

Thanks for thinking along, @VitorLuizC. I think such a solution could work (and it would have a few other advantages), but it would be a breaking change. So that would amount to option 2, "give up on backwards compatibility".

[tiagox]

+1 to support ESM.

My main use case is consistency. We have a big project with a lot of dependencies only supporting CJS. It would be great being able to finally migrate all the CJS code to ESM to simplify bundle processes and to keep a consistente codebase (without having several ways to do each thing).

On a very personal note, the three caveats you've (@jgonggrijp) mentioned, are more likely a nice to have rather than must have.

[jgonggrijp]

@tiagox I always write all my code in ESM notation. The bundling tools add an emulation layer for the modules that are still written in CJS or AMD. What stops you from doing the same?

Treeshakability and sourcemap-traceability are non-optional, as far as I'm concerned. Without treeshaking there is no point in modularizing and without a sourcemap, you cannot see what's going on in your code. I can give up on backwards compatibility, and there might be no other option, but that means I have to wait and pool it with other breaking changes. I don't want major releases to be a frequent thing, users should be able to depend on some stability.

(Just to clarify, though: I'm still interested in magic solutions if anyone can think of one.)

[valeriivolkovskyi]

I would skip backward compatibility. There are no magic solutions; it is supposed to be changed to move forward. I would like to contribute with many things, but ES modules are a bottleneck. Personally, I love Backbone not because of stability, but also because of its simplicity and approach to building applications.
In any case, in the future it will be necessary to remove jQuery as dependency

[jgonggrijp]

Thank you for your comment, @valeriivolkovskyi. There is a lot to unpack!

I would skip backward compatibility. There are no magic solutions;

I didn't mean magic magic, rather "something I have not thought of yet". That being said, if SystemJS is not the silver bullet and I don't find anything else (and I suspect that will be the case), then backwards compatibility is certainly out of the window.

it is supposed to be changed to move forward.

Could you elaborate a bit on this? What are you referring to with "it"?

I would like to contribute with many things,

Which things? If there are too many to list, could you just give a few examples?

but ES modules are a bottleneck.

Do you mean they prevent you from making those contributions? How?

Personally, I love Backbone not because of stability, but also because of its simplicity and approach to building applications.

❤️

In any case, in the future it will be necessary to remove jQuery as dependency

Why?

[valeriivolkovskyi]

I mean, I believe it's crucial to transition to ES modules as soon as possible. This transition will accelerate opportunities for contributions to the framework and enhance its modernity, provided that the community is interested in it.

Which things? If there are too many to list, could you just give a few examples?

I have some ideas to improve re-rendering performance. Or I would help with ES classes, or other issues. Do we have a roadmap BTW?

Why?

are you believe jQuery is still worth keeping as a dependency?

[jgonggrijp]

@valeriivolkovskyi Please excuse me for firing all these questions at you. I just want to get to the essence of what you are saying.

I mean, I believe it's crucial to transition to ES modules as soon as possible. This transition will accelerate opportunities for contributions to the framework and enhance its modernity, provided that the community is interested in it.

Do you mean "ES modules are the modern way to do things, so Backbone will look cooler if it uses them, which might attract new contributors"? Or is there more to it?

I have some ideas to improve re-rendering performance. Or I would help with ES classes, or other issues.

But only after Backbone switches to ES modules? For what reason(s)?

Do we have a roadmap BTW?

Somewhat. I previously published this list, which is mostly about Backbone's state of maintenance. We are currently at stage 3. I intended to do ESM in stage 2, but postponed it for the reasons discussed in this ticket.

Other than that (i.e., concrete features), there has been a vague road map in my mind. I will try to list it as concretely as possible over here:

1. Don't change much unless it is necessary; I believe Backbone is mostly fine the way it is. Keep making small enhancements when they come up.
2. Nevertheless, I anticipate some breaking changes that I was hoping to aggregate in a single major release:
   • Upgrade to Underscore 2.0, once I have released that and find it stable enough. (Underscore 2.0 will support Map, Set, iterators, promises, etcetera, and become stateless and properly treeshakable.)
   • Reconcile with the new TypeScript-inspired class sugar, if we ever find a good way. See the numbered list at the end of Embrace prototypes #4245 (comment) for what I mean by "good".
   • Reorganize the way Backbone depends on jQuery and/or base Backbone.ajax on fetch by default.
   • Switch to ES modules (though I originally hoped to do this in a non-breaking way, as mentioned before).
3. Now that jQuery 4 is in beta, this might inform future change in Backbone as well. This might go with step 2, or it might warrant another major release, if necessary.
4. Resume just adding small enhancements.

are you believe jQuery is still worth keeping as a dependency?

I do, for the simple reason that a lot of Backbone's functionality depends on jQuery. Do you believe there is a need to get rid of jQuery as a dependency?

[techcto]

+1 remove jQuery -- Great talk -- I found myself refactoring my old backbone app, so glad this is going on. We just upgraded to the latest Webpack and swapped out all of the requires for imports. I am looking how to make the Router work with webpack ports, I am guessing an upgraded Router would also be nice.

[jgonggrijp]

@techcto What do you mean by an "upgraded" Router? What would you like to see changed about it?

[kuwv]

Do you mean "ES modules are the modern way to do things, so Backbone will look cooler if it uses them, which might attract new contributors"? Or is there more to it?

Let's not be dismissive. What backbone does well is provide a low barrier of entry to powerful SPA capabilities. Using standardized features does that better than esoteric or deprecated ones.

[jgonggrijp]

@kuwv let me get a few things out of the way.

• I am not being dismissive; I am just asking whether @valeriivolkovskyi is only talking about the coolness factor (which is there, anyway) or also about something else. I am not saying that his argument is wrong if it is only about the coolness, I just want to make sure I don't miss something.
• This question about coolness was in relation to contributing to Backbone, not using it.
• If you are building SPAs, you are using a bundling tool, and if you are using a bundler, you can use anything you want with ESM. There is absolutely nothing stopping you from doing import { Model } from 'backbone'; today with Backbone 1.6 or earlier.
• UMD is neither esoteric nor deprecated. Please leave out the hyperbole and stick to the facts.
• Don't forget that I want Backbone to become ESM-based, too. We are all on the same side. The question is not about whether to make the transition, but how.

[jdittrich]

If that's a block, maybe the solution is just an initialization function that receives $. So the consumers will provide it...

I know this breaks backwards compatibility, but this makes sense to me. I wonder if it is worth to explore if this would be easy to migrate to with a simple code change for backbone users or if it would create scenarios which get some stuck, unable to migrate.

[jgonggrijp]

@jdittrich For what it is worth, it makes sense to me, too. In fact, I think it is the most straightforward way to retrofit Backbone in an ESM jacket, if our aim is to keep Backbone otherwise the same as much as possible.

That is not necessarily my aim, though. Maybe I have been unclear about this. I want to offer the users stability that they can depend on, but if we're going to have a major update anyway, somewhere close to the library's 15th birthday, I'm thinking that maybe it should be as disruptive as possible. Pool as many breaking changes as we can think of, so that the next breaking update can wait another 15 years. It might as well be a full redesign? Thoughts welcome.

(And then continue to support Backbone 1.x for a while in parallel, as @kuwv has also suggested. And make sure there is a way in which Backbone 1.x and 2.x can be used together in a single application, to ease the transition.)

[jdittrich]

Pool as many breaking changes as we can think of, so that the next breaking update can wait another 15 years.
It might as well be a full redesign? Thoughts welcome.

Gladly! I just wanted to stay on discussion here, and it seems most concerns were around maintaining backwards compatibility and transitioning to ESM. But maybe we want another issue for collecting potentially breaking changes? (or is there already one?)

[jgonggrijp]

@jdittrich Good suggestion, thanks. I just opened a new ticket for that purpose, see reference above. @kuwv and @valeriivolkovskyi, I encourage you to reply there, since you seemed to be full of ideas.

Yes, the concerns around here are about transitioning to ESM and backwards compatibility. I'm afraid those two cannot be reconciled, though I have not looked deeply enough into SystemJS yet to be sure. Then again, if you suggest a "less-breaking" conservative solution, I get the impression that this is what you prefer. Is it? There would be nothing wrong with that.

[jdittrich]

"less braking" conservative solution… get the impression that this is what you prefer.

Yes. At least when (for now) keeping Backbone mostly as-is, but transitioning to ESM, it seems that an "less-breaking", minor migration-effort-solution (like having to inject jQuery) might be a good solution.