vite dev/test/build from app #1695

patricklx · 2023-12-01T23:36:49Z

this can now already load most from root folder ./app and tests folder instead of rewritten_app folder. a few hacks where required.

the only files read from rewritten_app are :

config/environment.js
assets/test-support.js
assets/*.css
index.html

most problematic where asset loading.
vite does not allow loading static files from outside of root dir, so setting root to /app will not work.
using the CWD as root, will not load the index.html file.

this is where the custom middleware comes in to handle this parts.

load index.html foim app dir
load assets from public dir
load assets from node-module package
load from rewritten-packages

missing parts to allow editing live reload:

environment.js -> i think we should just call the vite-app/config/environment.js again at runtime (browser) and merge it again , like this, developers can edit the env and have it load by vite.
assets/vite-app.js
transform index.html in vite --> need replacements for content-for in a separate file so we can do it our selfs and expose api to build index.html
working tests
working vite build
rollup plugin for assets
auto rebuild when lock file changed or no . embroider dir

patricklx · 2023-12-05T13:39:42Z

some tests are failing for unknown reason.
when I run test with pnpm test "--" "--filter" "release-compat-namespaced-app" it fails
but when I run it with node ./node_modules/qunit/bin/qunit.js --require ts-node/register *-test.ts "--filter" "release-compat-namespaced-app" it works...

patricklx · 2023-12-05T15:06:40Z

Should i add a Test for vite app?
Can just be vite build

patricklx · 2023-12-05T15:23:04Z

@ef4 this is ready for review

ef4

Thanks for giving this a go. Your spike showing this working is a helpful guide for several things, especially the middleware setup.

But this is not mergeable as is. A lot of stuff here is hard-coded in ways that might work in your app but definitely break in others. And @embroider/vite should only need to do vite-specific things.

Can we find time for you to pair with either me or @mansona? Because your help on this would be great, but it's so far not aligned with what we're trying to do. The effort that went into this PR could go directly into fixing these things in @embroider/core instead of hacking around them.

ef4 · 2023-12-05T17:56:14Z

packages/util/package.json

@@ -52,6 +52,7 @@
    "@ember/test-helpers": "^2.9.1",
    "@embroider/compat": "workspace:*",
    "@embroider/core": "workspace:*",
+    "@embroider/shared-internals": "workspace:*",


I think you have some unrelated changes in here. I don't see anything in this PR that would need this change.

Yes, not sure why tests where suddenly failing. It was failing we this missing dependency...

ef4 · 2023-12-05T17:56:59Z

packages/util/package.json

@@ -91,7 +92,8 @@
    "qunit": "^2.19.1",
    "qunit-dom": "^2.0.0",
    "typescript": "^5.1.6",
-    "webpack": "^5.74.0"
+    "webpack": "^5.74.0",
+    "semver": "^7.5.4"


Yes, failing tests because of this missing dependency... Not sure why.

ef4 · 2023-12-05T18:02:01Z

packages/vite/src/build.ts

+  return new Promise((resolve, reject) => {
+    const child = fork('./node_modules/ember-cli/bin/ember', ['build']);
+    child.on('exit', code => (code === 0 ? resolve() : reject()));
+  });


I don't think this is the best place to manage rebuilds of the addons. This kind of rebuild is extremely expensive compared with ember s.

We could still do the persistent caching you have here in order to avoid even a first-time build of the addons if none have changed since your last build. But that logic would go some place more like

embroider/packages/compat/src/standalone-addon-build.ts

Line 13 in 3480aa6

export function convertLegacyAddons(compatApp: CompatApp) {

Yes, well it would also be quite rare to do build. Only if .embroider folder does not exist and if package-lock changed and index.html change.
Though i would like to rebuild the index.html only instead of full build.

ef4 · 2023-12-05T18:03:20Z

packages/vite/src/build.ts

+
+export function build(): Plugin {
+  let resolverLoader = new ResolverLoader(process.cwd());
+  const engine = resolverLoader.resolver.options.engines[0];


engines other than the first need this feature too.

packages/vite/src/build.ts

ef4 · 2023-12-05T18:25:01Z

packages/vite/src/resolver.ts

+          server.watcher.add('./' + f);
+        }
+      });
+      server.watcher.add('./app/index.html');


I don't actually think we should bother supporting app/index.html. It's unnecessarily weird when we could just follow the normal vite convention and put index.html at the root.

It's not like people can use their original app/index.html anyway. Its format must change to work with this, if I'm reading correctly.

Sounds good. Currently I'm rewrite it in rewriteHtml and just load the one kn rewritten-app.
I move tge index.html to the root then

moving to root will not work, because the ember build will not work... unless we just put an empty index.html there... but then we will have 2 index.html files

I think i found an option to ember-cli to set index.html path. Will try that

ef4 · 2023-12-05T18:26:26Z

packages/vite/src/virtual-files.ts

@@ -0,0 +1,125 @@
+import { relative } from 'path';


All of this entry generation is adding a second implementation for us to maintain instead of fixing the implementation we have. It doesn't belong in packages/vite because it's not vite-specific.

Sounds good. I can look to replace the code.
webpack should support the same glob pattern

The glob pattern is only supported by vite...
Although there us a custom loader for webpack, it has a different syntax.
I think we should keep this for the app entries.
The addon entry file can be reverted to use the generated one by ember build

replacing this with compat app builder

packages/vite/src/virtual-files.ts

patricklx · 2023-12-06T18:52:33Z

I'm working now on using compat app builder to rebuild html and app assets when relevant files are added/removed. Looks working fine.

patricklx · 2023-12-07T15:11:24Z

@ef4 I made some changes

index.html at root
Asset related stuff is using compat app builder , i think like this we have a good base to possibly restructure internal code
added some more json files with info to use thr compat app builder to rebuild assets when relevant files changed
there where some issues when switching between app & tests. I fixed it by invalidating all.

patricklx · 2023-12-07T15:44:19Z

packages/compat/src/compat-app-builder.ts

@@ -57,6 +57,7 @@ import type { TransformOptions } from '@babel/core';
 import { MacrosConfig } from '@embroider/macros/src/node';
 import SourceMapConcat from 'fast-sourcemap-concat';
 import escapeRegExp from 'escape-string-regexp';
+import { configReplacePatterns } from 'ember-cli/lib/utilities/ember-app-utils';


Im not sure if there is another place where embroider does the replacements

* load env from root

patricklx · 2023-12-09T09:14:10Z

looks like the pr #1686 broke this...
Cannot find components/example. Any idea why?

…not like it

ef4 · 2023-12-09T23:47:17Z

Perhaps it was relying on the resolvableExtensions in the node resolver, which is not there anymore after #1686.

For normal module imports that would not be an issue, since we can just configure vite to resolve .ts. But it is potentially an issue for discovering components in non-strict templates, since we're using nodeResolve there.

That gives me an idea: we could probably use defaultResolve instead of nodeResolve for component discovery. That would mean that our component search would understand .ts automatically when the prevailing packager does.

patricklx · 2023-12-10T09:02:43Z

Also, I needed to add back the addons.ts for the depOptimizer as otherwise i was getting duplicates for glimmer-vm which does allow it.

patricklx · 2023-12-11T15:46:56Z

@mansona maybe you can already have a look at this now.
I tried to implement the required changes without affecting normal builds

patricklx · 2023-12-12T14:34:44Z

packages/compat/src/compat-app.ts

@@ -200,7 +209,7 @@ export default class CompatApp {

  private get indexTree() {
    let indexFilePath = this.legacyEmberAppInstance.options.outputPaths.app.html;
-    let index = buildFunnel(this.legacyEmberAppInstance.trees.app, {
+    let index = buildFunnel(this.legacyEmberAppInstance.trees.indexHtml || this.legacyEmberAppInstance.trees.app, {


required to have index.html at root for vite

patricklx · 2023-12-12T14:35:10Z

packages/core/src/ember-html.ts

@@ -83,7 +83,7 @@ export class PreparedEmberHTML {
  implicitTestStyles: Placeholder;

  constructor(private asset: EmberAsset) {
-    this.dom = new JSDOM(readFileSync(asset.sourcePath, 'utf8'));
+    this.dom = new JSDOM(asset.source || readFileSync(asset.sourcePath, 'utf8'));


used to load in-memory html content

patricklx · 2023-12-12T14:35:52Z

packages/core/src/module-resolver.ts

@@ -532,22 +532,41 @@ export class Resolver {
  }

  private *componentTemplateCandidates(inPackageName: string) {
-    yield { prefix: '/templates/components/', suffix: '' };
-    yield { prefix: '/components/', suffix: '/template' };
+    yield { prefix: '/templates/components/', suffix: '.hbs' };


needed to add extension and /app prefix since component resolution is only done with nodeResolver, can revert when it uses defaultResolve

patricklx · 2023-12-12T14:36:44Z

packages/core/src/module-resolver.ts

@@ -1041,10 +1060,10 @@ export class Resolver {
    // but then come back to the original location here in the fallback when the
    // rehomed request fails
    let movedPkg = this.packageCache.maybeMoved(pkg);
-    if (movedPkg !== pkg) {
+    if (movedPkg !== pkg && !pkg.isV2App()) {


if its a app then no need to error here. (in vite, app is at root, and everything is loaded from root)

patricklx · 2023-12-12T14:37:30Z

packages/vite/src/addons.ts

@@ -0,0 +1,23 @@
+import { ResolverLoader, packageName } from '@embroider/core';


added this back because i was getting duplicate packages, like glimmer-vm which does not like it

patricklx · 2023-12-12T14:50:58Z

tests/vite-app/tests/setup-harness.js

@@ -0,0 +1,59 @@
+/* eslint-disable no-console */


this is to support ci tests with vite

patricklx · 2023-12-12T14:51:39Z

packages/vite/src/resolver.ts

+    if (!result) {
+      result = await context.resolve(
+        request.specifier,
+        request.fromFile.replace('/package.json', '/app/package.json'),


need to look into /app folder

patricklx

@ef4 my current wip to virtualize generated assets. Am I om the right track?
149d5b7
Now i only need to make some files paths absolute instead of relative in the assets.
When this works i extract this into a separate pr.

patricklx mentioned this pull request Dec 2, 2023

[vite entry point] add gjs + gts extension for resolvable extensions #1679

Closed

patricklx marked this pull request as ready for review December 4, 2023 11:30

patricklx changed the title ~~vite load from app~~ vite dev/test/build from app Dec 5, 2023

patricklx force-pushed the vite-load-from-app branch from f386052 to 6567d93 Compare December 5, 2023 12:56

ef4 requested changes Dec 5, 2023

View reviewed changes

patricklx mentioned this pull request Dec 7, 2023

Resolver bugfixes #1686

Merged

5 tasks

patricklx force-pushed the vite-load-from-app branch from ee7f862 to 74a4290 Compare December 7, 2023 15:05

patricklx commented Dec 7, 2023

View reviewed changes

patricklx added 17 commits December 9, 2023 09:02

vite load from app

43ff779

wip

c124d5c

wip

40aaaa5

vite load from app

bc0a0c0

* generate entry file

aca0506

* load env from root

working tests

870d431

prevent loop in vite build

ce408fd

fix vite test

238fa38

fix vite test

7752af1

fix build

eff2fdd

add build file

53fcc85

fix lint

abfeab4

fix lint

3aa2856

revert

4239dda

revert

3883e96

fix lock file

effd7e6

fix pkg

19d9eac

patricklx added 2 commits December 9, 2023 09:03

fix

16d1dd8

fix env

b9ddc23

patricklx added 3 commits December 10, 2023 00:26

re-add addons, otherwise I get duplicates for glimmer-vm, which does …

7f160ac

…not like it

resolve ts gjs gts extensions

a58c41b

resolve components helpers with explicit extensions

a12e437

patricklx force-pushed the vite-load-from-app branch from e30d1b9 to a12e437 Compare December 9, 2023 23:32

ts resolving

8aabfac

patricklx added 10 commits December 11, 2023 12:33

update

0b2d0d7

fix lint

a0f5f64

remove package hack

6c38e0f

fix env update

69b633c

fix lint

0b596f6

revert

ac7ee5e

fix lock

15a1f53

fix lock

dfbdf35

fix types

74b64d0

fix eslint

a25434c

fix

a2980e7

patricklx commented Dec 12, 2023

View reviewed changes

patricklx added 2 commits December 13, 2023 12:42

virtualize

149d5b7

wip

a9c2869

patricklx commented Dec 13, 2023

View reviewed changes

patricklx added 2 commits December 13, 2023 22:56

fix

475ca54

fix

a4c0c2e

patricklx closed this Dec 22, 2023

		@@ -0,0 +1,23 @@
		import { ResolverLoader, packageName } from '@embroider/core';

vite dev/test/build from app #1695

vite dev/test/build from app #1695

Conversation

patricklx commented Dec 1, 2023 • edited Loading

patricklx commented Dec 5, 2023 • edited Loading

patricklx commented Dec 5, 2023

patricklx commented Dec 5, 2023 • edited Loading

ef4 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patricklx Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patricklx Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

patricklx Dec 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patricklx commented Dec 6, 2023

patricklx commented Dec 7, 2023 • edited Loading

Choose a reason for hiding this comment

patricklx commented Dec 9, 2023

ef4 commented Dec 9, 2023

patricklx commented Dec 10, 2023

patricklx commented Dec 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patricklx left a comment • edited Loading

Choose a reason for hiding this comment

patricklx commented Dec 1, 2023 •

edited

Loading

patricklx commented Dec 5, 2023 •

edited

Loading

patricklx commented Dec 5, 2023 •

edited

Loading

patricklx Dec 5, 2023 •

edited

Loading

patricklx Dec 5, 2023 •

edited

Loading

patricklx Dec 6, 2023 •

edited

Loading

patricklx commented Dec 7, 2023 •

edited

Loading

patricklx left a comment •

edited

Loading