Developing external extensions (v0.1)

[carletex]

This PR aim is to start the conversation on how to improve the development of external extensions. (I've been discussing with Shiv some of this)

Currently, there is no easy way to develop & test external extensions: you basically have to (blindly) create it, push it to GitHub, and test. This is not an issue for small extensions where you just add some new folders/files, but it gets super messy and annoying on bigger ones.

External Extension Dev workflow

We haven't found a perfect(simple) workflow, but this is where we are at:

Let's say you want to create the eip extension: it adds a new /eip page + a Header link + a new script to the root package.json (I've pushed it on this PR to test)

Phase 1

1. You clone create-eth repo
2. You create an instance with yarn dev & yarn cli, in the directory my-instance
3. You go to my-instance, run the magic 3 commands (chain, deploy, start)
4. Make all the changes you want: well, with the limitations of extensions, you can only add new files/folders, package.json tweaks, args.mjs files(??)
5. Run a script that will copy the changes (git magic?) to the create-eth repo (in a extensions folder with the name of the extension, e.g. extensions/eip)

Phase 2
This is what this PR is currently implementing (avoid having to push to GitHub, as you can test locally)

1. You can run yarn dev in the yarn cli -e eip --dev. The --dev is key here, so it'll create symlinks, instead of copying the files
2. You create an instance e.g. my-dev-instance (the eip is installed alongside)
3. Test (even tweak the symlinked files in my-dev-instance!)
4. You create a repo inside my-dev-instance and push to github. You could have all of your dev extensions there:
   • We could gitignore the extensions folder (maybe it should be called externalExtensions?)
   • You have all your external extensions there with their repo.

We could do this in different PRs. (super important also is too have a good guide for external extension developers.. and us :D)

This PR is not in a mergeable state, but pushed it to start the discussion. Would love for you guys to test it out and give feedback. Also about the workflow itself, and think if we can improve it.

[damianmarti]

I got this undefined/undefined message when creating the project:

✔ 🚀 Creating a new Scaffold-ETH 2 app in test-eip with the undefined/undefined extension

[damianmarti]

@carletex I tested it and it's working great!

What should I do if I want to change something in the Header? Should I change the header inside the new test project, test it, and then change the Header.tsx.args.mjs inside the extension, and finally create a new project to check that all is ok?

[technophile-04]

Thanks @carletex this is really nice !! And also works nicely!

---

Should I change the header inside the new test project, test it, and then change the Header.tsx.args.mjs inside the extension, and finally create a new project to check that all is ok?

yeah was thinking the same, but seems .args.mjs will be hard to figure out.

We could handle this in different PR but what we if watch the .args.mjs files ? So whenever the user changes something in extensions/eip/extensions/...args.mjs file we on the fly call the .template.mjs file get the output and throw it inside the my-instance ...Lol easier said then done, but will also think of some more alternatives

[carletex]

Thanks Damu & Shiv for testing and your feedback!

I got this undefined/undefined message when creating the project:
✔ 🚀 Creating a new Scaffold-ETH 2 app in test-eip with the undefined/undefined extension

I see that you figured this out, but what was the issue? So maybe we can hint that something is wrong & stop.

What should I do if I want to change something in the Header? Should I change the header inside the new test project, test it, and then change the Header.tsx.args.mjs inside the extension, and finally create a new project to check that all is ok?

Yes, it sounds ugly haha. Without this PR, you'd have to push it to your git repo to test it tho!! :/ So it's a step in the right direction haha

We could handle this in different PR but what we if watch the .args.mjs files ? So whenever the user changes something in extensions/eip/extensions/...args.mjs file we on the fly call the .template.mjs file get the output and throw it inside the my-instance ...Lol easier said then done, but will also think of some more alternatives

Yes, something like this will be great (But will also destroy your changes in the instance file haha). Maybe if we are in --dev mode, this CLI doesn't exist after creating the instance, and enters "watch" mode, looking for changes in your extensions args.mjs files? It could be another yarn script too, but the CLI have the context already (where the instance is located, which extension you are testing, etc), and the script could be ugly with params.

---

There are many things to figure out for the DX on extensions and I don't see a clean & easy path for it. But we can start merging PRs (that move us in the right direction) and see what we end with. The good thing is that these PRs shouldn't mess with the regular use of the tool (we need to make sure we are not breaking anything haha)

[rin-st]

I also tested and it's working great! Thanks!

[technophile-04]

I got this undefined/undefined message when creating the project:
✔ 🚀 Creating a new Scaffold-ETH 2 app in test-eip with the undefined/undefined extension

Even I get the same thing, but it does not exit the cli everything works fine.

I think the culprit is getArgumentFromExternalExtensionOption(options.externalExtension)

create-eth/src/main.ts

Line 34 in 99f6005

)}${options.externalExtension ? ` with the ${chalk.green.bold(getArgumentFromExternalExtensionOption(options.externalExtension))} extension` : ""}`,

Since when you do yarn cli -e eip --dev, the value of options.externalExtension is just "eip" instead of it being an object.

[carletex]

Update:

renamed extensions folder to externalExtensions & gitignored it & add it to eslint/rollup ignore to avoid warnings in the console while yarn dev.

Even I get the same thing, but it does not exit the cli everything works fine.

Oh shit, I'm getting the same lol I thought it was failing for Damu! But it's just the undefined thing. Will fix now!

[carletex]

✔ 🚀 Creating a new Scaffold-ETH 2 app in test-eip with the undefined/undefined extension

Fixed here e4be88a

[damianmarti]

Tested again and everything is working great!!

I think we already talked about this: when changing the file linked, the webserver watching the files does not reload it, so you have to always change the file inside the generated project if you want to see the changes live (maybe we can add this warning somewhere in the documentation later).

We can try to improve it later, but it's a great step forward to a nice extension dev workflow!!

[carletex]

I think we already talked about this: when changing the file linked, the webserver watching the files does not reload it, so you have to always change the file inside the generated project if you want to see the changes live (maybe we can add this warning somewhere in the documentation later).

Yes, definitely something weird going on there with symlinks haha.

[technophile-04]

We need to ignore externalExtensions in .prettierignore config as well:
https://github.com/scaffold-eth/create-eth/blob/main/.prettierignore#L8

It seem to format the files inside externalExtension on save

[carletex]

Done!

We need to ignore externalExtensions in .prettierignore config as well:

---

I'm also getting this. Since they are symlinked, I see the changes in the templates folder. Does it happen to you guys too? I think this shouldn't happen, right? (maybe a fix is not symlinking those special files?)

[technophile-04]

I'm also getting this. Since they are symlinked, I see the changes in the templates folder. Does it happen to you guys too? I think this shouldn't happen right (maybe not symlinking those special files?)

Yup yup happening to me too

[damianmarti]

I'm also getting this. Since they are symlinked, I see the changes in the templates folder. Does it happen to you guys too? I think this shouldn't happen right (maybe not symlinking those special files?)

Yup yup happening to me too

yes, me too!

[technophile-04]

Does it happen to you guys too? I think this shouldn't happen, right? (maybe a fix is not symlinking those special files?)

Ohh actually this is related to #45 (comment), previously were ignoring the package.json in copyOrLink func and then merging it below (While merging it doesn't create symlink) so that's why I think Dani had that logic, I that PR I unignored so pacakge.json was symlinked because of copyOrLink function

Just pushed a small change to find the above thing at 2ff78df

[technophile-04]

Tysm @carletex !! love the logic clean and short! added a small comment but rest of all looks nice!

Also just tested it throughly, Cloning https://github.com/scaffold-eth/create-eth-extensions inside externalExtensions dir :) and tinkring a bit, everything seems to work.

Also tested nicely normal things and extensions just to be sure we didn't break current version and it all also seems to work!

[carletex]

Fixed the small tweaks from Shiv + fixed the conflicts from main.

[technophile-04]

Just pushed a minor nitpck, renaming template to extensionName in validateExternalExtension at 1064ae8

Merging this!! Thanks Carlos this is really nice! and other for review!!