Tusk Discussions

cdorn’s concept of an overlay

You create an overlay with tusk to instruct tusk to load a specific package (directy or via a catalog) from a different location instead of the default one it wants to it allows you to splice in sources that you can work on

kriskowal and cdorn Oct 13 2009

<kriskowal> ah, sure
 what's your driving use case?
<cdorn> kriskowal: thanks for outlining all the commands
 they depict pretty much the same I have been working towards
<kriskowal> i presume we just need to bridge the gap
 it would be good to know what you're goals are though
<cdorn> yeah, I think we have less of a gap than you may think
<kriskowal> probably
<cdorn> my rough goals/assumptions are:
 - seas and projects into which you install developer tools for the purpose of building packages
<kriskowal> define "package" in this context
<cdorn> seas can be frozen and reheated (i.e. you can checkout complete developer environments consistently)
 package is a set of code (/packages/*) that is to be used as a component in a program
 say you are building jquery
--> tschaub ([email protected]) has joined #narwhal
<cdorn> jquery would be the package and in the sea you also have test applications, demos, docs etc...
 you then put the jquery package into a catalog to use it as a ependency in another sea
 when someone else uses the package and wants to make changes to it
 they checkout the jquery development sea and tusk links the jquery code in the program to the dev sea etc...
<kriskowal> one sec.  parsing
 so, the use case here: someone has a project (with its own sea), so they go into that sea's packages and clone the jquery package (which can be a sea in its own right, but not in this case)
<cdorn> you can have seas for building gae apps or xulrunner aps that come with a ctalog of commonly used tools
 right
<kriskowal> so with the new scheme packages can have their own catalogs
<cdorn> a sea is simply a special package
 only if a package is a sea - only seas can have catalogs
<kriskowal> but there can only be one active sea…so i'm a little mixed up
<cdorn> and the sea catalog are tools added to the sea catalog for the purpose of developing the packages in the sea
 no brianer install of dev tools
<nrstott> with all due respect cdorn, this sounds immensely more complicated than gems or easy_install ;0
<cdorn> you "work" on one sea at a time
 I have multiple terminal tabs open
<kriskowal> k
<cdorn> tusk shows me which sea is active - works real nice
 nrstott: it's all unter the hood and optional - the basic layout is actually quite simple
<kriskowal> yeah, i've been using echo $SEA for that
<cdorn> all this functionality comes down to a few config files in the right places
<kriskowal> so we have a jquery/catalog.json?
<nrstott> cdorn: sounds good then.  as long as the basic use case is easy and powerful enough for a broad range of cases
<kriskowal> that catalog contains the refs for packages that may or may not be in the universal catalog
 in any case, if you activate the jquery sea, tusk will implicitly know how to install these packages?
<cdorn> right
<kriskowal> by virtue of what axiom?
<cdorn> for distribution you then put the jquery package into a proper catalog for distribution
 I am thinking that when you activate the jquery sea it looks for packages to install in the sea catalog first
 and then in your general catalogs
 if you are working in the jqueery sea you are only concerned with building jquery
<kriskowal> in present tusk, the default catalog is {sea}/.tusk/catalog.json
<cdorn> so you have no real need to install external packages unless you intend to add new dev tools
<kriskowal> which has the advantage of users being able to edit it without checking it in
<cdorn> sea catalogs live along package.json and are intended ot be checked in
<kriskowal> and if {sea}/.tusk/catalog.json does not exist, it gets copied, not from {sea}/catalog.json, but from the universal catalog url
<cdorn> they record which tools are intended - and confiregured/pre-integrated to work in the dev sea
<kriskowal> so, do you check for {sea}/.tusk/catalog.json or just presume that {sea}/catalog.json exists
 if there is no sea/catalog.json, what happens?
<cdorn> I presume {sea}/catalog.json exists (if it is a sea) - it also record the packages in the sea
 for the purpose of inclusion into other projects
 it's a simple package
<kriskowal> so, you have to opt-into a repository if you create a sea
<cdorn> can, don't have to
<kriskowal> how do you make default=easy?
<cdorn> you cna have local only seas and refer to the sea catalog using file://...
 default=easy?
<kriskowal> yeah, the simple case needs to be easy.
 like, you create a sea, and install some packages you've heard about
 e.g., this should work: tusk init; source bin/activate; tusk install jquery
<cdorn> yup, it will
<kriskowal> how's that work?
<cdorn> given that you have added a catalog that contains jquery as a package (or we ship that catalog by default)
 jquery gets installed into the sea
<kriskowal> and furthermore expect all seaworthy packages to ship a default
<cdorn> and recorded in package.local.json
<kriskowal> i posit that all packages should be seaworthy by default, and that sequence of commands should still work.
 on that note, i'm thinking that package.local.json ought to become .tusk/package.json
<cdorn> I was hoping to keep all files that are local as *.local.*
<kriskowal> anyhow… is there a clear distinction in the new tusk between packages that are loaded with the require(id, pkg) syntax and those that are available in an ambient sense?
<cdorn> but .tusk/ would still work
 hmm
 not really
<kriskowal> are all packages loaded with the two-param?
<cdorn> all you have to do is add a package as a dependency and you can load modules from it
 yes
 you specify a package to set the context of where to load the module from
<kriskowal> ah
<cdorn> if no package is specified it sticks with the last package context
<kriskowal> what's a package context
 oh
 yeah.
<cdorn> depenencies in a package.json form the context essentially
<kriskowal> uh oh.
<cdorn> so you specify the package alias you mapped
 sorry wrong wording
<kriskowal> no, i follow
 so this has major ramifications for how modules are loaded and whether lib/packages.js is relevant
 is the process of downloading packages hooked into the module loader at this point?
<cdorn> right, I have not begun to think of the big picture yet - I just added the features on top without breaking existing use-cases
 no, module downloading is all in tusk right now
 but we can build a dynamic loader and use it for the module loader and tusk
<kriskowal> and that's managed by "reheating" i presume?
<cdorn> they will use the same "locator" json
 right, at the moment it is: tusk package install -f
 but reheat makes more sense
 you add a dpenedency: tusk package add ...
 ad then tusk reheat
 you can add the dependency to the sea package or any sub-package
<kriskowal> so, you download a seaworthy package, source bin/activate it, then tusk reheat to bring in its transitive dependencies
<cdorn> I call them dee-sea packages
 right
<kriskowal> so, ihab and i are only partially in agreement about packages as you've implemented them.
<cdorn> and if you have a workspace management tool installed you can get your old paclage.local.json back too
<kriskowal> i'm at this point willing to entertain the two-param require, but not willing to use it exclusively
<cdorn> right, I only use it to switch package contexts
 once you are in a package you cna use local/relative includes
 you could even set up a search path for each package that looks outside of the package
 that is how system modules are found right now
 it seems to work well and I have several intricate use cases in narwhalrunner
 it does a lot of context switching
<kriskowal> how do you manage contexts.
<cdorn> they are implied
<kriskowal> it's my impression that with the two-arg require, there's no need to support the module search path
<cdorn> right, but that is where the current implementation is a bit loose
<kriskowal> hm.
<cdorn> it looks in the package first (if it can determine the package) and then falls back to the search path
<kriskowal> which would be best modeled by cascading loaders.
<cdorn> if we can set some ground rules down the package loader can be dramatically simplified
 I can run it against a bunch of tests as we refactor
<kriskowal> that is, each package would have a loader.  failing to find a module in one loader would cause it to defer to a parent loader, in this case, the system loader
<cdorn> yeah, I would like to wrap this into loaders properly
 yes
<kriskowal> but, ihab is pretty adamantly against that approach
<nrstott> what's his argument against it?
<cdorn> too much overhead?
<kriskowal> ah, well.  ihab wants a brave new world where there is no universal catalog
<cdorn> universal catalog?
<kriskowal> where all packages contain their own catalog
<cdorn> I don;t see the relevance to this
 cascading loaders will still work with that
<kriskowal> he wants a system where no-sys-admin ever needs to get involved, where there is no "tusk install <package>"
 no, he wants nothing to be implicitly in scope
<cdorn> that is possible with my approach as long as you have an application-server-tusk installed
<kriskowal> if you want the file module, and you're not in the package that provides it, he wants that package to be reference explicitly.  require("file", "narwhal")
 tell me more about narwhalrunner.  this is the xulrunner engine for narwhal, right?
<nrstott> kriskowal: ihabs 'brave new world' sounds terrible
<cdorn> yeah, you specify the meta data for an app: http://github.com/cadorn/narwhalrunner/blob/master/packages/test-application/package.json
<nrstott> I would absolutely hate to work in an environment where a 2-arg require was required
<cdorn> and can then compile it into an xulrunner app
<nrstott> or rather a 2-arg require for your example
<cdorn> the toolkit abstracts all the conext management and loading
 it essentially takes away all the uglyness in writing xulrunner apps and standardizes it to more of a builder approach
<kriskowal> cdorn: how does that work
<cdorn> the compiling?
<kriskowal> context management
<cdorn> well, for xulrunner apps you have an application or extension context
<kriskowal> i'm somewhat familiar with the ugliness of xulrunner development.  i wrote a template application called stantz some years ago.
<cdorn> ok, let me restart
<BryanWB> kriskowal: cdorn: i am a bit late to this party, but i just put most of your combined commands on this wiki page http://wiki.github.com/tlrobinson/narwhal/tusk-specification
<cdorn> it takes all the commonalities between extensions and applications on xulrunner and firefox and abstracts everything to be able to load commonjs modules
 when a new window is created you narwahlize it after which point you cna load modules from your package
 it also abstracts DOM and CSS id's by prefixing them to avoid naming conficts etc...
<kriskowal> m, yeah
<cdorn> BryanWB: thanks, I'll add my comments as I go along and make these work
<cdorn> kriskowal: the key for me is a predictable dev environment you can freeze and reheat and that any other contributor can install themselves and work within - that eliminates so much "setup overhead"
<kriskowal> yes.  definitely an important feature.
 kevin and i talked about that extensively early on.
<cdorn> and the ability to overlay your own changes over any dependency and contribute them back to the original project easily
<kriskowal> we were thinking of using another system like gems, but i wanted a pure-js toolchain that could be installed alongside narwhal, so i went ahead and did the minimal subset, deliberately baking virtual envs in and postponing versions management and freeze/reheat
<cdorn> tusk-catalog is build around this premise - that even the dev environment is completely cataloged
 I have not spent much time on versioning other than "latest" and "source" (which I hope to get to this week)
<kriskowal> it's my hope to keep the implicit dependencies decoupled from the explicit dependencies
<cdorn> what's the difference?
<kriskowal> it's presently possible in most languages, and narwhal present, to depend on a package without explicitly calling it out in package.json
<cdorn> right, the package loader supports this still
<kriskowal> so python has pip (a little vague on the details or accuracy here) that lets you look at your virtualenv (==sea) and write down all of the versions of every package installed, so you can reheat them somewhere else
 it also lets you have parallel seas with different versions of the same packages "activated"
<cdorn> that is important for testing
 I'll see how I can work that in
<kriskowal> one scenario i'd like to support is where there's a "system" sea, owned by the sysadmin
--> isaacs ([email protected]) has joined #narwhal
<kriskowal> so the system sea can contain many versions of a package, and have certain versions activated.  meanwhile, some other sea that's using the system narwhal engine would be able to find and install particular versions by linking the system tree
<cdorn> yeah, should not be too difficult to get working
<kriskowal> i was thinking that would be accomplished by having a "{sea}/versions/{name}/{version}" as a repository for package versions that aren't necessarily activated.  they would then be symlinked into "{sea}/packages/{name}" to activate
 or copied, if the os is stupid
<cdorn> that is where I thought "revision" would come in handy
<kriskowal> but, the user sea would mirror that schema, so even non-priv users can have a read/write packages and versions tree, but inherit the system ones if desired.
 is there any semantic distinction between "version" and "revision"?
<cdorn> the system sea exposes activated package versions via a catalog with revison such as "latest" or "3.4"
 version to me means a specific version or set of code, revision to me could mean version, "latest", git commit ID ...
<kriskowal> in the apt world, tags include "stable", "dev", "testing", &c.
<cdorn> revision does not imply spability
 and it is not intended to
<kriskowal> would "3.4" and "revision" exist in separate name spaces?
<cdorn> "3.5" and "latest"? - yes
 they do
 and they can be symliked
<kriskowal> alright.  bottom-line is that i need to read your code thoroughly
<cdorn> it's still a bit of a mess
 the "planet" concept only exists for testing
 so you can init tusk for an isolated planet
<kriskowal> i'm not sold on the "planet" stuff
<cdorn> It's more internal than anything to identify the system-wide tusk install
 will not be exposed to the CLI
<kriskowal> ah.  so, the planet is the narwhal sea
 one case that i have to keep in mind is that sometimes narwhal is installed in the sea as a package.
 one of the driving use cases is creating a GAE war/WEB-INF sea
<cdorn> I have not dealt with installing or relocating narwhal - I am going to nail down everything else first before I extend it to support that
<kriskowal> which will be tricky
<cdorn> not too bad - tusk now runs from a sandbox so it can load itself and then manipulate a sea in any way it sees fit
 that was needed for testing
<tlrobinson> planets?
<cdorn> a planet is the keeper of many seas
 on a system
 kriskowal: so overall we are close enough on the same page?
 still a lot of details to work out
<kriskowal> i think we are a hard refactor away from mergability
 but, that too i think we're in agreement about
 i need to read the code and verify that the big picture works
 and we need to implement the CLI
<cdorn> you will see what is "new" and what is older
 there are a lot of functions in need of deprecation
 I am going to add all my workflow tests and then refactor the CLI to bring it in line with what you have outlined
<kriskowal> we only need to maintain backward compatibility with master.
 if there's anything intermediate in there, skip deprecation and ax it.
 we also need to make sure that there's a migration path, and we need to test it
<cdorn> right, I just need to finish all the tests to ensure it's not needed :)
 I have been keeping backwards compatibility for the most part
 I'll revisit when I have it working the way it needs to
 does the package loader still pass all your tests?
 the package loader is probably something I'll need help with - we need to define what should be possible and how it should be implemented
<kriskowal> when i switch over, tusk install doesn't exist and tusk package install <name> does not work
<cdorn> right
<kriskowal> i have to comb the package loader changes.
<cdorn> scrambling to get that working again ...
<kriskowal> i do have opinions about the argument order, which i'll prolly just have to change.
<cdorn> I will migrate the existing CLI t the one you outlined
<kriskowal> i'm not sure that a package sub-tree is necessary
 tusk is first and foremost a package management tool.
<cdorn> package sub-tree?
 oh in the CLI
 yes
 I agree
 the commands were organized for clarity during development
 I know they need work :)
<kriskowal> anyhow, all hands meeting. pizza must be consumed with our east-coast brethren
 ttyl