One of the biggest pain points of a large multi package project like Interledger is making changes across package boundaries. It’s messy, hard to track, and testing is very complicated. Versioning is tough, and dependencies can vary significantly.
I’m curious if others here would see value in organizing the various js Interledger packages into a multi-package repo i.e. a monorepo. Orchestrating a monorepo is not simple but fortunately it is easy due to many in the js community having built fantastic tools to deal with them.
Specifically I would propose utilizing Lerna with Yarn Workspaces to manage the packages. This combination would allow Lerna to take care of versioning and publishing across packages while letting Yarn Workspaces elegantly manage the dependencies. I believe Yarn is the superior package manager choice here because of its focus on reproducible builds and its Workspace feature which allows a monorepo to hoist every dependency to the root node_modules, cross-link package dependencies and only require a single lockfile. Bootstrapping the entire project can be done with a yarn install.
Examples of projects that are currently using this Lerna/Yarn Workspaces combination are Babel (which Lerna was originally written to manage), React, Angular, Ember, Meteor, Jest, Gatsby and Vue-cli.
I think the advantages a single multi-package repo could bring to interledgerjs would help tremendously with ease of contribution, development cycle quality of life and overall productivity.
Do you think the community would be open to this project?
I support this 100%. Currently trying to work on the code base potentially requires you adding PR’s to multiple repo’s and waiting for this to be merged before you can actually affect this changes you want.
The real question is what are the boundaries of this monorepo? Is it just everything to run a connector+dependencies? Or does it include pretty much everything in interledger-js? I think the best would be to have a single repo for a core connector where it is easily extendable. And then have another community organization for contributions that can extend the monorepo/core functionality.
As the person that originally pushed for separate repos, I’ll add some context. The reason I was against a monorepo back then was because I specifically didn’t want it to be very easy to make breaking changes across packages. At the time, one of our biggest pain points was that we kept tweaking the protocol so a live network could never get off the ground. Now that there is a live network and many of the core protocols are final that’s not a concern anymore.
+1 to keeping the boundary limited. However, I think the monorepo should include ILP packets, the connector, and STREAM. This seems like the minimally complete set of ILP repos (a packet format, a router, and a transport for packets), leaving plugins to a community contributions organization.
I put together an example mono repo for interledger to see what it would look like:
Things I learned:
Although the core set of packages isn’t that large, they collectively import a host of small ilp related packages, such as ilp-store-wrapper, ilp-plugin-mirror, ilp-logger. Forcing it all into a monrepo would make it really easy to start deprecating some of these and generally tidying up.
The way packages handle things like linting, typescript building, test running, npm publishing are all different. Tidying this up would allow nice monorepo things like a single build or test command.
Many packages are pulling in old versions of other ilp packages. Updates are not being fully propagated.
A mono repo would make it easier to add integration tests across many components and catch breaking PRs.
A single GitHub repo could be a nice focus for PRs and issues. Many packages have old forgotten PRs.
I’m continuing on this and have put the following repo together as a starting point.
I think we should pull packages into the mono-repo one by one and not just dump a lot of stuff in from the start. That way we can start to deprecate some of the dead wood.
Please read the README for some motivations for the design and let me know what you think or if you have issues running/using it.
This is based on reviewing around 20 example Typescript mono-repos and trying to take some of the best from all of them.
I propose that we deprecate existing modules and move to using these with the @interledger scope. This allows us to go all in with Typescript and shed some of the hacks we’ve had to make to accommodate the evolution of the various module systems.
Users can migrate to the new modules in their own time, the new modules will always only depend on other new modules.