Rebar3: Pre-Alpha State.

Fred Hebert mononcqc at ferd.ca
Sun Nov 16 09:59:23 EST 2014


Hi everyone,

Some of you may have noticed a new repository under the rebar org in github:
https://github.com/rebar/rebar3

We felt like we should be writing to explain why this repo is there, what's its
state, and what we are planning for it.

First of all, the cause behind this repository is the series of bugs or
behaviours proper to the existing rebar versions. Rebar has been pretty great
for us and the Erlang community as a whole, and still is. However, where it
used to be an escape hatch to move away from custom makefiles and limitations
of Emakefiles and to provide more usability, it has started to show its
weaknesses for some needs and functionality.

Specifically, we're thinking of issues such as:

- https://github.com/rebar/rebar/issues/86 - deleting a tagged dep and re-fetching yields inconsistent results
- https://github.com/rebar/rebar/issues/159 - update-deps stopped working with pre_hooks
- https://github.com/rebar/rebar/issues/170 - update-deps breaks on some transient dependencies
- https://github.com/rebar/rebar/issues/178 - infinite loop in dependencies
- https://github.com/rebar/rebar/issues/239 - update-deps isn't recursive
- https://github.com/rebar/rebar/pull/349 - deps dir detection for coverspec files

Along with many issues we've had in the past regarding:

- Recursive application of functions (or the opposite of it)
- Lack of ways to express whether dependencies were test dependencies,
  actual dependencies, or plugins
- Conflicts when running multiple hooks on similar commands
- Difficulty to support all the different features rebar has by us
  maintainers when we don't use them nor have projects to test them with
- Other stuff.

We shouldn't pass under silence the good aspects of current rebar though,
including but not limited to:

- Portability
- Convenience and mindshare (thanks for all contributions from everyone involved)
- Extensive test suite
- Breadth of features
- Lots of work and extensive reference documentation (have you tried
  `rebar help` over the last year?)
- And so on.

Still, the problems mentioned before have been felt time and time again, and
Tristan and I have had a hard time keeping up or even tackling most issues
properly. The deps handling (specifically the support for more deterministic
results when handling transitive dependencies) and difficulty to maintain all
the supported features given how they waive their way through the execution
path for us encouraged us to try and prototype what we thought could be a
decent model to tackle this.

That's the raison d'être behind rebar3. We wanted a testing ground for fixes
for these ideas. A kind of demo that could let us try and figure out how things
*should* work to be easier to use, without being tied down by keeping backwards
compatibility, which is frankly challenging.

Rebar3's been mostly worked on by Tristan so far, and he's done fantastic work
on it. Some of us have done little bits of work to try and help him where we
could, but it's mostly his work (notwhistanding the existing code bases he's
used, obviously). Right now, though, rebar3 is still at a pre-alpha level.

We wanted to share the work, to get people to try it and comment on it, and to
contribute if they feel like doing so. Here's what was introduced in rebar3:

1. More Deterministic dependency handling

The new dependency fetching model works on the idea that the top-level is
right. When transitive dependencies are downloaded, and that many applications
in the system depend on the same one with many versions, the topmost instance
of it in the dependency tree is kept. As soon as the project is built a lock
file is created and can be committed to make sure subsequent builds of the app
will yield similar results.

Rebar3 does not do version matching. This turns out to often be cumbersome, and
only works when/if we assume the vast majority of the community uses the same
kind of versioning scheme. Currently, most people have github repo, countless
projects are configured to fetch stuff from 'master' branches, and things would
just break too hard if we were to rely on versions. That's just the state of
the community today.

What we do instead is ref matching. On compile, rebar3 checks  if all your deps
are the same branch/tag/ref listed in the lock or config file if there is no
lock file. Moreover, deps are only recompiled if they are refetched. Meaning,
don't make changes to source in _deps/... because it won't be compiled.

In any case, this should fix issues with updating and getting deps.

2. Faster fetching (or at least, we hope)

Common dependencies now support working from a package index
(http://packages.rebar3.org/). These packages are terser to use in
configuration files. Only {AppName, Version} should be required. The advantage
they have is that they will be pre-compiled (regular .beam and .app files
only), and their dependency tree will have been pre-resolved already.

The feature is still highly experimental, all the libs haven't been tested to
show they properly resolve. Things should improve over time.

3. Command requirements are well-defined

Rebar3 is aware that running 'compile' requires 'get-deps' to have run before.
There should no longer be a need to chain up commands in order to go 'get-deps
compile test', and commands no longer 'cd' into the directories they act on.

4. Test dependencies can be specified as such

The {test_deps, [...]} tuple works similarly as {deps, [...]}, but only test
commands should require test_deps to be built as dependencies. This should,
over time (existing libraries will still suffer from this) avoid many of the
conflicts that may have existed when trying to build apps just because they
relied on different versions of meck or proper.

5. Rebar3 knows about your apps vs. deps

When handling code paths, internally, rebar3 knows that some apps are your
'project apps', and others are your dependencies, and should be able to build
on that to provide smarter handling of activities like recompiling and whatnot.

Additionally, rebar3 will look into the _checkout/ directory to look for
dependencies and symlinks to dependencies you are specifically working on to
know when to override existing rules and avoid trying to crush modifications
you are making, and to recompile them when required.

6. New plugins interface and handling

Internally, rebar3 uses the same interface as plugins. A partial tutorial
exists at http://www.rebar3.org/static/plugins.html

Plugins can be specified in a {plugins, [...]} tuple on a per-project basis,
similarly to dependencies and will be fetched before compiling a project.

Because the interface is the same between what we used internally to build
rebar3 and what people will be able to contribute, plugins can mor accurately
specify what they depend on in terms of execution, can make the distinction
between user code and dependencies, and so on. On top of this, it should be
simpler in the future to either provide some plugin index, or to graduate them
into the main rebar3 repo, and users will more easily be able to work around
defects or test new alternative implementations that way.

7. Templates now allow for global configuration

This one would arguably have been trivial to add to rebar 2.x. Templates were
otherwise made a bit dumber, but now support some self-documenting features.
See the tutorial: http://www.rebar3.org/static/templates.html

8. Relx integration

We've had lots of issues dealing with the complexity of reltool in the past,
and we believe Relx is the future of release handling in Erlang. Releases are
now handled with relx for this reason.

9. Brought back Dialyzer checks

The name says it all

--

While you should be able to build projects with rebar3 (even if they use rebar2
projects as dependencies), given not everything is ready yet, we have broken
lots of stuff. Some of it includes:

- Reltool support for releases
- Support for most non-Erlang compilers
- Templates (mustache's gone, it's pretty much only ErlyDTL now)
- We dropped tests, which will be a huge freaking pile of work to bring back up
  to speed if we keep going that way
- quickcheck support
- Plenty of edge cases

There's a lot more we dropped, or rather, haven't had time to build back in yet.

Our hope, to some extent, is that the community would help build these back as
plugins that could be graduated in over time into the main rebar. In some ways,
we'd probably like to be able to have rebar bootstrap itself with these plugins
in, so that they might be maintained independently from core rebar, while still
being part of usual releases. I think this could make overerall maintenance
swifter and more efficient, with better definitions of ownership.

We're eager to get your feedback!

Regards,
Tristan and Fred.



More information about the rebar mailing list