How to get Hudson to correctly build multiple modules changed by a single commit

Consider a Maven project with multiple interdependent modules: let's say, three jar modules A, B, and C, which are dependencies for a war module Z. I have a separate Hudson build for each of these modules, so that only modules that have changed are re-built.

My issue is that if I commit a changeset that changes both module A and module Z, Z may be built before A and fail, before A completes and triggers a rebuild of Z which now passes. Allowing builds to regularly fail for reasons to do with build order开发者_如何学Pythoning rather than "real" failures desensitizes us to real failures; we end up ignoring builds which have legitimately broken because we are used to assuming it will eventually flip back.

I have been managing this through the use of quiet periods, blocking when upstream builds are running, etc. But in practice, my build has more modules than the example I've given, many of which take a while to build and test. I also have a small horde of diligent developers making frequent commits.

This means my jar modules are constantly building, only rarely leaving a gap for my war module(s) to build. So the war doesn't build very frequently, meaning it takes a long time to find out when we've broken it, and also takes longer to identify which change broke it.

Also, the constant running of builds means that if I commit a change that touches jars A and B, the war file Z may be built once for jar A (which builds quickly), and then again for jar B (which takes longer). This makes it hard to understand the results of a given commit.

I've considered using the join plugin, but this appears to require all of the modules to build every time. Since I actually have quite a few jar modules, I really don't want to have to build them all every time, I only want to build the ones that have changed for a given commit.

Are there any better ways to handle this?

Thanks

This is always a difficult problem (and I've re-written this answer more than once!)

In terms of a technical solution, you want something that will wait for the build of several different jobs to be not running before it starts to run. If it's difficult to quantify, it's going to be difficult to put in place. I'll be very interested to see what technical solutions are suggested in this thread.

I guess you have to look at why your jobs are being run, and how often. If there's any code that requires unit testing in your WAR, could you move it out into it's own module? That way you can run only integration tests every hour/30 mins using the war and not worry about where and when the individual modules are at.

You may want to also look at what your modules contain. Do they ALL have to be modules? Can you perhaps reduce the fragmentation - it might help reduce the complexity of what you are attempting to schedule :)

I understand and applaud your efforts to get as much tested as soon as possible - but sometimes a smoke test is all you can do if there's a constant churn of code.

The approach we're now looking at is combining some Maven modules into single Hudson jobs, rather than having a one to one mapping of modules to jobs.

Specifically, if a war module's dependencies are fairly small and quick to build on their own, building them in the same job with the war ensures that all of the code from a single commit is built together, at least for that given war file.

This does result in duplication - we have multiple war files using the same jars, so the jars are essentially rebuilt for every war, rather than once only. But in practice, the jars are quick to build, and this makes the war files conceptually cleaner.

This would be less attractive if the jars took a while to build and test, since the combined jars + war job would then be quite long, giving us long feedback loops for problems within the jars. Getting the balance right is important.

So my takeaway: don't assume that one Hudson/Jenkins job per module is the best way to go, and don't be afraid to rebuild the same code in multiple jobs.