Managing big multi-module projects in Subversion/Maven/Hudson

We have a rather big java application which consists of tons of maven artifacts. Each subsystem has its own folder in the subversion tree with trunk/, branches/ and tags/ subfolders and hosts multiple artifacts. The hierarchy of Maven POMs has multiple levels. Even dependency managemnt is split up into different modules. The application is deployed as multiple WARs and other Zip/JAR stuff.

In the process of building and deploying it, I often have to view the application as one unit. And I came across a bunch of difficulties I wanted to share here.

1. Subversion Layout: It starts with the fact that I can not just checkout the whole code base with one command. It is simply too big and would take too long when I start at the top level and checkout all the branches from all subprojects. I guess reducing the number of artifacts is currently not really an option since it would cause too much fundamental changes. But may be a different subversion layout with aggregated branches folders would indeed help. One workaround I tried is to make a separate folder with lots of subversion externals, that gather together the right subfolders. I can then for example build the whole thing with one maven command from a top-level multi-module artifact.

2. Releasing: A release is done over the whole code base. First I make a huge dependency graph, where I use a custom tool written specifically for the project, since it is not possible to do this with Maven directly. Then I have to incrementally release one layer of artifacts in the dependency tree, starting at the bottom with artifacts that do not have any further dependencies, build them, if successful adapt the parent POMs with the new versions, and move on to the next layer. Sometimes I feel like doing all the versioning work twice, once in subversion and once again in all the POMs. The release plugin does not seem to be able to also adapt dependency management automatically.

3. Hudson: In Hudson we have like 20 jobs for all the parent artifacts. And for each artifact there are 3 active branches. This m开发者_如何学Pythonakes like 60 jobs. Everytime a branch changes it causes a lot of mouse clicking work in the browser. At least the automatic build triggering of dependent jobs is working. But then again this causes problems if a module is split up into a compile job and a report job. Usually compiling is done a lot more frequent that generating reports, and the latter should not slow down the former. But then whenever the report job runs, it also triggers all the compile jobs. Hudson does not seem to be able to only trigger dependent jobs if the artifact is really deployed. I also tried to make one huge multi-module build for the whole project, but the incremental build feature does not work correctly. And without it builds are taking just too long. But building dependent artifacts is required, since you can never be sure what a change it one artifact causes.

4. Aggregate Reports: I did not yet figure out how to generate aggregate reports for unit tests, test coverage, checkstyle and others for each level of parent POMs. Some plugins even do not support a aggregate option. For others I have to do the build again for each level in the hierarchy. Maven site reports are okay, but they seem to be fairly static and only of limited use when browsing them. Maybe the better option here would be to use something like Sonar Source or some other source code analysis tool. Also the integration of these reports into Hudson seems not ideal to me.

From my perspective I would like to have just one big repository, where I can checkout one directory, build everything from source, make one tag/version for a release, have no dependencies on any external repositories and configure one job in Hudson. But this is maybe far away from what the different developer teams need. It just somehow seems to be difficult to find a manageable way for both.

I don't know if this is a simplistic answer, but I have a similar project (codewise): 2 big applications, one with about 10 modules and one with 15 modules. We have is everything in one subversion repository (e.g. svn/trunk/app1 and svn/trunk/app02). In the root folder (svn/trunk) we have a very simple pom that define the version and both applications as submodules.

With this set up we can do most of what you mentioned. Even the separation from each development team, as each team commits to a different folder / project. We even store other artifacts, such as documentation and design (svn/trunk/documentation), so we can track what documentation we had for a specific version.

I know that this doesn't answer your first concern (you cannot checkout all the code base), but why do you need to "checkout all the branches from all subprojects"? That doesn't sound right. Do you have a fast connection to the svn server?

Note: We don't generate combined reports, as we're more interested in having a report per application, as they're quite different.