How to debug the performance of a wrong setup of a build machine?

We have to setup new build environments regularily, and the process seems not so simple. Today I have got a new build machine, and the first Maven build was so slow, that I wanted to clarify why the performance was so bad. But how to do that?

Our context is:

• We use multiple build machines, each project gets its own.
• Each build machine has a similar setup, so that projects can start immediately and don't have to configure a lot.
• We have the following tools preconfigured:
  • Hudson (currently 2.1.1, but will change)
  • Artifactory 2.3.3.1
  • Sonar
  • Hudson, Artifactory and Sonar have their own Tomcat configured
  • Maven 2.2.1 and Maven 3.0.3 (with no user configuration, only the installation has a settings.xml)
  • Ant 1.7.1 and Ant 1.8.2 (not relevant here)
  • Subversion 1.6 client

All tools should work together, especially the repository chain should be:

1. Build machine Maven repository
2. Build machine Artifactory
3. Central company Artifactory (is working as mirror and cache for the world)
4. Maven central (and other repository)

So when the Maven build needs a dependency resolved, it will be first looked-up in the local Maven repo, from there in the local Artifactory repo, then in the central Artifactory repo and only then on the internet.

We normally have to use proxies to connect to the internet, we don't need it in our intranet.

The first build (Maven Hello World) was built in around 45 minutes. In that time, all bootstrapping was happening, but I would have thought by using our chain of repositories (where the central repository is well filled), the build would be much faster. So I think the focus of the debugging will be the network, the local build is not the problem. So configuration and interaction of Maven and Artifactory is under consideration.

How do you debug such an environment? I have access to the build machine (as sudo) and to the central repository, but I do not know how to start, 开发者_StackOverflowwhat to prove, where to look. So what is your experience, what are the tips and tricks you would like to share?

Here are a few things I have done up to now. If you have additional advice, you are welcome!

I suspected the chain of repositories to be the source of evil, so I addressed that first. The reasons are:

• The real build on the local machine (of a hello world program) may differ in milliseconds, but not minutes.
• Network makes a difference, so attack that first.

The chain of repository is interesting, if something is not found locally. Here are the steps to ensure that that is the case:

• For Maven, delete the contents of the local cache. If the local cache is filled, you don't know if a resource if found in the local cache or elsewhere. (Do that at least at the end, if everything else is working again.)
• For Artifactory, find that cache as well, and clean it by deleting its contents. It is only a cache, so it will be filled a new.
• If you use a clever browser for measuring the lookup, ensure that what you asked for is not in the cache of the browser.
• Else use a tool like wget to ask for a resource.
• Try to minimize the sources for failure. So try to divide the long distance of your lookup in smaller segments that you control.
• Don't use Maven for doing the lookup, start first with the Artifactory repository (only), and later then with Maven.

This led to the following tests I wanted to do. Every time I ensured that the previous prerequisits were met:

1. Ask for https://<my-project-artifactory>/repo/<my-pom>. Expectation:

   • Local lookup will fail, so has to find the resource in a remote repository in the central company Artifactory.
   • Possible effects could come from proxy, artifactory lookup.

   Result: Lookup for a simple POM needed ~ 30 seconds. That is too much.

2. Remove the proxy. With wget, there is an option --no-proxy which does just that. Expection:

   • Faster lookup.

   Result: No change at all, so proxy was not the reason.

3. Ask for https://<my-project-artifactory>/libs-snapshots-company/<my-pom>. So change the virtual repository to a real remote repository. Expectation:

   • Artifactory knows where to do the lookup, so it will be much faster.

   Result: POM was found immediately, so the 30 seconds are Artifactory doing lookup. But what could be the reason for that?

4. Removed in Artifactory all remote and virtual repositories (only left our companies ones and the cached Maven central). But use again https://<my-project-artifactory>/repo/<my-pom>. Expectation:

   • Artifactory will find the repository much faster.

   Result: POM came in an instant, not measurable.

5. I was then courageous and just started the build (with empty cache locally). The build needed then 5 seconds (instead of 15 minutes the same morning).

So I think I have now better understood what can go wrong, a lot of questions are remaining. Please add your ideas as answers, you will get reputation for them!