Troubleshooting slow web page load times, how to do it/get better?

I'm the primary develope开发者_StackOverflow社区r at a web firm, but often end up doing some sysadmin stuff, and was wondering what resources are available for learning how to troubleshoot slow page load times. My experience with sysadmin tools is almost none. I'm relatively proficient at the Linux/Unix command line, but have never used any type of packet tracking software and only know the basics of using dig for ip resolves. My experience with apache and mysql is mostly limited to configuring initial setup and then using them.

Are there any good books or web sites that cover the topics needed for accurately diagnosing website performance/bottlenecks and if so what are they, or is the gamut of technologies used to large and experience/time with using the technologies typically how people get good at this stuff?

Overall, there's no substitute for experience. A concept as broad as "slow web page load time" could be hitting a bottleneck in any number of different places:

1. Client is slow to resolve IP from domain.
2. Network between client and domain is slow or congested.
3. Server is slow to respond to request.
4. Requested page is large.
5. Requested resources embedded in page are large.
6. Page contains server-side code that requires significant processing.
7. Database is slow to respond.
8. Page manipulates a lot of data before responding.
9. Rendered page on client contains a lot of code and runs slowly.
10. etc.

For any given page, it's a matter of know where the bottlenecks could be and determining what that bottleneck is in order to address it. Having a full and complete understanding of everything that goes on from end to end in "loading a page" is essential. Identifying patterns of slow load times across multiple disparate requests will help narrow down the potential bottlenecks. etc.

It's very much a case-by-case scenario.

Have you diagnosed exactly where the problems are? Slow load times could be caused by anything, such as:

1. too much going on in the dom/js
   a) not caching js or other resources on the client
   b) not minifying/compressing resources
   c) making too many requests with ajax/doing silly things in the browser like redrawing dom that doesn't need it.

2. too much going on in the server
   a) no cache on db tables, no indexes
   b) not handling long running tasks asynchronously
   c) improperly configured proxies/apache servers

3. network issues -- wish I knew more about this.

Step 1 is always to figure out where the worst slowdown is. Do some metrics on the server to make sure it is doing easy stuff fast. And hard stuff reasonably fast. Look in the browser to see how long loading resources is taking. Look in the chrome/firebug profilers to see how much time the javascript is taking to run.

You will probably find a bunch of things that could be improved. Prioritize and address the issues...