Two things I ran across today that I need to look further into.

http://latencytop.org/ - This shows what processes are waiting on and looks very interesting.

The other is Group Scheduling in Linux 2.6.24 and above. This allows you to put processes in a group so that one set of processes can share the same resources. For example, you can assign two different users 50% of the processor and one use could be running 50 compiles but should not affect the performance of the second user.