I'm going to get 8-10 machines to act as drones for the test. They are old, retired desktops, but they can handle running 10 threads, with 10 connections on each thread, for a simulated load of 100 users each. That will definitely stress the system more than the user community.

As part of my setup/experimenting, I used the load balance administration program to stop one of the servers in the web farm. The other server picked up the load after a few seconds.  Out of 20,000 socket connections, only 100 failed! I expect that the ratio will be higher in the event of a failure, since the load balancer detection process has some latency. I did a similar "shut down" in the SQL Cluster, and the delay was about 45 seconds.  This will certainly impact the users more, but at least the backup machine will handle the load while I fix things.

So far, I'm very encouraged about our hardware.  I hope to get more testing done over the weekend.