too many connections made the broker stall

Encountered a problem when too many sockets were open and put in "CLOSE_WAIT" status by the broker.

gridmsg001 was rebooted as quick fix, but should be found a different way to keep it up!

http://support.bea.com/application_content/product_portlets/support_patterns/wls/TooManyOpenFilesPattern.html

Update: problem related to excessive number of connections being open at indiana.edu. track of the error seen through checking the log errors. Indiana corrected their algorithm but still todo, find a way to block connections otherwise, a typical Denial of service may occur :S

Using the analysis scripts:

Example of usage:

python rNetworkAnalysis.py fastConsumerSummary.csv comparingDataBufferDataDumping/Consumer*

python timeSeriesAnalysis.py fastConsumerTimes.csv > fastConsumerTimeSeries.csv

python mAnalysis.py fastConsumerTimeSeries.csv ( results goes to temp.csv )

python lagAnalysis.py fastConsumerTimeSeries.csv > fastConsumerLag.csv