Unexpected load peaks during test

Oct 1, 2008 at 11:53 AM

HI

I ran a test for 3 hours, normal distribution think time and a step user load pattern which ramps up to the maximum 2000 users after 34 minutes.  Response times and average test times generally appear good through the test ( sub 1 second, however the 95% test times are not good and there are high peaks on occasions in the response times.  I think the cause of the high 95% test times are an abnormally large amount of load being generated against the system under test – however this only occurs two or three times for short periods – but is large enough to cause poor 95% times.  I have included below the tests running against the times to show this.  The average number of tests running is 22, however there are peaks of 121 at 01:28, 508 at 02:45 and 82 at 02:54.  I cannot explain why these peaks are occurring, so my question is – can you help with explaining why this is occurring? 

 

The load is comprises of three simple tests that do not contain any think times, the test mix is configured at 31,22 and 47 percent; this think time is 240 seconds (we have spent a lot of time ensuring this is realistic)

 

 

 

Test Running.

 

 

-              -             

00:01:50               2             

00:03:35               0             

00:05:25               2             

00:07:10               3             

00:09:00               3             

00:10:50               1             

00:12:35               5             

00:14:25               6             

00:16:10               11          

00:18:00               5             

00:19:50               10          

00:21:35               5             

00:23:25               7             

00:25:15               12          

00:27:00               13          

00:28:50               16          

00:30:35               13          

00:32:25               10          

00:34:15               16          

00:36:00               15          

00:37:50               22          

00:39:35               15          

00:41:25               21          

00:43:15               13          

00:45:00               16          

00:46:50               12          

00:48:35               17          

00:50:25               24          

00:52:15               19          

00:54:00               15          

00:55:50               14          

00:57:40               22          

00:59:25               15          

01:01:15               25          

01:03:00               19          

01:04:50               15          

01:06:40               14          

01:08:25               25          

01:10:15               13          

01:12:00               16          

01:13:50               14          

01:15:40               19          

01:17:25               16          

01:19:15               23          

01:21:00               14          

01:22:50               14          

01:24:40               21          

01:26:25               15          

01:28:15               121        

01:30:05               15          

01:31:50               17          

01:33:40               15          

01:35:25               8             

01:37:15               18          

01:39:05               18          

01:40:50               17          

01:42:40               13          

01:44:25               8             

01:46:15               13          

01:48:05               7             

01:49:50               18          

01:51:40               9             

01:53:25               13          

01:55:15               18          

01:57:05               26          

01:58:50               24          

02:00:40               34          

02:02:25               25          

02:04:15               8             

02:06:05               42          

02:07:50               12          

02:09:40               16          

02:11:30               22          

02:13:15               14          

02:15:05               11          

02:16:50               27          

02:18:40               25          

02:20:30               11          

02:22:15               27          

02:24:05               24          

02:25:50               27          

02:27:40               19          

02:29:30               16          

02:31:15               7             

02:33:05               59          

02:34:50               23          

02:36:40               18          

02:38:30               18          

02:40:15               15          

02:42:05               27          

02:43:55               14          

02:45:40               508        

02:47:30               46          

02:49:15               6             

02:51:05               22          

02:52:55               20          

02:54:40               19          

02:56:30               82          

02:58:15                   23          



Thanks

Andy

 

Coordinator
Nov 8, 2008 at 12:44 AM
Hi,
 Couple of things I can take guesses at:
1. During this period, is there some background workflow task happening?
2. During this period, is GC doing garbage collection thereby taking up system resources to slow down the performance. My guess would be that this is the cause. Because caches expire and a flush is forced upon GC to come collect causing the slowdown and the cache misses cause the system to repopulate the caches in CRM which also leads to slowdown. In order to verify Look at the performance counter "%Time Spent in GC" to see if it shoots up during the slowdown.
Thanks,
Adi.