logo

Extreme Stress Testing a Domino Server

Earlier this year I talked about OutOfMemory Java Exceptions I was experiencing. Now I find myself back in the position of writing Java Agents that could potentially leak memory and I'm still really non-the-wiser as to whether I'm clearing up after myself properly in my code.

The code I've written generates PDFs and both emails them and attaches to document. There are OutputStream, InputStreams, ByteArrays, Streams, MimeEntities not to mention all the Domino objects being used.

I've done my best to close, flush and recycle everything. But I'm still paranoid. I don't want to be responsible for killing no server.

Stressing Out

So, I've been stress testing the server. Big time. To do this I added a Page in the database and threw together the tool you see below, which lets me tell it which URL to stress test and when I press "Go" it makes a request for that URL once every second until I tell it to stop.

image

The output you see in the textarea is the JSON I got the Agent to return. The Agent does it's usual business of creating the PDF and storing in a document. It then returns some information about the JVM's memory usage. To do this the Agent ends with this code:

out.println("Content-Type: text/plain");
out.println("");
out.println("{\"memory\":"+
    "{\"total\":"+Long.toString(Runtime.getRuntime().totalMemory())+
    ",\"free\":"+Long.toString(Runtime.getRuntime().freeMemory())+"}}");

I then left the Ajax code running and calling the code once a second. Overnight!

By the morning it had made 45,000 calls. Total Memory had fallen from 30.8MB to 18.4MB.

The "free" memory had fallen from a range of between 19mb to 400kb to a range of between 4.5mb to 500kb!!

I then left it running overnight again the next night (with no server respite or restarts). The next day the total memory had risen back to 25MB! Odd, huh?

So, although it looks like the total amount of memory available to the JVM fell overnight the first night, the server managed to regain some of it the next night.

The server is a fairly low-spec VM with 1GB of RAM running on Windows 2003.

server

I've been keeping an eye on the amount of memory the nhttp process is using up and it seems fairly consistent and not about to pop.

Heading For a Fall

Is what I'm seeing typical and expected? Or does the gradual fall in total amount of memory the JVM has mean that, at some point, it's going to break?

Anybody interested in this "tool". I'm not sure whether to make it more configurable and add charts etc. There are lots of powerful web server stress testing software out there, but they're all overkill for what I wanted.

Comments

  1. Nice idea - had similar issues / concerns with an XAgent which was caching a reasonable amount of information in the JVM.

    It is possible to allocate more memory to the JVM, in fact when 8.5 was released the default was 256Mb to cater for X-Pages but this caused issues on 32bit operating systems so it was defaulted back to 64Mb from 8.52 onwards.

    See this technote - http://www-01.ibm.com/support/docview.wss?uid=swg21377202

      • avatar
      • Jake Howlett
      • Thu 9 Aug 2012 04:15 AM

      It's odd that (after a clean server start) the total JVM memory reported is ~31MB, which suggest the JVMMaxHeapSize setting is 32MB. But the server's ini file says it's 64MB. Odd.

  2. don't make the mistake to take the "Mem usage" column of Windows Task Manager for serious. It's misleading big time.

    See http://www.juliusbuss.de/web/youatn.. ..f/dx/huge-memory-leak-in-xpages-engine.htm?opendocument

  3. You might also want to be aware of this setting to. Bit strange this one. Some sort of bizarre exception management for HttpJVMmaxheapsize.

    http://www-01.ibm.com/support/docview.wss?uid=swg1LO58189

    While they're at it, I wish they would make one ini variable called "pleaseDontCrash=1"

  4. Hi Jake,

    The link Julian posted was a question I asked him, the memory question was causing us huge concern. We did have some memory issues, major coding blunders, and we still have some slow memory leak issue. We're using XPages, backing beans, pdf creation etc.

    A really useful tool we have used for load testing is JMeter, http://jmeter.apache.org/ you can create really simple tests, just calling a url every x secs or whatever.

    In our case, we had to set up tests to authenticate loads of different users, maintain cookies, SOAP POSTs, calls with json return values and parse those values then do something else, loop if statements, gradually ramp up users, have sudden flood of users logging on, pull dojo framework files, etc., etc. We could then start adding\ removing items in the test until we pinpointed calls that were killing the server. It also sort of allowed us to work out user load per server. Having said all that...no tests prepared us for when live users started logging on, they do stuff no test will cover!

      • avatar
      • Jake Howlett
      • Thu 9 Aug 2012 07:11 AM

      JMeter seems very powerful. Too much so for me. I figured it would be quicker to create the Ajax tool I showed above than to work out how to setup test plans and whatnot.

      Is the slow memory leak you have just something you've come to accept? How do you deal with it - occasional restart?

      Show the rest of this thread

    1. Nick,

      Thanks for the information about JMeter maybe we could use it to determine the root cause of why the mwlug.com which is written in XPages keep crashing the JVM everyday.

    • avatar
    • Toby Samples
    • Thu 9 Aug 2012 07:55 AM

    Jake,

    If its going to be an ongoing thing, you should check out YourKit, it is an amazing Java Profiler. Lets you check things to an extreme detail level. It probably is suited more for Xpages as it is a heavier user of the JVM, but Java Agents can run amok also.

    http://www.yourkit.com/

      • avatar
      • Jake Howlett
      • Thu 9 Aug 2012 11:49 AM

      Thanks Toby. I might check that out. Is it easy enough to get it working with Domino server?

      Before I do though I want to work out if I even have a leak or if the patterns of memory usage fluctuation I'm seeing are normal behaviour...

      Show the rest of this thread

  5. There is a sample chapter from the XPages Portable Command Guide available at http://www.ibmpressbooks.com/articles/article.asp?p=1833573&seqNum=2 which covers use of the "tell http xsp [xxx]dump" commands and how to analyze the information they produce. I don't have personal experience using them, but their description leads me to understand that they are specific to the Domino server's JVM, not just the XPages environment. If so, that could give you a more accurate way to measure memory in use.

  6. In our latest version of our Domino based CMS, we have started playing with Metrics (http://metrics.codahale.com/), to get live information from a running application.

    • avatar
    • Jorge
    • Thu 9 Aug 2012 07:46 PM

    Have you tried BadBoy? Very straight forward tool for load testing on a smaller scale. http://www.badboy.com.au/

      • avatar
      • Rami Muurimäki
      • Fri 10 Aug 2012 02:01 AM

      My votes goes to badboy and JMeter tools. If I remember correctly, you can even combine these tools by recording the sequence in browser and then playing that sequence using JMeter. I did some performance tests using these two tools some years ago.

    • avatar
    • axel
    • Mon 13 Aug 2012 05:28 AM

    Did you attach any .jar files to the agent instead of putting those in lib/ext folder on the server?

    From my experience there is a bug in the domino agents classloader and I remember Stephan Wissel having confirmed that on an older discussion on your blog.

    In my case this issue caused 95% of my headaches with OutOfMemory Errors in Domino agents.

    If you know this point, the rules for explicit recicle aren't that dificult and the agent runtime cleans most objects correctoy.

      • avatar
      • Jake Howlett
      • Mon 13 Aug 2012 07:39 AM

      Nope. All JAR files are on the server's lib\ext folder. I've learnt that mistake before ;)

  7. I have a question I'm hoping someone can answer.

    I noted the above caution on importing jar files. Does this apply to Java Script libraries as well? Am I safe there or am I better off saving those off as jar files and deploying to the lib/ext folder? More to the point: will using Java script libraries raise the same memory problem as imported jars?

      • avatar
      • Jake Howlett
      • Mon 13 Aug 2012 03:04 PM

      I don't know for sure but I'd be very surprised if it didn't apply to Script Libraries as well.

      • avatar
      • axel
      • Tue 14 Aug 2012 07:15 AM

      I disagree with Jake on this point.

      JavaScript library is a very diffent issue, as they are not loaded by the JVM. There might be different issues with those, but the attach-jar-files-to-agent problem is java only, not javaScript. Its an classloader issue and there are no classloaders in JavaScript.

      Show the rest of this thread

    • avatar
    • Ferdy
    • Mon 13 Aug 2012 01:23 PM

    It's been a while since I dealt with this problem, but I do remember this:

    - The memory fluctuations you see can be because of garbage collection. It's not instant, it's unpredictable when it runs.

    - I once fixed a Java agent that had a memory leak, the problem was a loop, iterating over a view using the NotesDocument class. I figure loops are particularly dangerous when not cleaning up properly.

Your Comments

Name:
E-mail:
(optional)
Website:
(optional)
Comment:


About This Page

Written by Jake Howlett on Thu 9 Aug 2012

Share This Page

# ( ) '

Comments

The most recent comments added:

Skip to the comments or add your own.

You can subscribe to an individual RSS feed of comments on this entry.

Let's Get Social


About This Website

CodeStore is all about web development. Concentrating on Lotus Domino, ASP.NET, Flex, SharePoint and all things internet.

Your host is Jake Howlett who runs his own web development company called Rockall Design and is always on the lookout for new and interesting work to do.

You can find me on Twitter and on Linked In.

Read more about this site »

More Content