Domino + Java: What To Recycle?
So, I've written a Java-based WebQuerySave Agent to resize uploaded JPEGs photos attached to a document. Users can upload any size of file and the code reduces them all to a maximum of 450px wide.
The code is working but there are intermittent memory issues, like below:
28/12/2011 13:24:24 HTTP JVM: java.lang.OutOfMemoryError
When this hits, the Java agent fails to complete (obviously, I guess) and the photo is not resized. Ouch.
Before I pass blame to the server I need to make sure my code isn't to blame. I have a suspicion it's because I'm not clearing up after myself properly.
In order to perform the image resize the code uses objects based on the following classes:
- EmbeddedObject
- InputStream
- Stream
- JPEGImageDecoder
- BufferedImage
- ByteArrayOutputStream
- MIMEEntity
Which of these need recycling? Which need flushing? Which need closing? I find it all a bit confusing and I'm not sure how you even are meant to know or work this out.
Is it good practice just to call .recycle() on everything? And .flush() or .close() on any class which supports those methods?
Skip to the 
You can subscribe to an individual
Hi,
The OoM Error ocurrs after the agent run a few times (a)? Or is it thrown when your users are trying to upload very large images (b)?
.recycle() is only for lotus.domino.* objects and in case of a) it will be one of those.
If you don't close Streams properly, you get another exception.
In case if its more a b) case, I guess JPEGImageDecoder is the culprit, because it needs a lot of memory with large images. You may solve the issue by giving the server-JVM more heap space or restricting the images processed by the agent to a certain threshold of mega- or kilobytes.
Anyway, first try to find out the reason for the OoM. a) or b) case.
A notes java agent does recycle the lotus.domino.Session it creates and all the Tree of Domino-Objects generated with this Session. You shall not recycle() those. Recyle() in Agent does only make sense,
if you a) have loops or
b) have Domino Objects created with new (happens with RichTextItem)
You may try JPEGImageDecoder OutOfMemoryException query on the often excelent http://stackoverflow.com
regards
Axel
Reply
Hi Axel,
At first I thought it was a case of b) and only for large images so I restricted users to only upload <6MB files. But then it happened for much smaller files too (~2MB) and I started to suspect it was a case of a).
I'm going to suggest a heap size increase and I'm going to make sure I recycle the EmbeddedObject and MIMEEnties objects.
Thanks for the help!
Jake
Reply
Show the rest of this thread
I should point out that this is a fairly old server running Domino 6.5.5
Reply
I would suggest recycling almost any Domino specific object. Other objects should be cleared by the Java garbage collector. The garbage collector probably also deletes Domino objects but the internally used C objects are not cleared.
If I remember correctly Bob Balaban has written some excellent stuff about Domino and recycle.
Practical example e.g.
http://www.nsftools.com/blog/blog-11-2004.htm#11-03-04
XPages would handle this better but probably are out of scope for you.
Reply
Jake. Its years since I did much with Domino/Java but I remember solving some memory issues by changing the JavaMaxHeapSize in the Notes.INI. Don't know if that is much help to you.
Reply
Sorry, just noticed that you had already looked at the heap size! Will read properly next time :-)
Reply
Show the rest of this thread
Do you use external jar files ? if yes, place them in jvm/lib/ext rather than import them in your agent
recycle domino objects except currentdatabase, currentsession
tuning heap size will help also
be carefull with closing and nullifying ressources that are no more used...
Reply
The short answer is, when you need to clean up the back end of a Domino object you use recycle.
For example (the most common). If you are in a loop reading documents from a view. Then you should recycle the current document for that iteration of the loop.
You need to be careful because Java objects can share the same back end. So if you recycle a document that is being used by another java object, its back end can also be recycled.
As for a definitive answer, you would need the full stack. For example if the out of memory is occurring in AgentLoader then try moving any embedded jars to lib/ext as Michael mentions.
Reply
How would you get the full stack trace from an OoM exception?
Reply
Show the rest of this thread
If you have CONSOLE_LOG_ENABLED=1 then it should show up in the console logs in the IBM_TECHNICAL_SUPPORT folder.
XPage errors are in the same folder in an XPage log file.
If it is a Java application (ie. DIIOP) it should print the full stack unless you are wrapping it in the code.
Reply
Try to increase the Java heap size in notes.ini file. Usually this helps in OutOfMemoryErrors ... or just pushes the problem just a little bit further.
Always use the flush() and close() when dealing with io.streams.
Reply
Jake,
I've been using a WQS to turn attached JPEGs into thumbnails for a few years now, also using JPEGImageDecoder and the rest of those classes. I've never been able to throw it a JPEG large enough that I get the OutOfMemoryError and I don't recycle anything (I attempted recycling year ago, but it caused errors and ended up commenting them all out). As far as I know I am using the default heap size. I'll send you any code you want.
What I am curious about is what you do with all of the resized JPEGs. Do you store them in a richtext field (what I do) or do you reattach them as .jpg files?
Reply
Hi Chris,
The JPEG is sent to an output stream and then passed to a MIME-type field like so:
Stream stream = session.createStream();
stream.setContents(new ByteArrayInputStream(out.toByteArray()));
MIMEEntity Files = doc.createMIMEEntity("Files");
Files.setContentFromBytes(stream, "image/jpeg", MIMEEntity.ENC_IDENTITY_BINARY);
Perhaps I could skip the MIME stuff though and just use the server's disk (using the MIME trick above saves me having to use disk I/O operations).
Jake
Reply
The ultimate way is to see exactly what objects (deep size especially) are taking the bulk of the heap. We use JProfiler to attach to the running JVM and we can see right away in resl-time the growth of objects and who/what is creating them.
Reply
Good point. But this may work only with Domino 7, as JProfiler only supports VMs 1.4.2 and above. Domino 6.5 runs with 1.3.1.
They offer a 10 days test period.
Here is at least an article how to profile Symphonie Applications with JProfiler. www-10.lotus.com/ldd/lswiki.nsf/dx/How_to_launch_Symphony_with_JProfiler
I am not sure, if you can connect JProfiler to the Domino VM. It would be interesting to know if someone has done this against a Domino VM.
Reply
Domino 6.5 is Java 1.3, so you deal with stone age stuff (kind of). The general rule: what you create is what you recycle (so you won't recycle the session). What I found in agents: if you have external jars and the agent runs multiple times it starts bleeding memory since a new classloader is instantiated every time.
The solution here is to deploy your jar into lib/ext so it isn't part of your agent, but on the general Java classpath (you can use an LS agent to do the deployment - see the patched updatesite.ntf on OpenNTF). That improves the situation quite a bit. You actually could put everything in the jar and use the agent only to hand over the collection for processing.
Reply
YES. I forgot that. Allways put any third party jar in lib/ext. Admins tend to don't like it.
Reply
Hi Jake, first of all - I really enjoy reading your Blogs, thank you!
I've seen this error running similar Java code (resizing JPEGs) on my sites. It has probably to do with the increasing size of filesize on the JPEGs being uploaded. I tried different approaches to solving this (JAI and other libs). But at last I solved this by using Imagemagick installed on the serverside (I don't know if you have the possibility to install software on the server?) . I'm uploading the files to a servlet (uploading multiple files at once) and resizing them with Imagemagick before reattaching them to the document in the wQS agent.
Code snippet from servlet:
private static boolean convert(File in, File out, int width, int height, int quality) {
if (quality < 0 || quality > 100) {
quality = 75;
}
ArrayList command = new ArrayList(10);
command.add("C:\\ImageMagick\\convert");
command.add("-scale");
if (width == 0) {
command.add(" "+height + "%");
}
else {
command.add(" "+width + "x" + height);
}
command.add("-quality");
command.add(" " + quality);
command.add(in.getAbsolutePath());
command.add(out.getAbsolutePath());
return exec((String[])command.toArray(new String[1]));
}
Hope this can be of help :-)
Reply
Back in 2008 I needed to create thumbnails of images and limit the size of an image. I used something I found which I imported into a java agent and attempted to call it on a WebQuerySave event. Being that I didn't know much about java (though a lot about C++) I wasn't successful running the java agent.
I finally created a script library with an object of the code I found to resize the image called thumblib. I then used the following at the top of a LotusScript agent to call the java library.
Option Public
Use "thumblib"
Uselsx "*javacon"
It worked. I did find a bug with the java code in that if I uploaded an image with the same name as a previous image it had cached the previous thumb and would return the old file. Looking around I changed the code to the following, which fixed the problem.
// load image
// Image image = Toolkit.getDefaultToolkit().getImage(imagename);
// Use createImage instead of getImage because
// getImage may cache the image which we don't want.
Image image = Toolkit.getDefaultToolkit().createImage(imagename);
Reply
Hi Jake, the short answer is "Yes", assuming you can be sure you're done with the object in question.
Domino agents automatically release all their (Domino) memory when they finish, by recycling the agent's Session object. However, if you don't clean up as you go, you run the risk of never reaching the end before running out of memory.
Feel free to read much, much more than you ever wanted to know about recycling and garbage collection in both LotusScript and Java on my blog (http://www.bobzblog.com)
Reply
See my post pursuant to issues such as there:
http://www.critical-masses.com/jakeofalltrades/stack-vs-heap-why-it-matters-notes
Reply