Eureka! -- Generating File Attachments in Agents Without Unrestricted Rights
On Tuesday I was bemoaning the lack of a way to stream binary data directly to the browser from an Agent. Me, moaning? Surely not!
Despite the fact nobody knew of a way to stream data directly I did manage to get something out of the whole exercise. What came of it is a way to generate, store and (optionally) email automatically generated files without using the server's disk.
So what? Well, not needing to write to the disk means you don't need to worry about who signed the agent last and/or what access the ID has to the server. Any old developer can now write an agent that works with file generation!
As an example, here's a form that will generate and attach a PDF to itself during the WQS event. It can even email it, as you can see to the right.
You'll just have to trust me for the time being that the agent is run without unrestricted rights.
The demo form also does the same thing with a text file and an Excel file!
I can't show you a CAPTCHA demo online as this server is Linux and without X11 running it can't use the Java graphics libraries.
Although the approach is the same for them all the difference between the Text/Excel and Image/PDF is that they are based on text and binary data respectively.
Generating the text files, as you'd imagine, is easier and can be achieved with a small amount of LotusScript. Here's the code needed in a WQS agent to attach a text file.
Generating binary files like the PDF requires Java (we need to use a ByteArrayOutputStream as a temporary buffer). Here's some commented code that's an abstract from the WQS of the above demo.
There's probably more detail that's worth discussing further and so I'll probably write a full-length article on the topic.
Next week I'll outline how you go about downloading the DEXT app that the demos live in. In the mean time have a play with the demo. Don't abuse the send-email feature though please! I'll be keeping an eye on its usage and will cut it off if used badly.
Skip to the 
You can subscribe to an individual
Very nice.
I need to read an file attachment (a text file) without using the server's disk in LS.
Is it possible ?
Grazie !
Ciao
In Java yes. There's a getInputStream() method on the EmbeddedObject class. You can call read() on that to get the text.
The LotusScript embeddedobject doesn't have a "get stream" method but I'm sure there's a way to read it straight from the document. Anybody?
Reading the files is related to how they are written - Turn off session.Convertmime, use GetMimeEntity and then you can walk though the MimeEntities looking for attachments.
This may require specifying 'store contents as html and mime' and all the headaches that entails.
You can actually use these methods to move attachments from one document to another in memory, but it's a bit of a pain.
Nice work, Jake!
I'm very interested in creating Excel and PDF files. So I'm looking forward to downloading the demo. :-)
Markus
Brilliant! Thank Jake!
One thing I've been doing in one of my systems for years is detaching an uploaded attachment with a new name and then reattaching it. The sole purpose of this was to rename the attachment.
Is there some way to rename an attachment in-place on a Notes document?
I like this. really it's usefull.
Thanks
Fabrice & Jana. Thanks! It means a lot to get feedback like that. Makes it worthwhile to know people like it so much.
Rob. In LotusScript? That would depend on the answer to the first question asked today alayasf.
In Java, yes. Get an inputstream on teh embedded object and read it all in to a temporary ByteArrayOutputStream and then do as in code I posted. Want me to try?
"I can't show you a CAPTCHA demo online as this server is Linux and without X11 running it can't use the Java graphics libraries."
Well, if you are using Java 1.4 you can run AWT headless - without X11.
System.setProperty("java.awt.headless", "true");
I have used it with FOP for generating PDF's.
Logos in SVG were handled by Batik, that requires AWT. The whole thing runs on my Linux box without X11.
In order to use setProterty you will need to created a new file:
/opt/ibm/lotus/notes/latest/linux/jvm/lib/security/java.pol
Content of java.pol:
grant {
permission java.util.PropertyPermission "java.awt.headless", "write";
};
Keep rocking!
Bo Frederiksen/BoFrede
Hi Bo,
I'd tried that but couldn't get it to work. I didn't add a .pol file though. Either way it's getting back to the point where the agent needs unrestricted access (to use System.setProperty) and needing to create files on the server means it's not very practical.
Still, I might give it a go just to show how you can create images on the fly...
Jake
This looks like it will be really useful and generating PDFs in this way is very timely for me. Looking forward to the download being available.
Thanks a lot, Jake.
Jake,
This looks so promising to many things that we do. I'd love to get even a "Hello World" demo to learn how to create these files "on the fly" for our users.
Please post a demo. :-)
I read your blog entries every time they pop up on Planet Lotus.
Thanks for your contributions to the Notes world!
-- Jeff
Hi Jake,
I have used this technique with Domino, to process PDF documents using the iText library. Be aware that Domino does create a temp files representing the byte array on its own. If you do not remove the ByteArrayOutputStream object, the temp file will remain on the server file system.
Learned this the hard way, when the server ran out of disk space.
This may not be true in later versions of Domino. This is the case on Domino 6.5.x.
That should be
/opt/ibm/lotus/notes/latest/linux/jvm/lib/security/java.policy
not java.pol.
First gratz on the method, here's a further question regarding the technique.. Lets say you were trying to stream a file from the filesystem back to the browser...
...would this also be possible it seems that maybe by working in a similar way to how dext does, appending the FileInputStream to a body field and writing this directly back to the browser, may be possible, withought having to save the document in the database first, although I have to admit, I can't get it work quite right, I can get the file but it's always corrupt...again, can do it using a servlet, but would an agent also allow this ?
Reference Mikkel's blog on the issue...
{Link}
keep up the great work Jake.
Nice work, Jake! Like some others, I can imagine a couple of places this becomes handy - like resizing uploaded images and reattaching. Referring of course to the ability to follow Bo's suggestion through.
Another thought I had was something a friend mentioned to me in a different context some time back. He had experimented with loading an entire Domino server instance into a RAM disk - he was running with 4 GB. He said the speed was phenomenal. I'm wondering what possibilities there are for creating a pseudo RAM disk... but not in the typical way we usually access the file system and underlying OS. Perhaps some sort of hash.
Andy. I tried various ways to stream data to the browser via rich text fields etc but none seemed to work. Escecially not in the WQO event. It always seems to need a doc.save() call to commit the MIME/file to the document.
ok well here's an agent i have semi-working in that it retrieves txt files eg. boot.ini from the server root.
it runs unrestricted .. and you call it like : http://yourserver/yourdb.nsf/getfile?openagent&file=c:\boot.ini
//code//
import java.util.Vector;
import java.io.InputStream;
import java.io.*;
import sun.misc.BASE64Encoder;
public class JavaAgent extends AgentBase {
public void NotesMain() {
try {
Session session = getSession();
AgentContext agentContext = session.getAgentContext();
Document doc = agentContext .getDocumentContext();
Vector vec=session.evaluate("@right(QUERY_STRING_DECODED ; '&file=')",doc);
String file2get = vec.firstElement().toString();
System.out.println( file2get );
OutputStream agentout = this.getAgentOutputStream();
agentout.write("Content-Type: application/octet-stream\n".getBytes());
// get an input stream for the attachment
java.io.FileInputStream fileInputStream = new java.io.FileInputStream(file2get);
// write bytes from input stream to output stream
byte[] bytes = new byte[1];
while ( fileInputStream.read(bytes) > -1) {
agentout.write(bytes);
}
// flush and close
agentout.flush();
agentout.close();
} catch(Exception e) {
e.printStackTrace();
}
}
}
maybe you can see something I am missing here..
thx.
I'll give it a shot Andy. Does it work with a JPEG file as well as simple text files though? Streaming text files is easy. It's non-text files that are the problem here.
I read that getAgentOutputStream() is buggy (see above somewhere). Have you not found that?
Jale
seems fine so far, except it doesn't work exactly as expected... but seems consistent.
Regarding the issue. only thing that springs to mind is that binary files are generally using control characters that get stripped out during transfer, these normally need to be stuffed before transfer.
The fact that the text file transfers ok, including some pretty big ones leads me to believe it's the characters that get removed causing the issue. ( and the bytes of the binary that turn up seem to support this )
So there are 2 possibilities that I have been trying, either base64 encode the binary before transfer, or convert it to mime and transfer it so that it only uses the same characters as the text file which it transfers correctly.
- strange thing is.. if it works as a servlet, and you can load servlets into agents too.. it still gets it wrong. I guess IBM only got around this in the inotes mail template by using an active-x control.. I wonder if the only reason it works in a document is it is encapsulated by domino before transfer back to the browser, if so maybe a webservice is the answer.
- it's just seems like a shame / missed opportunity but hey that's notes..
Sorry Andy, are you saying it does work JPEGs? It looks that way at first but then sounds like you're saying it doesn't.
I agree it's a massive shortcoming on Domino. When I first brought it up I was sure somebody would point out how easy it is. I'm a bit disappointed it involves a hack to stream data to the browser from an agent... hey ho, that's life with Notes, as you say.
Jake, have to admit I tested it using bitmaps, as that's what's readily available on the server windows directory already so ...No it doesn't work with these file types.
It does return some data, but the data is always corrupt.
It does work with text files. So this kind of says the process works, but something is causing issues with binary data, possably control characters.
The returned image file, when viewed using notepad is comparable with the original bmp file when viewd with notepad except for certain missing characters / truncation
hope this clarifies it.. always easy to miss out the detail, and I can't add it afterwards..
cheers
the other way involved...
looking at the way the calendar application you wrote (NotesButter) works, it appends strings to a rich text field and you set the type to Pass-Thru HTML, and write the richtext field back out to the browser without saving a file or document.
Maybe it would be possible to set the richtext field to MIME, read in the filestream from the OS, base64 encode it one byte at a time and then stream it back out to the richtext field with the correct header information.
when the browser renders the richtext field it should convert it back to an image - similar to the way Johan Känngård pointed out in the previous blog "Creating Image Files In Java Agents"
- just a thought
Hmm. perhaps i'm just too stupid, but i didn't manage to get this example to work. :-(
Here is what i did:
I created a form (named test) with these fields: Title (text, editable) and Body(richtext, editable). and a save-button. in the WQS i put
@Command([ToolsRunMacro]; "(test)")
then i created an agent (named test) with the code from this site in it.
to try if it works i call the URL: http://.../test?CreateForm
The only thing that happens is a "Form created"-Page.
What am i doing wrong?
Thanks a lot for your help!
The code I posted above isn't meant to be copy/pasted Raoul. It's just to give an idea what's involved.
Better to wait for DEXT to be released in the next few days.
I just thought so... :-/
So I will wait another few days.
Thanks!
this is too good .. thanks jake,
hope to see an demo of this ..
Jake,
I need to take a picture of a Notes document as is and create a PDF out of it?Can this be accomplished using your agent?Does this require any special software installed on the server Domino is installed in?
Thanks.
Hey Jake,
Pretty good tool. Waiting for the demo.
I have a requirement where any mail arrives then I need to see if any attachement is there which can be anything from text file to PDF to word doc or any thing for that matter. I want to read the content of the attached file and if contains specific key word then I need to send another mail. Is it possible and if yes can you give me a start up where to look and how to do it.
Thanks in advance for your help.
Siddhartha
Hi,
Really a great approch of streaming binary data...
I was searching a way to stream word attachments to encode it in base64 then paste the result in an XML.
I hope to find a solution soon with some of your methods.
But i need to learn a bit more of Java before.
thanks again to point to an idea to do that.
Ah, this is excellent!
Previously, I had an agent (formerly unrestricted) creating an attachment by writing to the server's hard drive a stream of data, and then attaching that file. Then, because of a security change, the agent became restricted, and it couldn't access the hard drive.
Now, I can simply create in-memory, instead of writing to the hard drive only to attach and delete! More elegant, and actually fewer lines of code. And no security concern! Thanks!
This is great stuff, Jake.
There's a couple of downers using MIMEEntry stuff for a Domino Web app (although it's perfect for attaching to emails, of course).
For a Domino Web app, you can't see the attachment on the Web afterwards. You've worked around this by using some JavaScript to display the attachment afterwards. (I'm referring to the Demos.PDF.Template form in your demo database on your other post http://www.codestore.net/store.nsf/unid/BLOG-20091123-0347 .)
That's fine, but how do you then get rid of that attachment over the web? You don't get the "Mark Attachment for Deletion" checkbox that Domino automagically generates for "ordinary" attachments. The only way that I can see to do that is the "old" way of writing out the pdf to folder on the server (assuming that your agent has the rights) and then attaching it to the rich text field.
Ignore my previous.
Using your method, Domino *does* automatically show the Mark Attachment for Deletion" checkbox if you're in Edit Mode. It's only in Read Mode that it fails to show the attachments, so you're bit of JavaScript is all that's needed.
Apologies for the confustion.
In response to alayasf at the beginning of the comments, I needed to do the same thing. Here is a simple solution that should get you started.
As Jake mentioned you can do it with a Java agent, I use a LotusScript Agent to call the Java Agent, so it should be pretty easy to adapt it to your needs.
http://www.hankos.net/A556E9/net.nsf/html0/resources/index.html?OpenDocument
Hope it helps
Is there any way the local/sever attachment file needs to be insert/stored in Oracle database by using Lotus Notes LCLSX Classes or any other alternative approaches would be really appreciate.
Thanks,
Anna
Some links on this page appear broken.
You can fix them?
thanks
Woops. Fixed now alayasf.