Fixing the Domino CheckBox Bug
The fact that Domino has a bug shouldn't really come as a shock to any of us. Any software application of that size is bound to have one or two. What is shocking is that Domino R5 has a bug that shows no sign of being fixed despite the fact that it has been officially documented. This article will discuss the bug and a simple solution. On the way I hope to introduce you to a few new concepts and to a way to "debug" Domino forms.
The problem (or "limitation" as Lotus would have us call it):
Likelihood is that you won't see the problem at all. It all depends on how you create your Submit buttons to save a web document. If you always use the convention of a button that uses the following as its formula then you will be okay:
@Command([FileSave]);
@Command([FileCloseWindow])
However, I know that I am not alone in prefering to create my own buttons that use, amongst other things, the following JavaScript call in the the onclick event:
document.forms[0].submit();
What's the difference? Not much. As far as the interaction between server and browser is concerned the method is the same. The reason I prefer to use my own buttons is by-the-by really.
The problem is when you use a JavaScript button to submit a form that contains a CheckBox field. If you submit a new document, with some values selected, then all appears well. However, if you then reopen the document in edit mode, deselect all the values and resubmit, the changes are lost. Now, I don't know about you, but I call that a pretty major bug. Not Lotus! They pass it off like so:
This issue was reported to Lotus Quality Engineering and has been determined to be a software limitation of web browsers.
To fix it they suggest using the "normal" @Formula button. The fact that this is a complete cop-out on their behalf I will cover later.
Some background investigating:
Before I cover the simple, yet ingenious, solution I think it's important that you understand what is happening in the background. In doing so I hope it gives you the knowledge and confidence to do some Domino-form-debugging of your own in the future.
When you submit a form from a browser to a web server all the fields on the form get encoded in to a HTTP "request" string that is to the server. This is normally an invisible step in the whole surfing experience but, with the aid of "packet sniffers" like Ethereal, we can take a peak at this hidden layer.
Note: Ethereal is something I plan to write about in more detail in an article all of its own so I will just go over the basics of using it here.
With Ethereal launched and capturing all traffic on port 80, open the form in question and submit some field values. When you stop Ethereal's capture you will see a list of all the packets exchanged between the client and the server, as below:
In it you can see a list of the information sent between the two machine. If you then right-click on the top-most entry there is an option "Follow TCP Stream". Selecting this will present you with a box as below:
In the box above you can see the whole of the conversation between client and server that resulted from pressing the submit button. The uppermost red part is the data that was POSTed to the server. The blue part at the bottom is the server's response. In this case it's simply the HTML to display a "Form processed" message.
Note: Hopefully you've learnt something even by now and can see how useful the ability to see this data could be.
In the above example I was using a very basic form that consists of one text field and one checkbox. It looks like the one below:
With this in mind let's look at the text of the POST portion of the above conversation and dissect it (I've removed some of the irrelevant parts):
POST /CheckBox.nsf/Problem?OpenForm&Seq=1 HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; T312461)
Host: epsdomsvr01
Content-Length: 41
__Click=0&TextField=test%20one&CheckBox=1
All make sense? The important part is in bold at the end. The number called content-length tells the server how many bytes of data to expect and the last line is the data itself. Look at the last line in detail. It consists of what are called content-pairs. Each content-pair is made up of the name of a field and its value. In this case we have the internal Domino field called "__Click", the field called "TextField" with value of "test one" and the "CheckBox" with its alias value of "1".
Now let's look at the same form being submitted but without any of the checkbox having a selection:
POST /CheckBox.nsf/Problem?OpenForm&Seq=1 HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; T312461)
Host: epsdomsvr01
Content-Length: 30
__Click=0&TextField=test%20one
Notice the difference? The content-length is shorter and the "CheckBox" is not sent to the browser. At this point you might be thinking that this is the bug. It's not. According to the W3C's HTML Specification a checkbox without a value is not considered a content-pair and so the browser is right in not sending field with the POST data.
The problem is a limitation of Domino and not as Lotus would have you believe with the browser.
What I can't quite understand is why the problem is only when you use JavaScript to submit the form. The above transcripts show the data sent when you use JavaScript. Let's compare that with the data sent when you use a button with the normal @Formulas:
__Click=e9b2f3906a21cd3480256c7e004af6f1%2F%24Body%2F0.1C4&TextField=test%20one
Notice the difference? The "__Click" field has a value other than its default of "0". This field is normally set by the onclick event of our @Formula buttons and tells Domino which button we pressed. Surely then this must be the key to it all. But why? What significance does this hold? If it can be done when you click one button then why not another and how on earth can it possibly be a browser limitation!? Lotus?
A solution:
Waiting for acceptance of this limitation being on the side of lotus would be like expecting a month of Sundays. We need a fix. Luckily I can give you one and ingeniously simple. What we need to do is fool the Domino server in to thinking there is a value being passed to the CheckBox field. If we can make this blank when there are no other options checked we can clear all the options in the field in the back-end document. Consider the form below:
The trick here is to add an extra hidden text field immediately after the CheckBox field, with the same name and with no value.
Let's see what difference this makes. First by looking at the make up of the CheckBox field and its possible values.
If in the normal form you select the first and last values the POST data looks like:
__Click=0&TextField=test%20one&CheckBox=1&CheckBox=-1
So, for each selection, a separate content-pair is encoded and passed to the server. So, as long as we send these values all at once and not with any other content-pairs in between, we can pass an extra one in on the end. This is what the extra "fake checkbox" is for. Consider the same POST data but with our hidden field tagged on:
__Click=0&TextField=test%20one&CheckBox=1&CheckBox=-1&CheckBox=
Now for the clever part. If you deselect both values and submit it again the hidden text field is still considered a content-pair even without a value and so is still passed along like so:
__Click=0&TextField=test%20one&CheckBox=
Domino at the other end translates this as meaning that it should set the value of the checkbox from the listed values "1" and "-1" to simply "". Because there's no value in the field that can be "" then all selections are removed as we'd expect. Et voila!!
You can try all this for yourself as I've kindly included the two simple forms in this demo database for you to download.
Summary:
Whether this is a bug or a feature or a limitation of Domino isn't really important. More important is the fact that you can get round most of these things if you apply some "logical hacking" to the situation. While I can't help thinking this shouldn't be necessary I do quite like the eureka moments that often ensue ;o)
If you liked, hated or plain didn't get this article I want to hear your opinions. I've always wanted to discuss Domino development from this angle but have shied away as I don't know what the interest would be.
Domino Checkbox Bug
Thank you for the article.
I am developing intranet applications for a major company in the pharmacutical industry. Domino is not the only development platform here.
A bug like this is not a thing that strengthen Domino as a developing platform. A bug like this and a response like Lotus' is nothing short of a scandal.
/Patrik
Reply
Re: Domino Checkbox Bug
Thanks for the alert, Jake (+ good solution). I completely agree that IBM's response is a lousy cop-out and the weasel-person who authorised it should be beaten with an old MSDN cd. But then I imagine all development platforms have mind-boggling bugs (where the supplier claims they are the fault of something else out of their control), certainly the ones I have worked on have, e.g. MS IIS and BEA Weblogic.
Reply
Checkbox bug
I don't think you do yourself justice with this one. Although I'm sure there are times when you would use a JS button over a formula, I think the error has more to do with how you call the submit.
By calling document.forms[0].submit() you are assuming that the code in the onsubmit button is called, and thus the doClick() code that Domino generates is called. Is this the case?
If not, either through your use of the submit() method, or as a result of browsers not calling the onsubmit code in response to submit(), then it appears that the fault is either in the browser or your code.
Granted that Lotus' response sounds lame, but it suggests that it is a result of onsubmit, and therefore doClick not being called.
What happens if you call doClick directly, assuming you can work out the button reference? What happens if you call onsubmit() instead of submit? The doClick code is important to Domino's processing of the POST request, so does need to be executed.
Reply
Good point!!
In my testing I neglected to add a call like so:
[<code>if (document.forms[0].onsubmit())<br /> document.forms[0].onsubmit()</code>]
Which is how I would normally code my buttons. Will go and check this now. You could have solved the bug a whole lot easire than I or jerome...
Jake
Reply
Show the rest of this thread
Re: Checkbox bug
Dave,
of course, part of the problem is about calling _doClick or not...
but can you do it in javascript ? i dont think so, mainly because the interesting thing in this function is that it fills the Click.value and that you can't possibly hard code things like '(...)/$Body/0.3968'...
why should i always have to use the domino generated code ? when you want to have different validation functions or processing (disable buttons, change a hidden field value...) before submitting the form, you'll probably need to do it in javascript...
so why domino would need to know the _click parameter when all we want him to do is to update the data like the form.action suggests it ?
And, to show the fact that IBM/lotus response is not appropriate, look at the Team-Rooms, they use a javascript validate() function that ends with a submit() and is directly called from the Save button... Should i disable javascript in this database or change the code and how ?
i think more and more developpers want to be able to have a complete control over the (x)html pages generated, and just want domino to process and store the data... so addressing these bugs or features would something nice.
Reply
I disagree with the fact that it's a bug
Jake, i have to disagree with you that the checkbox problem is something IBM should be solving.
Just imagine this. I have a form with a checkbox that is only visible in visible under certain circumstances, eg. in the client or when the document has a certain status. The document gets edited on the web and when it's submitted the checkbox value doesn't get posted. As result domino would reset the checkbox to empty!
-Laurent
Reply
Show the rest of this thread
bug analysis methods
did i read your article before posting my comment on your blog entry ? was it posted before ? am i confused ?
anyway your article is interesting and shows how to understand the way domino and your browser work (or not) together. and it really takes time to apprehend things like this...
we (domino developers) surely need to share our methods to analyse a problem, as mush as we need to share our code.
Maybe one or two short articles would be interesting, dealing, for example, with the http posts (why multipart/form-data and not urlencoded ?) and ethereal, how do you know what domino receives, how it is decoded, or more generally how to debug some code, which are the most famous domino bugs, what is the code domino generates about (_doClick and so on)...
we would then have a domino bugs killer cheat sheet...
Reply
Re: bug analysis methods
You posted your response to the blog while I was writing the articles. What's they say about great minds think alike??
Jake
Reply
Show the rest of this thread
We had this bug at MLIM...
Jake,
Steve C encountered this bug back when you and I were at MLIM (how many years ago is it now?). I showed him the same solution as the one you describe, as he insisted on using his own submit.
I prefer a different solution which combines the power of both @Formula buttons and custom client-side JavaScript. I use an @Formula button which allows me to do things like
FIELD Action := "Approve"; @Command([FileSave]); @Command([FileCloseWindow]);
or in a different button
FIELD Action := "Reject"; @Command([FileSave]); @Command([FileCloseWindow]);
(Where Action is a CFD field so is not saved in the doc but can be accessed by the WebQuerySave agent)
And I then use a bit of inline JavaScipt to modify the onclick event code domino has automatically generated for the buttons from just a call to _doClick(...) to include extra code at the start eg an if statement with a call to a validation routine. Event code in JavaScript is just another data object which can be manipulated.
By ensuring you always call the _doClick code you do not get the checkbox problem.
One day Lotus may let us do this properly by have two events for a button. Client side custom JavaScript which runs first, then if the custom JavaScript returns true the automatic domino submit runs, followed by the server-side @Formula.
Jim
Reply
Seems to be fixed in R6
I tried it in R6 (client web preview only) with a checkbox field on a form with a simple submit and it seemed to work ok. Anyone else tried?
Jim
Reply
Definitely broken in R7.0.3
Just two years after my last posting I can confirm that the old (preR6) behaviour is back and I've just had to implement the original solution (by Jake) on a system that's been working fine under R6.5 for the last two years.
I think IBM have lost the plot.
Reply
call the _doclick event
hi I understand that the basic premise of this discussion is to have have more and more control over how form is submitted rather than leaving everything for the domino to do.
Nevertheless ,here is one thing you can do ,kinda workaround .It is something similar to what have been suggested in previous threads.
Create a button with @command([FileSave]);@Command([FileCloseWindow])
Hide this button (Take care that it should be published on the web).
Create another button and access the object genereated by previous button(Formulae button)using ID or name (in Javascript) and invoke it's onclick event.(Someone in previous thread has suggested to manipulate or modify the _doclick of formulae event).
e.g If you give the id to formulae button as "fileSave" Then the code in the second button onclick event will be document.all['fileSave'].fireEvent("onclick");
This will in turn call the _doclick of the the formulae button ,to make life easy.
Reply
An explanation atlast for this problem
Thanks for the explanation, I have been searching for an explanation to this odd behavior for a long time. I came across your site a few days ago and found so many interesting articles that I am regular visitor to your site.
I liked your "Search" solution too, I was using a search agent and building a HTML page to display the results. More often than not the search used to end in agent timeout error, I was frustrated until I came across the simple solution that you suggested.
Great articles and great WORK...keep it up
Thanks Bala
Reply
sumbit alternative.
I've always created a button with the formula @filesave and FileClosewindow. By putting in the style property display:none; the button is invisible. You can then reference the button by document.forms[0].ButtonID.click();..
This allows you to do what ever you require in javascript, then use notes to generate the code to do the rest.
Reply
Broken link
Jakes; The link from 'officially documented' pointing to the IBM site seems to be broken.
Later
Reply
Re: Broken link
Thanks for that Cabous. We have IBM's redesign to thank for this one. Breaking the golden rule of web design - creating 404s. Fixed now, anyway...
Cheers, Jake
Reply
Not a fix...
This is not a good solution if parts of the form are masked from a formula which checks if the checkbox value is different than "".
Nice try.
This is a bug.
Reply
Ethereal Capture Filters
One of the things you'll need to know to make Ethereal work for you is how to construct capture filter strings. You can find the relevent documentation for this here:
http://www.ethereal.com/docs/man-pages/tcpdump.8.html
I came searching through looking for something to simply help me configure it to filter on port 80. Fortunately, Jake had a link to the ethereal site which in turn links to the above in the FAQ.
Peace
Jerry
Reply
Single Value Checkbox Fix
I know this is an old article, but since I ran into a similar problem the other day and pulled up this hit in my dizzying net searches, I thought I'd leave a similar hack/workaround I found on Notes.net (amazingly!) for my fellow dom developer.
http://www-10.lotus.com/ldd/46dom.nsf/55c38d716d632d9b8525689b005ba1c0/38f0ad0b2 fa26091852569c80068aa04?OpenDocument
Cleverly, using a combination of Input Translation and javascript, the checkbox value is now saving properly for me. Note this is for single value checkboxes (i.e. click here to subscribe), but I'm sure it can be tweaked to handle multi-select checkboxes.
Reply
The IMPORTANT part is
I tried this when explained to me (by someone reading out this solution) and it just did not work. I got them to send me the link and found the "important" step I missed.
Place the hidden field directly after the checkbox definition.
This is a eureka moment.
Thanks Heaps.
Reply
Re: The IMPORTANT part is
still not addressed in 7.x
Reply
solution
Thank you for this article, after one hour lost in finding the reason of this behaviour, I have found useful your solution.
Reply
Definitely fixed under 6.x BUT...
the problem still occurs if you disable a previously checked checkbox (because, for example, the current user is not supposed to change this value).
Disabled items are not "successful" and are not submitted back to the server. Thus Domino thinks "You didn't tell me checkbox1 was still set so you must have unset it".
The checkbox then loses its value. So the solution described here still works in that situation - or do as I do and undisable the checkbox as part of the submit process.
Only took me a weekend to sort all this out.
Reply
Re: Definitely fixed under 6.x BUT...
Actually, I don't believe the bug has been totally fixed in R6.
Try making the checkbox field, "Refresh on Keyword Change", there will be certain checkboxes which won't retain the checked state during a refresh without the fix given above.
Reply
You've saved me again
Jake,
Thanks for your great blog. I can't tell you the number of times I've found answers here.
Tonight, it was the checkbok, um, "limitation". I'm using Dojo's xhrPost to submit my form. This allows me to use the web query save to validate the form on the server side and return any error messages, if there are any.
I started noticing a strange occurrence. When I un-selected a checkbox, the empty value wasn't being returned to the server.
I googled the issue and immediately saw your site.
Your solution worked perfectly, and saved me hours of struggle.
Something you wrote and discussed six years ago came to my rescue here tonight.
I'm going to start my own blog to try to give back to the community. Blogs like yours are my inspiration.
Thanks again,
Jeff Byrd
Reply