Most Efficient LotusScript Loop
Last week I wrote an Agent in LotusScript that looped through a view. Nothing new there. I've written endless Agents with this logic in them. It's just that there's often a gap of weeks (if not months) between me writing each one. Most of the code comes straight back to me. The one thing that always escapes me is how best to perform the loop.
At this point I always find myself Googling for something "Most efficient lotusscript loop". Of the pages returned none seem to offer the definitive answer. If they do it's very well hidden among pages and page of other stuff. What I want to record with this blog post is that elusive answer.
The code I ended up with looked something like:
Set TheDoc = TheView.GetFirstDocument While Not ( TheDoc Is Nothing ) 'Logic goes here Set TheDoc = TheView.GetNextDocument( TheDoc ) Wend
Now, this works. But, I'm worried about its efficiency. I remember learning that there was a method of looping which was highly inefficient. I'm sure I heard about an approach which involved LS storing each looped document in memory. For a view of 10,000 documents the code had to store each one (or at least a stub for it) in memory. Not good for performance.
What I don't know is which method it was. Hopefully one of you guys do. So, in the future, if you (or I) search Google for things like optimal LS looping, most efficient lotusscript to loop a view, best way to loop a view or something similar you will find this entry. Then, hopefully, between yourselves, the responses to this post will settle it for good.
Skip to the 
You can subscribe to an individual
I always thought that what you did was best - get first then while not nothing get next.
Getting the nth value is definitely inefficient as that is a long winded way of "get first, is this n? no, get next, is this n? no, get next etc.".
At least that is what I recall...
CAroline
I have proposed a similar topic for an article in "The View" but it's been put back a few months, otherwise I would have an answer for you now.
My feeling is that there are a few key variables involved in determing "The best" looping code. For a start, does the loop update information or just extract from the documents ? Do the values exist in the view or are they stored in the document ? I have a gut feeling that another important factor is the percentage of entries in the view that are relevant. If the code tests each view entry and only updates or extract a small percentage of documents then I think that certain code would be more efficient than in a situation where all, or nearly all, documents are updated.
Ethann
That rings bells Caroline. Must be that.
Ethann. Bit more in-depth than I was thinking. I'd be interested in the article all the same.
Yes, using GetNthDocument is much more inefficient than GetFirstDocument and GetNextDocument.
The more documents you are looping through, the worse GetNthDocument performs. I believe this is because it always uses the first document as its reference and counts through until it finds the 'Nth' document requested. Whereas doing your way Jake, it only has to refer to the previous document and count one ahead.
Try to use NotesViewEntry instead of NotesDocuments.
Much faster for most loops operations (Depends on the kind of operation you want to perform on the document).
You could also use
Do Until TheDoc Is Nothing
...
Loop
instead of
While Not ( TheDoc Is Nothing )
...
Wend
- it saves you a 'not'
It is definately GetNthDocument that needs to be avoided. Very inefficient.
I know I will be proved wrong, but I can;t actually think of a reason to use this method?
If I have to get a specific document from a view, I will usually take the time to create a sorted column so that I can use GetDocumentByKey
I look forward to being shot down in flames!!!!
Mike
I always use
Do while Not (doc is nothing)
...
Set doc = view.getNextDocument(doc)
Loop
I've never had any use for getNthDocument...
As everybody here I am used to this kind of loop with view.getNextDocument.
As Vince said (for sure I will not tell a Vince is wrong ;oP), I also relay on ViewEntries when I only need to work with document informations (when building Web Views for exemple) and not need to modify document. In this case, ViewEntries are much faster as you only need to access the view index (I mean the information displayed in the view) and not the data stored in the documents. Another advantage in ViewEntries is that you can modify the information rendered by your agent with simply modifying your view and not the agent code.
And to end with, I cannot remember me ever used any getNthDocument. (sorry Mike, but I will not be the one who is blaming you)
Vincent
It maybe a bit a digression to the efficiency question, but I always had trouble remembering the method for keeping a pointer to the next doc when you delete or re-categorise a doc (which means it disappears from the view) as part of the logic
So I think you can use something like...
Set TheDoc = TheView.GetFirstDocument
While Not ( TheDoc Is Nothing )
Set NextDoc = TheView.GetNextDocument (TheDoc)
'Logic goes here
Set TheDoc = NextDoc
Wend
But then I discovered ....
TheView.autoupdate=False
The view does not get updated until the agent completes and you can process as usual..
Keep up the good work, Jake
Here's something that I came across earlier today:
{Link}
I know it is about 2.5 years old, but most of it is still pretty relevant ...
Hi jake,
I build an agent wich use Timer statement to give the exact processing time for some kinds of loops. The result, in agent log, show you that the way you are looping throw your documents are not the "quickest" (i'm not sure of my english on this word... ;-).
Here is my result on 1000 docs :
05/09/2005 15:04:38: While loop execution : 1,64 seconds.
05/09/2005 15:04:41: For / Next loop with GetNthDocument execution :2,50 seconds.
05/09/2005 15:04:43: Do Until loop execution : 1,63 seconds.
05/09/2005 15:04:46: View entry loop execution : 3,09 seconds.
05/09/2005 15:04:49: View doc loop execution : 2,95 seconds. <- wich you use
But my code only test the looping speed not the time your script take to you get data (with search, FTSearch or getview method...). This is very important to say.
I send you the agent script by mail, it's too big to copy/paste in this discution.
HTH
I use the
Do until doc is nothing - Loop
When I can assume, that my code inside the loop might alter (or remove) the curren document I code:
set doc = doccol.getfirstdocument
do until doc is nothing
set nextdoc = doc.getnextdocument
... logic goes here
set doc = nextdoc
loop
:-) stw
@Liam:
I've been developing Domino for many years, but the problem with deletion of NotesDocuments whilst trying to loop through a list was always a long winded one for me until you mentioned the AutoUpdate parameter of NotesView.
I cannot believe I haven't found this one before!
How did I do it in the most efficient manner previously? It was somewhat more complex and involved building up an array containing a list of UNIDs to process before doing the process itself:
dim session as NewNotesSession
Set db = session.CurrentDatabase
Set doc = view.GetFirstDocument
count = 0
While not (doc is Nothing)
count = count + 1
redim preserve docarray(count) as string
docarray(count) = doc.UniversalID
set doc = view.GetNextDocument(doc)
Wend
for index = 0 to count
set doc = db.GetDocumentByUNID(docarray(index))
call doc.Remove(True)
next index
Now I can do the same thing without the For...Next loop. Thanks!
GetNth is good for a couple of things -- getting a random document, view entry (or whatever's in the collection), or for getting the first or last item in a partial collection (getting the 81st through 100th documents by using GetNthDocument(81) followed by 19 GetNextDocuments -- the same as &start=81&count=20 in a URL).
I's like to stress one interesting point, that's been brought up in this discussion. Some posters stated (probably most of us would have agreed) that looping through NotesViewEntries should be more efficient than looping through docs. Now, Juliens findings seem to imply the opposite.
Why was I so willing to accept this assumption? Two reasons: NotesDocument seems to be the "fatter" object than NotesViewEntry and - as it was said before - ViewEntries only represent what's in the view index. But is this really true and/or meaningfull?
The big question is: How fat is the NotesDocument? We tend to think, that once the object is set, it contains all the document's properties immediately, including the items. But isn't this just a misconception, based on the common (ab)use of extended class syntax? Items are not properties of the doc, doc.field(0) is just a short hand notation of the GetItemValue("field") method, imposing an additional performance penalty, which is commonly estimated to be around 10%. Working on a remote db with the LS debugger, which - to the best of my knowledge - causes LS to be interpreted line by line, instead of the compiled object code beeing executed, seems to proof, that item values are not retrieved until the GetItemValue("field")(0) method (or it's "long foot" not-so-equivalent .field(0)) is called.
Furthermore, we CAN use data directly from the view index using NotesDocuments as well! doc.ColumnValues is explicitely recommended in designer help since the early days of LS. This would not make any sence, if the item values were already there within the NotesDocument object.
@harkpabst_meliantrop and @Julien:
I find Julien's timings very interesting, and had also always assumed that looping through NotesViewEntries should be the fastest way.
Julien, i am confused about your timings, though. You stated that While loop executions took 1.64 seconds, whereas View doc loop executions (which Jake used) took 2.95 seconds. What's the difference between the two?
The reason that I'm interested in this is that I currently have a major timing issue using NotesViewEntryCollection as opposed to NotesDocumentCollection.
I am using a regional address book containing approx. 12,000 groups and 20,000 users.
I am using the ($ServerAccess) view in the NAB to find all groups containing another group, then checking in the Database Catalog to determine whether that group is used within an ACL. Then checking further nested groups until no more parent groups are found.
I originally started using the following kind of code:
Dim s as new NotesSession
Dim db as NotesDatabase
Dim view as NotesView
dim vcoll as NotesViewEntryCollection
dim ve as NotesViewEntry
set db = s.Currentdatabase
set view = db.GetView("($SerrverAccess)")
set vcoll = view.GetAllEntriesByKey(GroupName)
Set ve = vcoll.GetFirstEntry
do while not ve is nothing
'do processing
set ve = vcoll.GetNextEntry(ve)
loop
This was in a scheduled agent and it timed out after 200 minutes. The logging that I put in place seemed to indicate that the 'Set ve = vcoll.GetFirstEntry' line was where major delays were occurring.
When I switched the code to use a NotesDocumentCollection instead of NotesViewEntryCollection, the agent run time reduced to 11 minutes - a massive improvement. But I've been unable to find any documentation as to why this massive differential in timing exists. Has anyone any ideas?
Tony
The only time I need to use GetNthDocument is in relation to a NotesNewsletter. A method of creating email notifications that's easy and gives geat results but not often used.
Because when you call the NotesNewsletter.FormatDocument function it requires the index number of the document in a collection. So if you want to get a handle on the same document you only have the index number as a reference.
But other thanthat I can't think of any use for it!
I've done a few tests from time to time (generally when I go through a period of self-doubt as to what the best approach is) and it always comes out with GetNextDocument as the way to go...
Sorry for the lack of informations in my last post. Here is some details :
Search or FTSearch method give you a NotesDocumentCollection. Looping a 1000 docs ndc give you this result :
With a While/Wend loop : 1,64 seconds.
With a Do Until loop : 1,63 seconds.
With a For/Next loop with GetNthDocument :2,50 seconds.
Looping a view with the same quantity of documents by view.getnextdocument : 2,95 seconds
In the same view using entry (vc.GetNextEntry) : 3,09 seconds
This is just a constatation in my application with my script. It's not conclusion.
Send me emails and i'll send you my script in return. It'll be nice to have another feedback.
A couple of points
1) You wouldn't catch me using a do until loop in this case because the code would generate a runtime error unless you put an extra check in up front for the scenario where the view is empty.
2) I've had a situation in the past when I needed to update documents (with say a status) with something that causes them (or some of them) to disappear from the view - this causes all sorts of problems. What would people say was the most efficient approach then? I'd go for a database ftsearch and then either a stampall or process each record in the collection individually if necessary using a while loop.
As an aside Jake - maybe you have the thought of java and looping through documents with the memory thing.
Using java you have to recycle your document objects on each iteration so you don't end up with lots of document objects.
In my own code I often use a while-wend-loop and never experienced this kind of memory problem... I think I agree with Steve, that this "feature" derives from your experience with java. I hope so... ;))
Anyway, I believe the most powerful LS-based loop through a view is a combination of a viewentrycollection and the while-wend-loop. I never stopped the time for looping through a collection, but it saves time because you only access the view once, when you're getting the collection...
But this implies, that you have enough RAM I think...
I think what is being missed is what is happening when people are using a view versus a document collection (versus a viewentry).
In what Tony Campbell-Cooke wrote, the changing in what he experience seems to be related to not having the view's autoupdate set to false. What he was faced with is the view refreshing causing his process to slow down. Moving to a DocumentCollection is like "taking a snapshot" of the view at a specific point (in time). Was the documentcollection faster? This really can not be determined as there is no knowledge of how many documents were being processed.
I would suspect that having the view's autoupdate set to false (theoretically) should give the same results as using a notesdocumentcollection. My concern on the document collection is that at some point it will be insufficient because of the number of documents being moved into memory.
As when it comes to viewentry, I have not seen any improvement to this over notesdocuments. If I was going to use some front end classes or want to capture computed view column value, I could use this class. But remember doing something in the UI is not going to be faster than something in the back end. I just think when it comes to processing you may want to stick with back end classes for any "processing" agent.
Just my 2 cents (or 2 pence for the UK crowd).
I actually have the VIEW article on this and it was from like 1999.
Since then I have used the following for looping...cuz they said so...
Set ncol = nvw.GetAllDocumentsByKey("<key>")
Set ndoc = ncol.GetFirstDocument
For x = 1 to ncol.Count
<do stuff here>
Set ndoc = ncol.GetNextDocument(ndoc)
Next
I've also used this walking a view by:
Set ndoc = nvw.GetFirstDocument
For x = 1 to nvw.AllEntries.Count
<do stuff here>
Set ndoc = nvw.GetNextDocument(ndoc)
Next
I think you might find this faster...I dunno...I just believed what THEY said. ;)
The Fonz
Whenever I need to walk though a view and update / delete documents, I do the following ( Learned at LotusSphere ):
Set view = database.GetView("View")
Set doc = view.GetFirstDocument
While Not doc Is Nothing
Set nextDoc = view.GetNextDocument(doc)
Call doc.Remove(True)
-or -
doc.FieldToModify = "xyz"
Set doc = nextDoc
Wend
In response to Patrick Corey:
The database that I was working on was my own development copy of the regional NAB, and as such, no other updates were being made to it. So in that situation, I don't think that setting the view's autoupdate to false would have made any difference. (Please correct me if I'm wrong)
I did a very simple exercise comparing the performance of navigating using a notesdocumentcollection and a notesviewentrycollection, using a very simple view with a single unsorted column, using a script provided by Julien Bottemanne, on the same database containing 34,000 documents. The results were: using the notesviewentrycollection navigator: 21.68 seconds and using the notesdocumentcollection: 26.13 seconds.
Interesting! There are obviously other factors that need to be taken into consideration when comparing the benefits of the two methods.
Tony
Dorian mentioned above:
>>You wouldn't catch me using a do until loop in this case because the code would generate a runtime error<<
Please note that this is only true if it was:
Do
.....
Loop Until TheDoc Is Nothing
The versions posted above with 'Do Until TheDoc Is Nothing' in the first line work perfectly also for empty views.
There was an article on notes.net around five years ago (by either Ned Batchelder or Bob Balaban explaining exactly when each type of loop was most effieicent and why (due to the data structures' efficiencies.) Try asking them...
Here is a small experience :
I ran my agent (if you want it email me) on the Lotus knowledge base (34 000 docs) and here is my result :
07/09/2005 08:34:03: While loop execution : 27,97 secondes.
07/09/2005 08:38:11: For / Next loop with GetNthDocument execution :248,77 secondes.
07/09/2005 08:38:34: Do Until loop execution : 22,79 secondes.
07/09/2005 08:38:53: View entry loop execution : 19,02 secondes.
07/09/2005 08:39:21: View doc loop execution : 28,01 secondes.
This result mach with Tony's. View entry loop is the quickest on KB.
So i decided to run my script on a big db (15 000 docs) but with documents with few item inside (4 text items) and here is the result :
07/09/2005 08:53:30: While loop execution : 1,65 secondes.
07/09/2005 08:54:43: For / Next loop with GetNthDocument execution :73,14 secondes.
07/09/2005 08:54:44: Do Until loop execution : 1,68 secondes.
07/09/2005 08:54:51: View entry loop execution : 6,76 secondes.
07/09/2005 08:54:54: View doc loop execution : 2,92 secondes.
It seems that lighter (in items) your docs are slower is the view entry loop (compare to the other loop). But on the other hand, bigger are your documents quickest is the view entry loop.
So we need to adapt our loop on the documents we are working on.
Google reveals a white paper from a good source.
{Link}
In summary ppg. 10-12, from fastest to slowest ...
forall (for lists & arrays)
for...next
while..wend
do...while
Tried (20-12-2009) : whitepaper was removed ("404 Error ..
We're sorry but the page you tried to access could not be found."
You've picked the fastest way. If you want faster the CAPI kicks ass but takes time.
For what it's worth, I've seen where a for loop through an ls:do result set was much, much faster than a while loop through the same result set.
Another tip I just discovered today: Say you want to loop through 220,000 small documents and compile some stats. Actually it is a domlog.nsf and I want to add up requests from the same IP to identify who is slamming my server with a bot (Turns out it was Google).
Anyway, I tried a while loop and it got to about 150,000 an then just slowed to such a crawl, I could see it would never finish. I tried a for loop and it slowed t around 400! Tried view entries instead of doc and it quit around 300. Back to the while loop it quit around 300. Rebooted the PC and tried again and it got to about 50,000 before slowing to a stop. The CPU got to 100% and I had to cill my client with Task Manager.
Well the problem was that I had a print statement in the loop to print a record counter to the satus bar of my client. I just wanted to know how far along the agent was. I don't know why, but I could only get a few hundred prints most of the time (or 150,000 after a fresh reboot). When I put the print into an if statement using MOD to print only every 100th record, the while loop flew through all 220,000 documents no problem.
Hope this helps someone to not beat their head up against the wall for so many hours like I did...