Christmas Come Early
Dear Santa,
For christmas I want a new @Function. It has to be called @ViewCount("ViewName") and it has to be really really super-dooper fast. Faster than Tommy's next door. He says his mummy is getting him SQL this year and that it can do it in one zillionth of a second with 3 million trillion documents. I want mine to be infinity+1 faster than that. Please. I will be a good boy all year. I promise to stop saying nasty things about Domino.
Jake (aged 29)
The more I think about the code Domino generates the more I am convinced we need to take charge. This is especially true with websites. It's no longer the case that you can impress The Boss merely by showing them a Notes database in a browser that they can edit and save. Remember when you could press Submit and amaze people with how the page updated in the browser, there and then!? Nowadays they want it all. They've used the web long enough to know exactly what you can and can't do. Now, if I had a pound for every time I had said "You can't really do that with Domino"...
If you let Domino do the coding for you there will be numerous problems. For example, normal domino views don't navigate all that well. I've created a sample database with 23 documents in it, named numerically. Look at the normal Domino view that used ViewNextPage style hotspots. Notice there's a previous link even though we're on the first page. Press it and the page simply reloads. Press the Next button and notice the view starts with the last document from the previous page. Keep pressing Next until you get to the last page. Press Next again and it takes you to a page with just the last document in it. Keep pressing Next and this page keeps re-loading. Now explain all this the Boss/Customer and why you can't change it.
Well, I've got someway towards a solution. All we need to do is find the count of documents in the view and set a field on the view's $$ViewTemplate to this value. This is done in a WQO agent. The rest can be done with @Functions. Here's the first bit of code I tried:
Set nav = view.CreateViewNav()
Call doc.ReplaceItemValue( "Total", nav.Count)
I also added some code to the agent that computed the time before and after this code, worked out the difference and sent this to the page as well.
Opening a view with 5,000 docs took 2 seconds, with 10,000 docs it took 7s and with 15,000 it took 9s. Not good! So I swapped the NotesViewNavigator for a NotesViewEntryCollection and the 15,000 documents were counted in 3s. Better, but still not great. Then I switched to a NotesDocumentCollection and used the following code:
Set col=db.AllDocuments
Call doc.ReplaceItemValue( "Total", col.Count)
This took 0s with 15,000 documents. Better. Obviously we dont want all document though. So I tried the NotesDatabase.Search() method, which also returns a NotesDocumentCollection, of which count is a property. I passed the search method the same formula as the selection criteria for the view in question and 15,000 docs took 0s. Increasing to 25,000 took 1s and 91,000 took 3s.
So, we can work out the number of documents in a view and create our own navigation. I've done so in this "super" view, which shows the same documents. Notice how next page doesn't include any documents from the previous page!
If you open the actual documents you'll notice that they have next/previous navigation at the document level. Not only that but they include the title of the relevant entry. This is based on the code I posted on Wednesday. The good news is that document-specific navigation of a NotesViewNavigator seems to be unaffected by the number of documents in the view.
Thanks for reading. Article to follow...
Skip to the 
You can subscribe to an individual
Does it all work with Categorised views? We have had some problems with view counts and next / previous pages with these...
Yes and no. It all depends how many documents you expect the view to contain. If it's 1000s then you can see there's a huge performance hit. If it never more than a few hundred, then you can use the NotesViewNavigator class and then the CreateViewNavFromCategory method.
If you have lots then you can make the search formula for the .search() method as complicated as you like. You can mimic a categorised view with a search formula.
There is a property in the NotesView class notesView.TopLevelEntryCount.
I never used it, but maybe it can be solution for flat views to get total number of documents.
Not sure about performance though.
Thanks Artur. I'll give it a go.
I'm 40, and I'm starting to think that santa doesn't exist. But when your article turns up, I'll put on a red suit and deliver notes clients to both of my kids for christmas!
Nice. The DomBlog search results page could make use of this! ;-)
I would add some more to Santa's wishlist:
Give me some new view properties:
- output as XML
- use this XLST stylesheet
- allow overrite
- tranlate/don't translate on server
... most better clients could digest an XML view if a processing instruction for XML would be given...
Otherwise there is allways the servlet.
:-) stw
I was thinking that it might be possible to make a few API calls to get the count, and just wrap those in LS for an open agent. However, the following caveat from the API docs seems to provide a hard limit to what you are looking for. Otherwise, you could probably write a few lines of code to get a count of docs in the view.
"NIFGetCollectionData is only useful for providing information about a collection which is not categorized. The index of a categorized collection is implemented using nested B-Trees, and it is not possible to provide useful information about all of the B-Trees which make up the index. If NIFGetCollectionData is called for a categorized collection, the data that is returned pertains only to the top-level category entries, and not to the main notes in the collection."
I was thinking about hiding the "next button" with some JS if the view tables didnt contain the max number of rows. I havent tried it but...
Hiding it is one thing "program". But this doesn't get round the fact that the links are un-search-engine-friendly. There's what the user sees and then there's what the search engine sees. Both are as important in their own ways.
Thanks for this Jake.
My own approach to date has been to rely on some fancy footwork in @Formulas. However, right now I'm sweating on this because of the 64k limit for @DbLookup/@DbColumn - I've made life a bit easier by counting a column with a single character in it.
Your evolving solution is a lot better - more robust, able to handle large numbers of documents. I guess I have avoided using too much lotusscript until now, but you've given me the impetus to go an make the change!
Thanks for this very interresting blog entry.
You're right, it is very difficult to say to client/customer that is quite complex to do with Domino whereas every PHP website can do this easy.
By the way, I am quite suprised by the use of "NotesDatabase.Search()"
From what I remember this method is very very slow...
But it seems to be quite fast with your test ???!!!
Maybe there is a mistake and you have used FtSearch (which is quite fast if the database is indexed).
I am also surprised because I remember there was a limit by the result return in a collection. 5000 I guess ?
Maybe this number have changed in R6. (I am always on R5)
Have a nice day !
Fabrice. It was definitely the .search method and not ftsearch(). I was a little surprised too. The test was in R6, where the third parameter to the function was 0, which tells it to return an "unlimited number of documents".
If I get the chance I will test some more. The test db was very simpl - one for and two views and the search criteria was simply "form='article'".
Stephen - ?readviewentries for your xml output? - I guess your point is the original view just outputting as xml if selected.
In the past I have also used this command in javascript to get a document count from a huge view - just retrieve a single xml page using ?readviewentries (put count=1) and parse the document count out of the result.
Always found it the fastest method - but obviously only suited for some applications.
I've got a javscript function somewhere where you just pass a view name and it returns number of documents in it - will try and dig it out or just write it again - not difficult and is something like 8 lines long.
Nice. To do things like this I've always created a form, an agent and an RTF field. I then did the entire thing in the WebQueryOpen agent, and outputted the view results into the field. I had full control over everything that way, I could make things work exactly how I wanted them to. Performance has never been an issue thankfully, as I've never used it for more than 500 documents.
Just a minor error report: your Normal View opens with "Super View" tab selected.
Thansk Zomek. An error on my part. Laziness really. There's only this template which works properly - {Link}
This is actually the issue that caused me to eventually leave my job as a highly paid notes developer and start-up on my own.
Domino doesn't do large data sets very well - my ex-boss will argue - your solution is a good one Jake, but we were using 8GB of data, and hundreds of thousands of documents - S L O W.
We decided we could either have speed or functionality - but it just wasn't going to work for both at the same time.
er, Jake, is not "3 million trillion documents" you should say "3 brazillion documents".
Kind regards
.::AleX::.
NotesView.EntryCount... works in categorized and flat views, and returns the correct number of documents, not entries (so the categories in a categorized view don't count the way they do if you select them in the UI and look at the selected count in the status bar.)
Well, I have a new challenge for this subject. What about when the docs contained in that view has a............. reader's field (!!!!!!)
I spent days and days trying to solve this, because in a standard view, the navigation next and previous works wrongly. So, I think is not possible manipulate the position in the view to navigate trough the view.
The only solution I found is categorize the view with that reader field. Then, I embebed the view in a form and set the "show only one category" with the CN value of @UserName. This let you use at least the standard previous and next buttons properly. Otherwise, the next button works right but the previous doesn´t. I think this is logical because you may be founding docs that you are not able to see because the reader field. It's very different in this aspect to navigate a view in web rather than in notes.
Please, I need a Guru to make a complete article about using reader's field. We are having performance troubles in large views. It's a good point to give your docs truly security, but upsets users that have to navigate that views. God bless that possible article!!