Efficent Use of Agents on the Web
I was going to write this entry yesterday, but decided to wait for today, so I could comment on the crazy coincidence that it's exactly a year since I last talked about Concurrent Web Agents. They're back on my mind and a topic I want to discuss some more.
I'm working on a fairly involved system at the moment that's using the combination of a WQO and a WQS Agent on almost every page. It's the only way I can make Domino do what's required. It's not that it should be a RDBMS or anything, it's just a couple of requirement that Notes can't meet. Each Form has a WQS and most a WQO. The site's homepage uses a fairly involved WQO to fake three embedded Views. Each view has a WQO. You get get the idea.
Anyway. For a website running on its own dedicated server what do we think is the best way to configure Agent Manager to make sure it all runs as fast and as smoothly as possible?
Codestore.net uses a fairly simple WQO on the homepage that populates the "elsewhere" links and the most recent articles section. Not only that but the Views here use this method of navigation, which involves another WQO. It's nothing too involved though and this site never seems to drag. What about bigger sites with more complex Agents though?
What confuses me is how web agents are handled? Are all these WQO/S Agents handled by the Agent Manager process? There seem to be two different types of Agents as far as the server document is concerned - web Agents and "normal" Agents.
Would increasing the number of concurrent agents in the Server Tasks\Agent Manager tab help us or is that not connected to the Domino Web Engine? If so, what's a sensible limit? It default to 1. What if we push it up to 5, 10 or even 20?
What about enabling concurrent web agents? I understand this is bad if you call more than one agent from a WQS event and the outcome of one relies on the other. Is this the case? My system should be ok in this sense, so, should I enable this setting?
It's all very confusing for us dev bods who simply prefer to churn out the code rather than consider the consequences. Maybe I should be reading Julie Wotsername's Agent FAQ on Notes.net?
Skip to the 
You can subscribe to an individual
IIRC the concurrent agents setting in the Agent Manager tab is how many scheduled agents can run concurrently. I do not there is a limit on the number of user-activity triggered agents running concurrently.
I think your best bet is to enable the "RUn Web Agents Concurrently" setting and rely on Java if you need to multi-tier/multi-thread your web triggered agents.
Web agents are not handled by the agent manager. They're handled separately by the http task. You can even shut down the amgr task, and they should still run.
Hi Jake,
If your Domino server is running on a Windows environment, you can use locks (CreateLock, Codelock, Codeunlock, Destroylock) in your LS agents so that you can be sure concurrents agents will not create conflicts or incoherent situations while working on the same data.
Instead of working with several WQS agents, you can chain them ; I use to do that and it works perfectly. Basically, you redirect your user towards an agent in your $$Return field, which, at the end of its process, redirects him towards another agent, etc etc. It relies on HTTP redirections (no crappy javascript redirection), and if you want them to be more flexible, you can even specify the sequence in parameters (query string), just giving an agent the URL it will have to redirect the user to. Just do not forget to add an URL encoding level every step of your sequence.
Ex : /plop.nsf/first?Openagent&redirect=
%2Fplop.nsf%2Fsecond%3FOpenagent%26
redirect%3D%252Fplop.nsf%252Fthird%253FOpenag
ent (3 levels of URL encoding)
Finally, to avoid most of WQO agents, I use, when possible, that technique : {Link}
It's a kind of data caching in the databases structure, so that you do not have to compute anything (no lookup, no WQO, just including subforms containing HTML code) to "build" a page with data coming from several documents/types of documents. Thus it gives great performances, but it makes the maintaining quite difficult and a bit risky.
Hope it helps,
Remember to check the "Web agents timeout" in the server document (in the "Domino Web Engine" tab). If one of your agents runs wild, this timeout make sure that it is killed, so you don't get a http thread locked.
Thanks guys.
Johan. What's a sensible value for this? 10 secs maybe? You say it risks *a* http thread being locked. Not the whole http task then?
So, I guess the question is, how do we beef-up a server to make it handle agent-intensive websites? Is there no server document setting that has a direct effect on WQ agents' performance, other than enabling concurrency?
Hi Jake,
on the thread time out, we had a shared server on a LAN and the http task was running an agent that returned something like a MB of HTML tables (for Excel display). We -had- to set our timeout to 180 seconds to let the agent complete.
That's a heavy load. If you have something that might take a lot of time, it makes sense to set this high. But, if performance -should- be good, and you have graceful failures (the user doesn't see errors or mangled pages) then a low setting should be fine, say 5 - 10 seconds max. Anything takes longer than that and people are tapping their fingers or clicking away anyway.
There's another consideration on ND6.x, too, and that's that the request thread pool is round-robinned, so threads are assigned in strict order rather than being pooled to see who's busy and who's free. R5 went for the first available thread, but D6 might assign a request to a thread that is already running as fast as it can.
That has apparently been fixed (made configurable) in D6.5.5 (and one would assume, in D7.x). See EriK Brooks' posting in the ND 6&7 forum here for more:
{Link}
I'd certainly recommend the notes.ini setting that Stan mentions (HTTPQueueMethod=2) - for the setup that you describe Jake. Details can be found here:
{Link}
Remember that you should disable persistent connections (on the server document) if you use this notes.ini setting. btw - you set it using a configuration document in the Domino directory.
As to http thread locking, then I have recently seen a wqo-agent that crashed and then made the Domino server unresponsive for all http requests. This issue is now an SPR with Lotus.
The most interesting piece notes web development, I've read forever.
Not for the sake of negativity, but because you've made me curious:
1. I think YoGis solution creates a browser-server roundtrip for any chained agent? If should in my understanding. $$Return returns http state 303 or some such. This should have network latency (performance) implications.
2. Might be a bit theoretical, but is the thread implementation in Domino part of the public interface? If not, it might - at least in theory- be changed in later versions?
3. I have allways abstained from WQO agents. Remember having read somewhere, that they are much more ressource intensive than WQS agents. Is that correct?
thx. u r great.
Axel
Axel : of course it does imply network latency overhead, as each agent execution means a HTTP request and response. I think it depends of your context. Currently I use it for an Intranet, and it's quite invisible for users.
I forgot to mention that it might not fill your needs, as it's more a kind of "WebPostSave" than "WebQuerySave". But sometimes, it makes your day.
I found out 'the hard way' that there is a difference in having Concurrent Web Agents enabled or not...
Because of 'unpredictable' (is does not always match my sense of logic) behaviour I try to avoid working with WQO agents. However life without WQS agents would be very difficult; I haven't been in trouble (yet) with different users changing te same document at the same time...
For my viewtemplates I use a similar approach by evaluating the Total document count. I use a DbColumn with the viewname @Subset(@ViewTitle;-1). This way I don't need a WQO for all my views and it's a generic approach (copy/paste works fine in most cases).
Jake,
Just out of interest, is there a particular reason as to why you use a WQO agent to build the "Elsehwere" links on your site?
I assume that each "elsewhere" link is a document? If that is the case why not use an @DBLookup in some computed text which returns a particular column from an "elsewhere view". Obviously you can design the column to contain html in the format you need.
I would imagine this is quicker than performing some loop in the WQO LS to get each document and do something with it to get the html you need. However I have not done any research into timings etc so therefore I may be completely wrong but it was just a thought !
Cheers
Lee
re: Richard's comments:
"Web agents are not handled by the agent manager. They're handled separately by the http task. You can even shut down the amgr task, and they should still run."
Although they 'should' still run without amgr running, we found that with
Run web agents concurrently? Enabled
the number specified in the "Max concurrent agents" (max of 10) field did affect performance in terms of the amount of concurrency that could be achieved with given HW resources.. at least on AIX, Domino 6.5.3
So there is some link between the Agent Manager Settings and web agents. Sadly, I can't find any documentation to support this.
Dave
IIRC the concurrent agents setting in the Agent Manager tab is how many scheduled agents can run concurrently. I do not there is a limit on the number of user-activity triggered agents running concurrently.
I think your best bet is to enable the "RUn Web Agents Concurrently" setting and rely on Java if you need to multi-tier/multi-thread your web triggered agents.