Combining Scheduled Agents - Good or Bad Idea?
The more projects I work on the more refined the techniques I use tend to become. The better of the new techniques from the last project I work on tend to make their way in to the next one. And so on. Lately I'm getting to a point where I'm really happy with the base starting point I have. Particularly things like the WebSession object.
The latest area of a database's design I've paid some attention to is the scheduled agents it inevitably contains. Have you ever seen a database littered with scheduled agents all set to run daily at 1am each morning? Surely there's a cleaner approach both for developer and the administrator.
What I've started doing in the last couple of apps I've worked on is have a single scheduled agent run at 1am each day, simply called "Daily Admin Tasks". Each time I need to do some cron task over-night I add the code in there, rather than create yet another agent to do the job.
Each chunk of code is like a pseudo agent and acts on a NotesDocumentCollection object. This is created using the database's search method and is a bit like replacing the Document Selection criteria of a standard scheduled agent.
Here's an idea of what a mean. This code has three tasks to perform each night:
'Pseudo-agent 1 Set col = db.Search({Form="Discussion" & ArchiveOn=@Today},Nothing, 0) If col.Count>0 Then Call col.StampAll("Archived", 1) End If 'Pseudo-agent 2 Set col = db.Search({Form="Survey" & Open=1 & CloseOn<@Today},Nothing, 0) If col.Count>0 Then Set doc = col.GetFirstDocument() While Not(doc Is Nothing) Call doc.ReplaceItemValue("Status", "Closed") Call doc.ReplaceItemValue("Open", 0) Call doc.Save( True, True ) Set doc = col.GetNextDocument(doc) Wend End If 'Pseudo-agent 3 Set col = db.Search({DeleteMe="1"},Nothing, 0) If col.Count>0 Then Call col.RemoveAll(true) End If
So far it's working for me and making my life a little simpler, which is always nice. I'm interested to hear what your take on it is though? Is this a good or a bad idea? If bad, why?
Skip to the 
You can subscribe to an individual
Sounds like a good idea in principle unless the agent becomes too involved and exceeds the agent execution time. Each "Pseudo" agent would better reside in its own routine and be called from the main agent for maintainablilty.
This is the technique I've used for a while now. However my "Daily Tasks" agent does one thing. It calls a Script Library routine. Why? Because if the design is used as a template, when a database is refreshed the agent (if it has been changed) is disabled. You then need to go back in there and re-enable the agent. Whilst not especially time consuming, this crucial step has on occasion been missed. By putting the necessary admin tasks in a Script library you can update it as you see fit, still knowing that it will be executed as and when.
One furtehr thing I do in the library is I also break down the tasks into what day of the week it is. My Standard library has a Daily subroutine, a Monday one, a Tuesday and so on. Some tasks need to be on work days, some on the weekends, some everyday.
Like Dave S says, watch out for execution limits. But our servers are set for 30 mins at night anyway, so it's not normally a problem.
I'd not thought of the time restrictions!
Using a routine in a library is a great idea. Will start to do that forthwith. The one down side I'd thought of to this approach was the trouble you can get in to if you don't plan ahead and think about each design refresh. This solves that. I'm guessing you enable the agent on the live system and then disable design refreshes on it?
Jake
I agree about having one "HouseKeeping" agent that does most of the daily work. Script library or not.
About agents in templates: You can have an agent active in the template, if it is a .ntf file. Agent manager will not execute agents in .ntf files.
/Peter
Hi Jake,
Goes without saying - but I assume you would always include some form of logging - maybe using {Link}
Having scheduled agents which do not log there activity / errors somewhere are a nightmare to support and I don't mean the log.nsf.
If the processes were more complex my first check would be failure consequences.
Weighing up if 1 part failed and caused everything to not run - would that be a big deal to me?
Have I built in enough error trapping that it would get around this?
I'm quite surprised no one has mentioned error handling yet (although logging activity/errors is similar).
Your code snippet doesn't show any, but I assume there's some in there somewhere, right? ;)
It'd be a shame if an unhandled exception in pseudo agent #1 meant that the other code wasn't executed, especially if some of those later tasks were considered vital..
Heh Steve beat me to it.
You might also want to add a sub/function to send somebody an email if there was an error ... nobody is going to be scanning the log(s) for errors on a daily basis.
Simply enabling an agent that is disabled in the template (or vice versa) should not trigger the design task to replace it. It occasionally did in certain releases, but that was a bug. So, simply offloading as much as sensible/possible into script libraries (thus minimizing the amount of changes to the agent itself) is usually enough to ensure proper operation, without the need to disable design refreshes.
Another interesting point is the nasty habit of scheduled daily agents to consider themselves overdue, if they have been modified after their scheduled execution time. Domino 7 and up obey a Notes.ini entry
Amgr_SkipPriorDailyScheduledRuns=1
which will prevent just that.
@Fabian - I hadn't seen that INI entry - Thanks!
That'll save me some headaches after we upgrade (don't laugh - we're still on 6.5.6)
I always use a script library as well. If your application runs in on a server where you are not allowed to run agents which happened to mewith a couple of clients, the administrator has to sign your agent every time you change it. If the code is in a library you only need the administrator to sign your agent once and you can then change the code in the library as required.
Also if your agent runs on all new and modified docs, I seem to remember the document collection your agent runs on is reset when modifying or enabling/disabling the agent.
My take on this is that it makes sense to combine related agents, but definitely not agents that have nothing to do with each other. The more you combine, the less you or your admin has control over balancing load. In many cases, this may not be a problem at all.
At work, we have quite an advanced mechanism. We can schedule, enable/disable and track agents without touching a single line of code, requiring only editor access to the database. In a production environment where you only have editor access as the maximum access level, this mechanism is a blessing. I cannot share the code but there is an extended agent class. In each agent, the first line of code instantiates that extended agent object. The constructor of the extended agent object will query an agent config document that contains all the agent's settings.
I guess how advanced you go depends on the size, complexity and practical limitations of your environment. Use what works best for you.
-- Ferdy
Like many good ideas, this one is good only when you consider it in moderation. Like Ferdy, I would agree with the idea when the context matches, and as long as there is some robust error trapping in place to avoid atomic process failures. A final consideration would be criticality; the more critical the process, the more likely I would place it inside its own container.
I run a single scheduled agent that simply calls other agents configurable in a keyword document. I use this strategy for a Local Notes Client since getting any scheduled agent to run locally/successfully requires extra steps with not only signatures but also saving it with the running ID.
This method also does not care about success/failure of any single point of code(within agent) and keeps running.
Thanks guys. Some interesting point that I'll take on board.
I didn't reply to any yesterday as I suffered a crushing hard drive failure on my laptop. More on that later...
Jake
Oh, and yes, there is error handling in my real version. I just left it out for the sake of brevity.
I usually create separated Agents with calls to Script Libraries for each "process". I prefer having the code centralized in Libraries rather than in the Agents.
Now, taking into account all that was said here, is there any performance issue with creating lots of agents?
Will Notes collapse if I have 200 agents running with different schedules?
@Pablo:
We currently have around 50 client databases each containing 7 scheduled agents (so totalling 350 or more) with some running as frequently as every 15 minutes and some running daily.
Admittedly we have a dedicated server to run these, but despite my initial scepticism, Domino does seem to manage everything quite effectively.
Regarding the enabling / disabling of agents in templates, one very basic approach I've taken in agent-heavy environments in the past is to include this at the top of each Initialize:
Dim ses As New NotesSession
Dim dbCurrent As NotesDatabase
Set dbCurrent = ses.CurrentDatabase
If (Lcase(Right$(dbCurrent.FilePath,4)) = ".ntf") Then
Exit Sub
End If
That way, if the agent runs in the template copy, it exits milliseconds later... no need to disable it in the template and prevent design refresh in the live copy.
Hey Jake,
A word on db.search. I'm fortunate enough to work in a very document heavy environment. We recently ammended our standards to prefer view.getalldocumentsbykey and getdocumentsbykey over db.search as performance with db.search degrades severly as the database size grows. I even ran into a problem with a database that has over 250,000 documents where db.search wasn't even returning proper results (and I recreated the database and the indexes to make sure).
Obviously, our new approach requires views in addition to agents, but you gain scalability into the extreme scale and there are some creative ways you can keep that simple too.
I prefer to create a "shell" agent that calls full-blown processing functions in a script-lib, complete with error control/logging that bubbles back up to the sched agent. It can decide whether to continue or chain to the next agent. I also use 'new/modified docs' triggers, and as each doc is stamped with the sched agent signature when 'updateprocesseddoc' method is called, you have the added benefit that changes to the ScripLib code will not cause your agent to reprocess 'old' documents - provided your shell sched agent remains 'constant/fixed'. Just a word of caution, I always included my own process-flag on the doc as well and set it as needed. You can't always rely on the in-built notes processes - this avoids a lot of embarrasing headaches later when an agent you thought would never change, changes!
But, also note: a document stamped in one part of the process, will be stamped for all agents run in the same agent, so you have to be careful to avoid either no processing, (skipped docs) or too much processing.
Oh, and by the way:...Does anybody know how to run a scheduled agent faster than 5/6 mins on the server? (I do this now by running an external visual-basic agent-runner via COM, but would like to avoid this if possible!)
One Agent to Schedule them all...
Error trapping can be done at this level as well as each agent level. Agent names, and whether or not they should run, could be obtained from a profile document in the database with a little work.
** scheduled Agent **
Dim session As New NotesSession
Dim db As NotesDatabase
Dim agent1, agent2 As NotesAgent
Set db = session.CurrentDatabase
Set agent1 = db.GetAgent("NameofAgent1")
Set agent2 = db.GetAgent("NameofAgent2")
agent1.RunOnServer
agent2.RunOnServer
-End-
Andy: Ah-hem, just a little quibble: I think you have inadvertantly Dimmed agent1 as a variant. The correct syntax is:
Dim agent1 as NotesAgent, agent2 as NotesAgent
...if you omit the type, you get a variant by default, which, incidently should work, but possibly not what you intended.
Well, I guess it was rushed, and it was more about the technique to chain agents from a central source, hadn't really tested it in earnest...
..But yes the syntax is incorrect, as you say.
cheers
Regarding disabling scheduled agents in a template...
You shouldn't have to. If the file is a .ntf agent manager won't pick the agent for scheduling.
On scheduling multiple agents like this is never something I've really had to do. The time factor and results if one agent failed would be my biggest concern.
If using the open log you really want your code in separate agents since it will then report exactly what agent was running when logging events or an error.
And I think I'd rather just see in the database that I actually had X number of agents scheduled.. this kind of masks the reality [good though if you want to hide stuff from admins ;-)]
Scott