While I could have used IE or Chrome instead, Firefox is the only one for which Gears and HTTPWatch work together and you really need a HTTP sniffer if you're debugging Gears apps!

Luckily, this morning I managed to downgrade back to 3.0.11, which is a simple case of downloading and running the installer. You don't even need to uninstall 3.5. That's one of the things I've always liked about Firefox - it never loses your settings.

Will Firefox 3.5 Even Need Gears

One thing I did get out of my brief look at FF3.5 is the (exciting) knowledge that it's moving towards implementing its own kind of Gears. I've not digested it all fully but if you read the DOM Storage page on Mozilla's site and the W3C's Web Storage page you'll see what I mean. It looks to me like browsers of the future will all have local databases to use for persistent data across sessions. Hopefully without the need for 3rd party plugins.

More To It Than "Going Offline"

While using Gears this last month I've been thinking more and more about what other uses it could have, apart from "going offline".

All you have to do is imagine you have a local SQL database that is easily accessed from JavaScript and you'll get an idea what you can do. It's like having a massive cookie to use for whatever you like. One example would be to keep a cache of values which are regularly used in forms/type-ahead by the user but which rarely change at the server. Why keep fetching them with Ajax when you could get them in an instant locally!

Hopefully I'll get chance to show some alternative usages of Gears shortly.

There's nothing wrong with the phone itself, per se. I love the look, feel, size, usability etc. It's just the software on it that sucks.

Yesterday I was trimming my finger nails and was conscious of not going too far back on the thumb as I need it on a daily basis to remove the battery cover to do a hard restart!

It seems to crash at least twice a day. Seemingly for no reason at all. Although it has been a bit better since I stopped using HTC's customised home screen, which I prefer to what I have now, so that's a pain too, but there you go.

It even managed to lose all my email and text messages on one occasion. Not a big deal, as I had copies of the emails in Thunderbird too, but the fact it has the potential to just forget all email account settings and delete the messages is somewhat worrying.

I've tried using Windows Update to get any "service packs" there might be but that too fails. Something about the date and time being wrong on the device. Great.

Who To Blame?

Things that don't work really get on my nerves. I don't mind (and expect) the odd bug in software but when I have to mess about removing the battery cover and waiting for it to startup on such a regular basis it gets a bit daft.

If this were any other piece of electronic equipment that I'd spent +?ú200 on (camera, DVD player, whatever) then it would be straight back to the shop and I'd be demanding my money back. So, why do I feel reluctant to even try and return the Snap? As if whoever I speak to is just going to say, derr, it's Windows, what did you expect!?

Who should I contact -- the seller, HTC or Microsoft? I guess I should start with the seller? I think I'll start with them and see how far I get.

No Way Back

Having used a smartphone for a couple of weeks I can now see that I need one and don't want to go back to not having one. I just think that Windows Mobile probably isn't the way forward. Perhaps Android is...

Here's the code:

var re = /^(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])((19|20|)\d\d)$/;

//Test format first
if (re.test(input)) {

 var bits = input.match(re); 
 if (bits[4]==""){
  bits[3]="20"+bits[3]; //Will last 91 years before breaking!
 }
 
 var entered = new Date(); 
 
 entered.setFullYear(bits[3], bits[2]-1, bits[1]);

 //Validate range here if needs be
                                                        
} else {
 alert("Not a valid date!");
}

The end result is a JavaScript Date object representing the date entered, which you can do with as you must.

When Hard Coding Is Ok

What keeps making me think about the code is the bit where I've hard coded the 21^st century in to any date where the century part isn't specified. In 91 years it will break. As an opponent of all things hard coding I can't decide how happy I am with it.

Although I know the application itself will, like me, be long dead by the year 2100 it's just bugging me. Is it ever ok to knowingly introduce a bug, even if you know the bug will never get to live?

A Simple Fix

In this case it would be trivial to remove the bug, so that it always added the current century when none was specified:

bits[3]= new Date().getFullYear().substr(0,2) + bits[3];

I just can't decide whether I should retrofit this fix or not. Maybe I should. Then, should a source of eternal life be found by then, I'll be guilt-free.

Anyway, here are the email addresses of the "people" who posted on Rockall's contact "us" page over the last 24 hours. Rather kindly, I always think, the spammer make it pretty obvious it's spam.

The Challenge

Your challenge is to make a short story that includes all of the words used below and in the order they appear!

present@rain.com
system@claim.com
shine@sheet.com
five@such.com
milk@tail.com
ground@character.com
fruit@safe.com
island@search.com
difficult@during.com
problem@felt.com
carry@size.com
pick@warm.com
while@wind.com
believe@back.com
bell@difficult.com
decimal@invent.com
since@state.com

Extra point for anybody who can make it topical and about Michael Jackson (the word "five" is in there, how hard can it be!?). The less extra words you use to pad out those above the better.

It turns out that Scheduled Agents have their own document/storage area that you can add to and read from. I never knew that!

You get to it via the SavedData property of the Session object (in LotusScript). See help page on it for more.

With a bit of tweaking I re-wrote the code I mentioned yesterday to more accurately find the documents created since it last ran, which is posted below.

This gets round the fact that the agent's LastRun property is the time it last finished. Any documents created while it ran won't ever get processed. Using the agents "saved data" area we can add a more reliable time to state when it began.

Set thisAgent = Session.currentAgent
Set agentDoc = Session.SavedData
        
Dim query As String, datepart As String
        
'Default date string
datepart = "@Adjust(["+Cstr(thisAgent.LastRun) +"];0;0;0;0;0;-30)"
        
If thisAgent.HasRunSinceModified Then
 If Not agentDoc Is Nothing Then
  If  agentDoc.HasItem("LastBeganAt") Then
   datepart = "["+Cstr(agentDoc.LastBeganAt(0))+"]"
  End If
 End If
Else
 datepart = "@Adjust(@Now; 0; 0; -1; 0; 0; 0)"  
End If
        
query = {Form="post" & @Created>=}+datepart+{ & Approved="0"}
        
agentDoc.LastBeganAt = Now 'Set this after query is built!
        
Set collection = db.Search(query, Nothing, 0)

'do stuff here

agentDoc.Save(True, True)

Forever Learning

How long have I been using Notes now!? Must be getting on for, if not more than, ten years. How this nugget passed my by I don't know. Going back to Monday's whine this is an example of one of the things I do love about this site. Without it I wouldn't know half what I do. The learning process is two-way! Thanks Rob.

Inside the code we then just access a NotesDocumentCollection of all the new documents, like so:

Dim collection as NotesDocumentCollection
Set collection = db.UnprocessedDocuments

All is well, until you want to restrict the documents you process. To do this the Notes Way you add a Document Selection criteria to the Agent, like so:

So now the collection of documents will be restricted to those built using the Form "post" and those where the field called Approved has a value of "0".

At least that's the way I've always assumed it to work. I always have trouble getting it to work as I'd expect it to though. Recently I cooked-up an alternative that I think is much cleaner and less error-prone.

The Alternative

A newby to Notes looking at the code above might wonder how on earth it works. It's not unless you know about (and notice) the Document Selection part that you'll see how the document selection is being made.

Another, much cleaner (in that somebody reading your code can see what's going on directly from the script), method is to build the document collection ourselves, based on the time the agent last ran.

Here's how:

Set thisAgent = Session.currentAgent
        
If thisAgent.HasRunSinceModified Then 
 query = {Form="post" & @Created>=[}+Cstr(thisAgent.LastRun) +{] 
  & Approved="0"}
Else 
 query = {Form="post" & @Created>=@Adjust(@Now; 0; 0; -1; 0; 0; 0) 
  & Approved="0"} 
End If
 
Set collection = db.Search(query, Nothing, 0)

Et, voila! You can now build the collection exactly as you want it and it's always immediately obvious what's going on to the next person.

Notice how, if the agent has been modified then we need to cater for this as the LastRan property can be used. In my case I just look for all documents in the last 24 hours.

A Little Gotcha

I don't know how the logic behind the Notes Way works but there's a gotcha in this method. If, for whatever reason, the agent takes a long time to run one day then the LastRun property could be, let's say, 10 minutes after it started. So, any documents created in that 10 minute gap won't be included the next time the agent runs.

You can get round this by wrapping the LastRun date with an @Adjust like this:

@Created>=@Adjust([}+Cstr(thisAgent.LastRun) +{];0;0;0;0;0;-30) 

I've taken 30 seconds off. How many you do depends on how long you think it might take.

Puttin It To Use On CodeStore

I first put this technique to use on a "document indexing system" for a client. I then remembered it and used it last week while making a change to the way codestore's backend runs.

Until last week every comment posted on this site triggered an individual email to me. This meant I spent a lot of time deleting email from spammers who often attack in waves of 30 or more messages a time. This distraction cost me time and hence money.

I now have a two-hourly agent that sends me a summary email of unapproved comments (those on blogs older than a week). I then glance over it to look for real names (surprisingly obvious which are, as most my spam comes from the likes of "zyzuaeyuwqhun"). I now delete one email every two hours rather than 30 or more. Doesn't sound like much but it has made a big difference.

The down side is that legit posters on old blog entries have to wait up to 2 hours to see it go live. This happens so rarely that it's not really an issue.

My fight to keep the site CAPTCHA-free continues!

One year on and she's turned in to a beautiful little girl. Maybe you shouldn't say it about your own child, but Minnie is probably the cutest baby in the world.

She's not a baby any more though and that's what I have to keep reminding myself. A year is a long time for a baby but flies by for the parent. It seems like last month she was born.

Later on today I'll try and get a photo of her in the same style I've been doing each year with Felix. That's if I can keep her still long enough. She really doesn't like being made to lye down against her will.

That's the other thing that you start to see in children after a year - their personality coming through. I can already see that Minnie is going to be a fun and caring girl but also have strong will power and, in later life, I think she'll be the kind of girl who knows how to get exactly what she wants.

Happy birthday Mooker!

Ok, but as long as you're not just dicking about on your website!

At this point I'll remind her that, without codestore, my company wouldn't exist and we'd probably not be able to afford the house we live in, lifestyle we live etc etc. More importantly we wouldn't have the quality of life that me working from home affords us.

Codestore & Rockall

There's no doubt that Rockall exists because of Codestore. In the first place that is. Whether Rockall continues because codestore does I don't know. I tell myself it does as this is what keeps me going with this site at the time when I wonder whether I should or not. The idea that, without the site, I would lose customers and end up like any other non-descript independent contractor and back on the open market place desperate for work isn't one I cherish.

While it's true that all my customers are in some way connected to this site it's also the case that 90-odd % of my income comes from about half a dozen repeat customers. I've had this same set of customers for the past five years or more. Although I do get new customers coming along now and then, most of them are people who have been with or known of codestore for the duration. None of them are "new visitors" who have happened upon the site and then approached me for my services. This leads me to wonder whether codestore is really the business marketing vehicle I think is.

I've been thinking about this more and more recently as I continue to contemplate the site's future. As it is I'm becoming less and less happy with the quality of the site and am getting to the point where I'm wondering why I do it and whether it's always better to quit while ahead.

Change of Direction

Half the reason I am becoming less happy with this site is that I don't feel it really reflect either me, my personality or what I'm really interested in. Following on from the idea that I need this site to continue working from home comes the idea that this site needs to be professional. For this reason I've put less of myself in to it of recent. That's never a good thing for a blog.

The other notion I have is that I need to keep you guys happy. Happy readers = more readers = more business. But then I've not had any new business come about from my activity on here for years now. So, who am I kidding?

What I need to ask myself is not so much why am I blogging but who am I blogging for. Should I be blogging for me or for you? The answer is probably a bit of both, but over the last couple of years I feel like I've been trying to make it more about you and so my interest has waned. Unless it's fun for me then there's little incentive.

Blogging as Fun

My thinking about this was brought to a head on Friday just gone when I posted an entry that received "No Comments". This is the first time that has happened pretty much since I began blogging. Although I can see why it got none (it's very specific and of little interest to you guys) the fact is comment numbers have become something of an obsession to me. So Friday was a bit of a welcome relief as I can now go forward from a clean start and worry not about how many of you respond.

Moving Forward

Friday's blog entry about Google Gears was a good example of what I've not been doing through fear of upsetting/losing my readers and that is talking about something you don't appear interested in. What I should be doing is talking about whatever it is I am interested and/or involved in at the time.

Within a day of Friday's post Google had it as the top entry for the same query I used in vain to try and find the solution the day before. That's what blogging is about for me. The fact I've added back to the community from which I take. Hopefully it will help somebody else out and then I've done my bit and will feel satisfied.

Flex is another example of something I've talked about recently but then stopped as I didn't sense much interest. The effect then is that I continue working away with Flex but not talking about it here. Instead I talk about nothing and it looks like the site is going to the dogs.

Rather than have long silences on here I think I'm going to try and talk about more random/fun things and more of what I want to talk about. Without worrying about it from a professional aspect or whether I'm keeping you all happy. Hopefully I'll get back to making a post a day. I do have plenty of stuff to talk about. Just maybe not what you want to hear all the time.

The title to this blog has a double-meaning. Not just about the notion that Rockall is only because of codestore but also that what goes on my website should be my business and not so much yours. See what I did there!?

Let's see where we end up in six months... hopefully I'll be putting far less emphasis on comment counts by then and having fun with this site once again.

My problem started when I wanted to create a local cache* of a file at a URL like:

http://www.mysite.com/mydatabase.nsf/MyForm?ReadForm

* Although I use the term cache, Gears isn't actually adding the files to the browser's cache. Instead it keeps them in it's own folder for serving up from its own "LocalServer" when offline.

I created a manifest file which looked a bit like this:

{
  "betaManifestVersion": 1,
  "version": "v1",
  "entries": 
        { "url": "MyForm?ReadForm", 
                "src":"MyForm?ReadForm&offline=true", 
                "ignoreQuery":true 
        },
        { "url": "gears_init.js" }
    
}

It didn't work and neither file were "cached" and, subsequently, going offline failed. After what seemed like hours of hair-pulling agony I worked out why.

If you are using the ignoreQuery option for a URL entry then you should not add a query to the URL part of the entry. The correct manifest would look like this:

{
  "betaManifestVersion": 1,
  "version": "v1",
  "entries": 
        { "url": "MyForm", 
                "src":"MyForm?ReadForm&offline=true", 
                "ignoreQuery":true 
        },
        { "url": "gears_init.js" }
    
}

I guess it makes sense really. If you're telling Gears to ignore bits like ?ReadForm why add it to the URL stored in the cache in the first place!? Just doesn't seem obvious to me. Hope that helps somebody else out.

Note that ignoreQuery only applies to the URL specified in the manifest entry and not to the SRC URL that is actually fetched and stored!

By specifying ignoreQuery the following URLs will all open the same page when in offline mode:

http://www.mysite.com/mydatabase.nsf/MyForm

http://www.mysite.com/mydatabase.nsf/MyForm?ReadForm&id=1

http://www.mysite.com/mydatabase.nsf/MyForm?ReadForm&id=2

http://www.mysite.com/mydatabase.nsf/MyForm?OpenForm&new

If we hadn't specified ignoreQuery only the first of the above URLs would work.

Case Sensitivity

Another problem I wanted to find a solution to was that of case-sensitivity. Gears' LocalServer is very sensitive about the URL you use to get back in to an app in offline mode, should you have closed your browser since going offline.

With the above manifest in mind the following URLs won't get you back to the app in offline mode:

https://mysite.com/mydatabase.nsf/myform

This URL would fail for three reasons.

1. The use of HTTPS
2. Lack of WWW from domain part
3. Lower case "myform"

The URL must be exactly the same. You can see why if you look at how Gears stores the files in the OS:

See how the folder structure is strictly dictated by the URL used to go offline. Even down to the port number used!

You can however cater for any known cases where casing of the URL might matter. For example, let's say you know there are users out there alternating between the following URLs:

http://www.mysite.com/mydatabase.nsf/MyForm

http://www.mysite.com/mydatabase.nsf/myform

If they went offline at the first then they might expect to get back on if they typed in the second, no?

To cater for this you can use the following manifest file:

{
  "betaManifestVersion": 1,
  "version": "v1",
  "entries": 
        { "url": "MyForm", 
                "src":"MyForm?ReadForm&offline=true", 
                "ignoreQuery":true 
        },
        { "url": "myform", "redirect":"MyForm" },
        { "url": "gears_init.js" }
    
}

Notice this second entry added. If the user typed the page name in in lower-case the Gears LocalServer would send the browser a 302 redirect message to open the proper-cased URL and all would be well.

You could keep adding entries to cater for other options like "myForm" but where you stop is down to you.

When Things Go Wrong

While testing offline mode I had lots of failed attempts at creating local versions of the required URLs. If for any reason a version failed then the easiest way to get back online is to remove the files stored. To do this you need to find the files and then remove the localserver.db file from the Gears folder and then the folder called "managed" inside the folder it keeps for your site/app.

I'm confident enough with SQL to get most things done but get a bit stuck when it gets advanced (for me at least).

There's an application I've been working on that has about five inter-dependent tables - joined by an ID from one that matches another. It all works as expected, just not as fast as expected as the amount of data grows. This is where my knowledge falls down.

At the moment I'm struggling with working out how to create an index to make a query run faster. Specifically a query that has an ORDER BY clause on it. With no ordering it takes about 0.0001s. With the ordering it takes 0.3s. Still not long but that's on a smallish dataset. As it grows so will this time.

I kind of get the principle of the index and why I need one in this case (the column sorted on isn't in any index) but I'm struggling implementing it and think I either need to go back to first principles or get/pay somebody else to sort it.

Anybody?

If you need to see a schema I can post that here.

Update: I just asked the question more specifically over on stackoverflow.com, which seems like an increasingly useful resource, so I'll see how that goes.

Update 2: No more than 30 minutes have passed and stack overflow has answered my question. It's the first question I've asked there and I have to say I'm very impressed. That's how technical forums should be built Lotus!!