Yes, you should keep toping the water. But, nevertheless, be careful. It is the kind of experiment that may go wrong. If you have good connections with any biology lab, ask them if they can autoclave the tin. It will be much easier.
You may want to try the ulce de leche that Co-op sells under the Fine Food brand. It is quite close to the real thing. Or, if you can find cajeta (the same, but for Mexico) it is also good.
By the way, the real thing is done by slowly heating extra sweetened milk to a sort of syrup or dense fudge is similar in taste to caramel. Just make sure you stir at all time to avoid burning the bottom. Sometimes half goat, half cow milk is used.
It's not the water levels but the can exploding. Also, there is no need to fast boil the tins, tins ticking over (simmer), keep topping up water. Leave for 2.5 to 3 hours. I'd go for the 2.5, I have seen caramel bombs exploding and it aint pretty.
I will be heading back from Norway in early August - small-scale import possible. It lasts a long time so 2-3 days in the post wouldn't do any damage (or might provide fodder for the "expired foods" thread ).
I really like the tang of salt contrasted with the caramel flavour. In recent years they started making variations including a chocolate version - I think it was a flop. You definitely need to use a cheese plane to get thin slices, or your mouth will be supercheesed.
I'm also having vague memories of seeing it once in Globus or possibly Jemoli, or closer to home for me Mölkerei Neff in Wald. I'll check next time I go in. The recipe made me think "your right to a less healthy life"..watching a pot on the stove at home trying to make cheese isn't quite the same if not tending goats at a mountain hut. I think the end product would be closer to what is called messmør or messmör (Swedish) - a spreadable version that is slightly saltier.
ISTR that there was a cheesemonger in Berne during the weekend market that carried some of it. 'Twould be nice if a local there could confirm. It was the market on the street, not the plaza..