Builder doing work without final agreement

I remember there was a similar thread on here about a gardener billing for work that hadn't been explicitly agreed. Keen to get your thoughts on whether (1) I am just not accustomed to instruction of work in Switzerland or (2) there is something a bit strange with this situation:

I asked a couple of builders for work to our outdoor garden and in/around the garage. They provided an offer for all 3 distinct bits of work. I asked one of them when they could do the work and they replied saying " Die Arbeiten können voraussichtlich ab Di 25. Juli ausgeführt werden". I replied along the lines of that timing works for us.

In my mind this was just me assessing price, availability and timing options amongst a couple of different firms.

It turns out that a few weeks later the builder has completed the work over a 4 day period when we were not at home (access granted by upstairs neighbour).

We had no other communication with the builder other than I actually emailed him on the day they started work asking them not to go ahead because we had to wait to assess options. They had already completed one day of work by then (albeit I did not realise).

What do you think? I was expecting a bit more confirmation/communication before work had started but maybe I'm losing my mind and should have communicated more clearly.....

It sounds close to

- When can you start?

- 25. July

- That works for me

I.e. a firm offer of work. In my opinion it will be hard to argue otherwise.