My impression has been that due to the society's multi-lingual nature, standards for "knowing" a language are high.
Almost everybody who's gone to school will know at least a smattering of French and German, but they won't say "yes, I speak French" unless their French is really good. The idea seems to be that once they declare themselves French-speakers, you are free to address them in French and expect them to understand everything.
So that spills over to foreign languages. Especially shopkeepers seem to fear that as soon as they say "yes, I speak English", the next thing out of your mouth will be "knowest thou whither yon argosy fareth?" and it will be their fault if they don't get it.
Many people on the street (in Zurich) do speak enough English for you to get by with them, but it's not rare to find one that doesn't at all; if that one happens to be the one person you need to talk to right now, then oops.
In the short term, it's good to know a bit of basic German to start a conversation and then politely ask if they speak English. Most Swiss (at least the ones in Zurich) appreciate it if you're clearly putting some effort to adapt and not coming in with your nose held high and expecting them to speak "your" language for you.
In the long term, you can get by in Zurich without German, but you can't really live . You would be forever on the margins, isolated from the city's heartbeat, hanging out only with small circles of people.
Standard German is IMHO better than Swiss German for this. It's easier to find quality instruction for it before you arrive, it will be useful outside Switzerland too, all government stuff is written in it, and the locals' prejudice against it mostly only applies if you speak it very very well
Note language instruction here is quite expensive. As Jrspet said, get as much of it as you can before you arrive.