I live very close to Olten (its 12min with the bus to the main station and 5/6 minutes with the car) and commute daily to Zürich and before I was working in Basel and used to commute to there as well. Like many have said, the connections are very good if you want to visit those cities.
I'm originally from Belgium so, Olten doesnt strike me as ugly or dull.
But I agree with one of the people here saying its better to rent for a year and see how it feels and if you like the see. Luzern is much more beautiful but you have to calculate the train costs which are very high in Switzerland.
In Olten, having a beard and having an asian type will not be an issue from my point of view as there are many caucasion here these are mostly from the balkan countries and mind their own business.
I have lived in Olten the past year. I had planned to stay here for a couple of months initially as it is close to my work and the move to Basel or Zürich. However, the town is bigger than I had initially anticipated. The old town is quite nice and the outdoor swimming pool was a lovely surprise during teh summer. That said it isn't as beautiful as some of the other towns but the big advantage I have found is it is so well connected to the surrounding cities with Bern, Basel, Luzern and Zürich within 30 mins.
I decided to stay here for this reason, I was able to go to any of the aformentioned cities at the weekends or for any evening events that may arise as well as being able to get from Olten to some of my favourite hiking/ running trails at the weekend much quicker from Olten than from Zürich or Basel.
I work in Olten and have a long commute daily, will consider moving next year to reduce the commute. As mentioned, the old town is nice, and I have a great view of the river, old bridge and hills from my office. Best view I have every had in any job.