Yes, serious concerns. Your son will, most likely, need to cross borders many times during his life. He won't want to be potentially burdened by having to show a signed letter from Mum and Dad, every time he does so. For as long as he is a child, you, as parents travelling alone or together, will - with an arrangement as you propose - add that little bit of stress to your lives, wondering, each crossing, whether it's all going to work out okay. Some countries will not to acknowledge a name in the format Smith-Brown, and could insist that he prove that his name is either Smith or Brown.
Yes! This! As Jim says, the laws about international kidnapping are - rightly so - increasingly strict, and you don't want, ever, ever, ever, to be in the situation where either of you could be charged with so doing.
Yes, with travel, but also just living in one place. To illustrate:
Even locally and in everyday life, when your child's name goes on lists, anywhere, there are going to be only some people who put his name down as Smith-Brown (the way you want it). Many others, however, will put it down as Smith Brown, others will put him under just Smith and yet others under just Brown and some under Brown Smith.
As an example, he won't be happy if you've already long signed your parental permission for him to go on the bus to the sports competition, but on the day the driver can't find him on the list and says he can't take along school-children whose names aren't on the list in the right way.
Some computer systems won't see the hyphen. Others will not accept the name at all if it has a hypen (as a non-aphabetic character) in it. Your adult son will be unhappy, to say the least, if he books himself a flight, and that booking system appears to accept Smith-Brown, but when he gets to checkin they require a passport showing only Brown, or they won't let him board. A letter from his parents won't help him, then.
If you do burden him with a double-barrelled name, the poor boy - and the poor man - is likely to be doomed to explain this, his whole life long. He will have to insist (as will you as parents on his behalf, while he's still a boy) that yes, his name really is Smith-Brown, and yes, he belongs to the group starting with "S" and no, he is only one person and not travelling with Mr Brown, as well, and yes, they must look again because he really did book/register/pay and yes, his name must really be on the list.
He may find himself in the uncomfortable position of having a passport in Smith-Brown, but his visa or residence permit in another country having been issued as only Smith, and not being able to re-enter the country.
His only escape from this lifelong recurring trouble, would be to try to get his name changed. Depending on the country in which he is living when he reaches adulthood, he may not even be permitted to make a change and set himself free.
I would urge you to reconsider and give your child the best start by assigning him a straightforward non-double-barrelled name.
Source: experience.