Meet in a neutral place and take cash only, no problem at all
... and when you set up the arrangement, casually mention that you'll be leaving your home only after your brother and his family arrive there (i.e. the house is not empty and burglarable).