Formula sits in the baby's stomach longer - it's much harder to digest. If you are giving your baby formula with each breastfeed, that will be sitting in his stomach and making him not feel so 'hungry'...
Also, you can try just 'topping up' at the breast - feed off one side, try a few things like ticking the feet, tickling the palms of the hands, to wake him up a bit if he does fall asleep. Then if that isn't really working, swap sides. Maybe even try a nappy change in between to really wake him up, then offer the other side.
You can switch sides as many times as you like...your baby will tell you when they've really had enough!
All babies will 'take' a bottle, even if their stomach is full - when you put the bottle in their mouth, the sucking reflex takes over, and they also suck/swallow to stop themselves from drowning in the milk! - so whether the baby 'will' take the bottle after a breastfeed isn't a good sign of whether they are still 'hungry'.
Getting your confidence back, though, is really hard, and trusting your body to provide the milk, is also really hard - I think it's important to know in this situation that it's not a lack of milk that's a problem - it's a very sleepy baby with jaundice...who was under-demanding.
My mum was told to give up on breastfeeding me because I was an underweight, c-section, jaundiced baby - instead she spoke to a breastfeeding counsellor, and they suggested to try offering the breast ever 1-2 hours, and not stretch the feeds too much - I would never let a newborn go more than 5 hours between feeds... and in a week or two you will have a much more 'demanding baby'.
In this situation, having a more 'demanding' baby is not a sign that your milk is running out, it's a sign that the baby is now 'waking up' from the newborn sleepiness, and has the spare energy to make those demands - if you just breastfeed every time you think the baby could be vaguely hungry...you may find that the situation turns around very quickly.
At the moment, the baby's stomach is no bigger than their fist - probably even smaller - so they need to be fed frequently.
If you do hang in there, I wouldn't be surprised at all if you post back in about a month (when the baby is 6-8 weeks old), complaining that you now have 'too much' milk
I would definitely recommend seeing a lactation consultant if possible. Paediatricians generally have minimal breastfeeding training, and not all 'maternity' nurses are breastfeeding-friendly. Also, a lactation consultant should give you 'options'...