Plan your own vacations when you want to, and plan a suitable combined vacation with this other family when you want to. If they freak out at your planning your own vacations as you see fit, then let them. Bowing to unreasonable demands because someone will make you pay emotionally if you don't is not a healthy option.
Carolyn HaxIt's hard to send your baby off on a plane without you, though that's less reasonable, because sending him off in a car is statistically a bigger risk.
Carolyn Hax