Plan your own vacations when you want to, and plan a suitable combined vacation with this other family when you want to. If they freak out at your planning your own vacations as you see fit, then let them. Bowing to unreasonable demands because someone will make you pay emotionally if you don't is not a healthy option.
Carolyn HaxI'm sure there are people who can toggle quickly from all-in caregiving to structured socializing, but I can't think of any offhand.
Carolyn Hax