You need to make plans for your future, so plan your own future.

There has been, for some reason (or more likely an unfortunate accumulation of reasons) a trend over the past several decades for parents to do the work of parenting in the isolation of their own homes - and not only that, this trend has overlapped with the other trend of much deeper parent involvement in raising kids. That you also represent trend No. 3, more people raising kids solo, has only exacerbated a close-to-no-win situation.

It takes awareness that it's not only not a bad thing to let others do things their own way, it is in fact an improvement. It makes life richer and more interesting.

I believe in innocence until there's proof of guilt and all that.

It's probably good for your body and brain to get moving occasionally.

One way to make tough decisions is to take incremental steps that don't commit you to anything yet.

Plan your own vacations when you want to, and plan a suitable combined vacation with this other family when you want to. If they freak out at your planning your own vacations as you see fit, then let them. Bowing to unreasonable demands because someone will make you pay emotionally if you don't is not a healthy option.