Some people can work amid chaos or conversations, and some can't - and while there's no doubt an element of brain wiring to it, there's also the possibility of acquiring skills that improve your focus.

If you take the time to listen to an upset child's story with empathy, and guide the child toward figuring out the root of the problem, then the result is often that the child not only calms down, but also in the future is less likely to get so upset.

People who make babies surrender their right to behave like them.

Plan your own vacations when you want to, and plan a suitable combined vacation with this other family when you want to. If they freak out at your planning your own vacations as you see fit, then let them. Bowing to unreasonable demands because someone will make you pay emotionally if you don't is not a healthy option.

When in doubt, respond to what you witness, not what you hear secondhand.

Make sure you have legal cover for what you're doing.

There has been, for some reason (or more likely an unfortunate accumulation of reasons) a trend over the past several decades for parents to do the work of parenting in the isolation of their own homes - and not only that, this trend has overlapped with the other trend of much deeper parent involvement in raising kids. That you also represent trend No. 3, more people raising kids solo, has only exacerbated a close-to-no-win situation.