Doesn't matter what you do, or how you do it, your neighbors are gonna talk about you anyway.
Life already has so many boundaries and pressures - why add more in the garden?