I started to use ngons because I couldn’t build one part of the mesh without messing up another part or adding an absurd about of unnecessary geometry.
Hmm, you could try splitting the faces around the detail into triangles (Ctrl+T) to stop any loop cuts from messing with that area.