Generally speaking you can always smooth your masks by pressing A and dragging your mouse to the Smooth mask option. This should help with a smooth falloff between masked and unmasked areas.
But if you post an image of the kind of rough result you’re talking about then maybe I could offer more specific advice.
