Yes, you model with mirror since like 97% of the model is symmetrical, then when it comes to making that small part that has asymmetry, you apply the mirror modifier and then model that part with no mirroring. One thing I like to do is duplicate the collection before I make any destructive changes to the model and save that copy on a “Trash Bin” Collection, just in case I have to go back to the before state. Or alternatively you can just do a Save As and leave a file before you make destructive changes to the model.
