Ok @daddios I converted mine to a multi-res Model (I am at 278 Faces). I will work on it a little more because I am not quite happy with the silhouette yet but I tried baking from Multi-res and it worked just fine for me even with the scale adjusted to the placeholder cone (around 1000m x 1000m).
I just followed this short from blender secrets and I had no problems:
