Ok, so that seems like a bug. I baked it multiple times (also super slow on my PC) and got the same bad result. I did manage to get a better result by setting the extrusion to 0 and switching to CPU rendering, which in theory should not have any effect whatsoever. It still took ages though, but by copying and pasting the objects into a fresh .blend file, I was able to bake much, much faster and without error.
