Thanks a lot, for anyone who has the following issue :
You baked you curved meshes and end up with weird UVs.
Select all the mesh in Edit Mode > Merge By Distance (basically you want to remove the bevel you added when doing the vertex painting) and bake again, it fixed the issue for me 🙂
