I think the problem is mostly the topology of your cloth mesh. Stretched diamonds and triangles along the edge are not such a good idea.
If you look at the mesh in weight paint mode it gives you an idea of how the pinning weights are stretching along the mesh. If you add a subsurf, or subdivide the mesh you’ll have more luck, but again, those traingles will not subd so well.
Hope that helps!
