Most likely it’s the collision distance, this is the distance from the collision object (character) when the cloth will form it’s shape over the object.
In this case you want the cloth to drape over the character more naturally, so you’ll need to lower the collision distance.
You may need to adjust the Impulse Clamping and/or adjust the Damping settings if the low Collision Distance causes the cloth to behave erratically.
