Hello Jahim, when you say that you made the floor passive, I guess it is a Rigid Body. But the shirt is probably a Cloth.
They do not work together. Make the floor a Collision Object. You can remove the Rigid Body or leave it on, if you want.
That should do the trick( I hope).
