That looks like the faces are folded onto themselves. Like when you take a sock and flip it inside out. Faces that should be in the back come up to the front as they intersect and it all makes a mess. Usually going over the messed up part with the smooth brush does the trick to un-mess the mess. Try it out and see if it helps.
