The more specific you get to the materials in your scene, the more optimized the render will be. So if you have no glossy surfaces, I believe 1 AA sample and 80 Diffuse samples would work better than 20 AA samples and 4 Diffuse samples. Exactly how much faster, though? I have no idea.
