I figured out the problem. I had deleted all the other axis in the key frame except Z since the handles were only going up and down. I am not sure why but after some trial and error I found out that the Y axis is what allows them to move up and down on the Z axis in this case.
