Blender fires a light ray into your scene. If there is nothing there to render, it takes a while for Blender to decide to stop continuing searching in thin air and fires another ray into another direction. Before firering a ray, Blender doesn’t know yet, there’s nothing there.
I believe the distance a ray travels is determent by the camera clipping. If Blender has reached this distance, without finding any objects, it shows transparency or whatever, depending on your render/world settings.
I’m certain someone else can explain this in more technical terms but this is more or less how Blender works.
Also, before rendering, Blender builds your scene for rendering and that takes time too.
That’s why 100 % alpha’s, also takes time to render.
If there are a lot of empty frames you might consider to skip them from rendering. I also encountered situations like this and sometimes you need those empty frames to match your clip in the final editing with other clips.
