Yeah that’s how it works, only the active object, and it is the same for most things in Blender, holding ALT and selecting adds/changes the selection for all the selected objects.
When it doesn’t add to any, you probably selected everything with A or used box select or some other method of selecting everything and no object is marked as the active object.
