Hi Josh,
You’re right, however the order of the modifiers matters. The ‘Smooth By Angle’ modifier should be at the bottom of the stack.
In the order you have, you’re smoothing then adding the Boolean. With the Boolean first then smooth you’ll get the correct result.