Yes, they take a toll for sure. I remember Shawn’s Blender taking 5 minutes to load because of the many addons he had enabled. Not to mention the many addons that cause glitches and make Blender misbehave and cause all kinds of headaches. I typically have 3 or 5 must have addons enabled at a time, then what I need I enable to use and then disable it. But there is not a hard rule against addons, try things out, see how it works for you.