It’s because the resolution in that area is too low. You can see this with a wider object:

By setting the count to 1, you’re using the same 4096 px map for the entire world, which is really big. By increasing the map count, closer objects get the resolution from the closer map which is more appropriately sized for them.