Sounds like a job for the Light Path Node. You can probably search for videos of it on YouTube, but I like this very old one that goes super in depth:
https://www.youtube.com/watch?v=MTFxyNzI5EE
It’s a super hack, but sometimes you need the artistic freedom and break the rules of physics.
