Hi Savvas @savvas ,
Hard (read: impossible) to tell from your description alone. Could you provide your .blend file (File > External Data > Pack Resources), please?
Also, it’s not unlikely, that Baking it in other Software, like in Substance, won’t make a difference. (You’d even add an extra layer, Import/Export, where things can go wrong.)