No, I don’t think there is a way to select faces based on their color other than by the material. If you think about it, in your situation where the color comes from the UV, is not that the faces are of that color, but that the face is pointed to that color. And I had never heard of face maps, I’ll have to look what’s that all about.
So you can stick to vertex groups for selecting the faces you want over and over again. You can also modify, add or subtract from the vertex group which makes it very handy.