Only speaking from my experience, in my universe:
Most Gems were visibly alien to humans, to a fairly significant degree. Even the most human looking Gem with the most natural skintone would have an uncanny, unearthly look to her. For one thing, while Gem bodies could be soft and pliable, we didn’t have pores or body hair, so our ‘skin’ was very shiny and smooth, and reflected a lot of light. There was always something about Gem eyes, too, our eyes never looked quite human. Mine didn’t have normal pupils. They were pale pink, with silver-white toward the center.
Most Gems had bodies that were somewhat unnatural for humans as well, either larger, or smaller, or strangely out of proportion. Not to mention our skin colors could be well, anything. I was bright white with a translucent blue glow. Amethyst was a kind a matte purple color. Garnet seemed to shift in tone depending on what angle you looked at her from.