I seem to remember their eyes slowly drift to one side

I looked into it and yes: https://1ft.io/proxy?q=https%3A%2F%2Fwww.nationalgeographic.com%2Fanimals%2Farticle%2Fflatfish-animals-science-colors-flounders

Flatfishes don’t start out flat. They start out looking like regular fish, kind of diamond shaped, and “as larvae, the eyes are in regular position on each side,” Burgess says.

As they develop “the eye begins to migrate, moving over the top of the head, eventually settling on one side or the other,” Burgess says. This also requires the bones in their heads to move. (Read about an early flatfish whose eye hadn't quite finished migrating.)

The flatfish’s bones are pretty pliable at this point, like the soft spot on an infant’s skull, so “as the eye moves, the bones in the head warp in that direction,” Burgess says.

An additional bone, found only in flatfish, develops right under the migrating eye, giving them that goofy asymmetrical look. (Watch a time-lapse video of the flatfish's amazing transformation.)

I also found interesting how they otherwise function:

Thanks to those pop-up peepers, “they can’t see the bottom directly underneath them, but they can see the bottom around them,” notes Jackie Cooper of the National Aquarium in Baltimore.

Their eyes can also move independently, widening their field of vision, Cooper says.

Once flatfish eyes get the lay of the land, they message the brain, which in turn sends signals back to the skin. This organ contains color-changing cells such as melanophores, which either expand or contract according to the background the fish is trying to match, Burgess says. (See pictures of fish glowing in neon colors.)

For instance, expanding their cells would make their color darker.

All this neurological relaying is “a pretty sophisticated thing to do,” Burgess says—not to mention it takes flatfish between two and eight minutes to blend in.