Indeed, they're not "inverted", but swapped with their complimentary colours. Staqring at the image for a while brings the brain to compensate for the colour saturation and in so doign corrects the greyscale image into the full colour one you see in the end.

You can see the process undo itself somewhat too, if you look slightly off centre briefly and then back after you switch to greyscale you will see the colours still, but with much lower saturation as your brain readjusts to the real image.