The only time I saw it go wrong was in an old build on gcc, where it was getting optimised incorrectly. That’s why I changed it to use a union, which should be guaranteed to do the right thing. What platform are you on?
I guess it could be a struct alignment issue - if the union it uses got misaligned then that could cause it. Would expect it to be the same in debug though.