Interesting observation! My hypothesis is that babies love outlets for the same reason they love little ants crawling across the floor, cracks in the wall, etc. They are an intriguing -- even distracting -- interruption of the normal.
As to outlets being installed upside-down, I believe that it is normal to install an outlet upside-down only if the outlet is partially switched. E.g., if a switch in the room controls the bottom but not the top socket, then typically the outlet is installed upside-down as an indication of this fact. I don't know if this is a matter of building codes, but it does seem to be a very common practice.