The power switch does need to be pressed fairly 'deep' (maybe about 4mm) before it clicks over to the 'on' position. Give that a try, maybe with the tip of a pen if need be.
Otherwise...we do stock a momentary version of this button, and it's possible there was a mix up when packaging the kits. If you indeed have a momentary button there, please email email@example.com
(and include a link to this thread) to arrange for a replacement.