That is just going to collect dirt over time. I would put a paper clip in each side to hold the clamp lip pressed in. So push on one side, insert paper clip, push otherside, insert paper clip. Pull on head. You want to use something as skinny as possible.
The orientation needs to be so the paper clip creates a "ramp" or taper for the retaining lips to slide down.