Nope it is fine to have them off and leave them off. As said just pull, they are pretty sturdy but be gentle. They aren't cheap to replace and pretty rare to find second hand.
Best to pop both ends out first I find on the glovebox one as it has quite a curve in it, first time they might be a bit stick but just be firm yet gentle.