I wouldn't do it! The letters and badge(s) are on there for about 8 years now and there will be a colour difference between the parts that were exposed to the environment and the ones covered. No matter if the car was garaged or not.
One of the previous owners of mine has removed the RS badge on the boot and under certain conditions (depending on lighting and/or if the car is dirty) you can see where the badge was located before. It's barely noticable but if you know and look closer you'll see it.
If you really are going to remove them there's a product called Eulex M (Eum) from a German company named Koch-Chemie. It's a product to remove glue, tree gum and rubber residue also from matt paints which I can recommend.
http://www.koch-chemie.de/en/Detailing/Washing_exteriors_and_pretreatment/#Eulex M