It will go through the hole if; A) You have small hands. B) You have BIG patience.
It's a right pain in the arse to be honest but it can be done... Next time you're bored and you've got a spare hour then give it a go. If you take the clamp off the trim and put that through first on it's own sort of turned at an angle then it'll go. I'd reccoment having an old cloth handy to put between the clamp and diffuser so you don't scuff the plastic as you're fiddling about. You WILL graze your knuckles though.
TBH i just take the diffuser of, only takes 5 mins then its a lot easier to work with then.
And no scratched hands