Yes, the phrase DOES mean to annoy someone, but it's because the person is getting on your Hampton Wick... substitute a body part. That's my interpretation of what's written in Brewer's!
Melanie Reid noted that 'The New Yorker' once decreed that one exclamation mark was enough for one person for one lifetime. They softened the 'rules' later and said that exclamation marks could be used on special occasions; one apiece for the birth of your children; a choice few to chase off a carjacker. Melanie goes on to say ' But with email and text, the damn things are ubiquitous.' It was interesting - or perhaps, it was interesting!