Sit both groups of friends down and talk to them about the problem that you're having. If you find some of your friends annoying, talk about it! There's no reason to be uncomfortable and not tell someone that they're bothering you. Make it clear to your friends what you feel, and get everything out in the open. If they're not willing to work with you, they're not really your friends in the first place. Good luck!