Im always rejected in recess why?

my clmates do not hate me they see me like everyone else i have 3 friends but when its recess they will ignore me and slowly walk away im friends with the popular girls but not too close if i come too join there group this one girls says wheres your friend so i thinks its a rejection cause she will tell me to go play with them not in a mean way what can i do in my older school i was walking at recess alone for 3 years and i am sick of it