I work at a restaurant and I can tell you I have had this happen to me many times! Usually when someone can't work (due to whatever reason) I get the call at 6am asking if I can come in at ____ time (usually noon, but sometimes 8am) to fill in! Most times I am available, so I do tell them yes, even though, I am not on schedule, I come in to fill the shift, its a way to show reliablity and to make extra money.
To answer your question- Your manager has the right to ask you to come in to fill a shift, however, you also have the right to turn him down without consequences, because you aren't scheduled for that paticular shift, so really both of you have equal rights on this.
My question here is why are you turning them down here? Are you not willing to work those hours because you don't feel so? Do you feel unsafe being in a sketchy part of town in the middle of the night? Is travel an issue?. I srongly recommend you have a discussion with your manager about your concerns, see if you guys can come up with a mutual beneficial compromise.