I think there's a lot more to mary-sueing than just overpowered or even out-of-place characters. Even in my early RP days when I was too dumb to understand why godmodding was bad, I never made a mary-sue RP character, as I see it. I just couldn't play them - even behind the mask of a character I was still too shy to even attempt making a character with the kind of personality I idealized, so even when I made overpowered characters I was painfully aware they shared many of my own flaws. By the time I was confident enough, I was also wise enough to know I could have more fun with a balanced character.
Or so I thought. When I started writing an ongoing novel for practice and as a kind of extended daydream, the two main characters (though I didn't realise it at the time) were ostentatiously sue. I'd originally intended them to be powerful and compelling, the gifted minority in a world of fantastic people, wise in their own way, but still young, ignorant, and coping with emotional deficiencies. But somehow, as I kept writing, they continued to overcome everything I threw at them with luck, smug panache and inappropriate adulation by the other characters that eventually ruined the whole purpose of the story and undermined my efforts to actually display the traits I was trying to establish. 'Next time,' I told myself, 'they will screw up. Character A will make a gamble that fails, or someone will get sick of his attitude, or B will suffer the consequences of his careless antics. But this time, it would be so much better if...'
Eventually I caught on to this, partly thanks to a bit of helpful external criticism, and went back over what I'd written with more of a critical eye.
Near the start, the protagonists had run into a racist drunkard, and, naturally, kicked his ass in short order. After all, he was drunk, and one of the pair was a skilled fighter. But I went over it again, and this time, made the drunk a war veteran. Why was he racist? He'd served in a particularly nasty border dispute and witnessed a lot of brutality.
Doing a second take, then, the fight starts as usual, but the protagonists quickly find themselves out of their depth. A's temper almost costs him his life, and B has to drop his carefree manner to save him. As an added bonus I get an extended and far more brutal combat sequence, and a subsequent 'recovery' scene that really helped the character development between the two. The next morning, the vet's sobered up and can't apologise enough. Suddenly, charging in like an asshole doesn't seem like such cool thing to do.
All in all, this adjustment allowed me to paint a darker, more neurotic, even pathetic side to A's attitude, hopefully making him a more sympathetic character, while still allowing me to keep the admirable traits that made him a suitable protagonist. It also added a bit of depth to B.
I guess the point of this self-indulgent rambling was that all you have to do to make a sue is to overlook opportunities to cut your characters down a notch, and all you have to do to fix it is bracket your prejudices and look a bit harder. The callous gaze of the Other is your friend.
GhaKha's point is pretty interesting too though - maybe it's just me, but it seems like Hollywood (and the popular fiction market in general) have been taking even more indulgent liberties recently with what I in my elitism see as 'bad writing practices', Dan Brown's work and Twilight being the most notable examples though also things like (please don't flame me), Harry Potter, though admittedly that has a lot of value in it too. I mean sure, explosions, hot women and shallow, larger-than-life characters are all par for the course, but Mary Sues are just one of a slew of worrying trends towards masturbatory self-indulgence both on the part of the author and their fanbase. I can't really condemn it without being an asshole, because a lot of people like that sort of thing, but I can't help but feel it betrays a lack of entirely appropriate shame.