Obviously, the criteria is different for everyone involved, and sometimes it shifts. I used to dislike AUs in general, and B/A, while not a pairing I dislike, never had me hunt for fanfic about them, but
The quality of the writer is certainly a factor. Sylvia Volk's Methos couldn't be more different from Kat Allison's, but they both convinced me in their respective stories. My personal interpretation of Methos tends to go a bit more in Kat Allison's direction, but not completely; at any rate, I saw both Sylvia's and Kat's renditions as valid interpretations of Methos as presented on the show Highlander. However, take Sylvia's zen-like Methos idea, give it to a far worse writer, and you end up with Saint!Methos, that bane of HL fanfic. Take Kat's ruthless and cynical Methos, do the same thing, and you end up with some of those badfics where the authors appear to think lifestyle a la Kronos is fun.
Sometimes, though, even a wonderful writer just doesn't do the trick of convincing me. Not that this is the writer's fault, I hasten to add; it's probably all due to my personal squicks or indifferences. I stopped reading
What I do think is ooc independently from personal kinks or squicks: sloppy use of characters without any regard of their aims and purpose in the original source. For example: when hunting for stories about Bester about a decade ago, when B5 was originally broadcast, I found one in which he abducted Delenn and mindraped her. I also found one in which he did this with Londo. (And here I was feeling thrilled at seeing a story listing both Bester and Londo as principal characters.) Huh? Now the show gives you leaveway to see Bester as the scum of the Earth as Garibaldi does, or as a self-sacrificial devotee to the telepathic cause as the interns who hero-worship him do, or as both, but what remains true in both interpretations are the man's priorities (telepaths ought to rule) and the fact his actions always have a purpose. He doesn't gratiously torture people left, right and center, let alone powerful alien ambassadors.
If you need a B5 villain to capture and torture Delenn, look up the Vorlons; I'm sure Mr. Sebastian would be available. (Not that he didn't have a purpose as well.) Or some OC who lost their entire family in the Earth/Minbari war due to her orders. If you want someone to torture Londo, then, to paraphrase what Londo says to Na'Toth in A Tragedy of Telepaths, take someone from the very long line of enemies he's made. But in neither case take Bester just because he's one of the more prominent villains of the show.
In other news,