I stayed for the part of the fandom I’m actually engaged with, yes.
“Fandom” isn’t a monolith. It’s not an all-or-nothing. You can and should choose the people you want to be around, and not deal with the ones you don’t want to deal with.
That’s what you do with people in real life (you hang out with your friends, and not with people you dislike), so why should this be different?
If you “left because of the fandom,” then you’re doing fandom wrong.
I just stayed, and still stay, for the show. None other has developed such an engaging, well written and orchestrated piece of media, and I highly doubt any other show will come close to what this one was able to capture.
Edited

your current filter.Edited
In my case, I came for the show,
but left because of the obnoxious fans
and the poor direction the show went in.