You're not nuts. You're conflicted because you recognize both the positives and the negatives in your situation. Sucks, don't it?
I was in a similar situation before massage when I worked for a Fortune 100 company. I enjoyed the work but not the conditions so I gave my written notice. Come to find out management really did value me as an employee but took advantage of me because I let them. They asked me to stay, and I came up with a list of demands that I stuck to like glue. My work conditions drastically improved when I established the boundaries! This experience improved the relationship with my boss since we learned better how to communicate. He gave me honest feedback as well which hurt at first but helped me improve myself in the long run.
One thing I always do before deciding to quit is make a list of pros and cons (which you've done0 and assign each item a number of importance on a scale of 1 to 10. Then I compare the two lists. For example, my last corporate job had a short commute which was worth 10 points to me. But it was boring, and interesting work was worth 5 points. So the short commute outweighed the tedious work. You might see the positives literally "outweigh" the negatives (or vice versa!)
Good luck, and I'll check back if you want to bounce around any more ideas.