The downsides were poor management in the department I worked in as well as too much "politics". There was too much favoritism, and the most dedicated and knowledgeable employees were over worked with no recognition or reward. Most employees are underpaid.
What I enjoyed the most was referrals, scheduling, ordering Xrays, being on phones, and working in the specialty department. And most of all my patients :)