The big downside is the workload, it wasn't uncommon for me to finish pretty late. Sometimes plans would be cancelled or I'd meet friends when they had already polished off a bottle of wine or two in the evening. But this is true of so many professions.
You need to have a pretty thick skin and a certain level of tenacity, people in industry often don't want to talk to you about the conference you are producing / selling. If you are going to let this upset you then this isn't the industry for you.
The nature of conference production is that you'll get an overview of a lot of topics but you won't learn about them in a huge amount of depth. Eventually I wanted to change this so that's why I left. This may frustrate some people.
Finally, production can be exhausting, not just the hours but the relentless efforts you need to go to to get speakers etc. I would say that it isn't something that many people could spend their entire career doing. That said, IQPC offer plenty of opportunities for people to progress into management so you don't need to be doing the actual production work forever in any event.