One of the major downsides at least on upper levels is the problem of further growth. Deloitte does not resemble the classical pyramid but has lots of Managers, Senior Managers and Partners limiting the growth.
The compensation is mediocre at best (I will not complain compared to many other jobs but in comparison to other consultancies it fares worse).