If you're not a managers "favorite', best of luck keeping your job. Middle management has a Boys Club mentality. So if you're not one of them, you're just a parasite in the way. Be prepared to be bullyed and targeted. Directors turn the cheek when management does this, and HR doesn't stop it from happening.
In addition to that, there is very little opportunity for growth. Once you get your position, you're pretty much stuck there unless you quit or at least threaten to quit.