It's super awkward when people ask you "where do you work?" Telling people you work at the IRS is definitely the worst way to introduce yourself at a party.
The outside employment request is ultimately what had me move on. To accept any self employment, a side gig or other work, you must first have approval from IRS. This is fair for some fields, as they don't want you working at HR Block and also IRS. But I missed out on thousands of dollars in freelance work because they took so long to approve me doing things that had nothing to do with taxes. Not sure if this was in place then, but it was a nuisance.
As you can imagine, most people calling the IRS are not happy. And being "the voice of the IRS" as a full time job can often lead to burnout. So most people only stay in this job 5-6 years maximum before heading on to new types of work.