By Joe Holden

PHILADELPHIA (CBS) — After less than two years on the job, Philadelphia’s managing director is calling it quits. As the No. 2 official in the city, Brian Abernathy has faced sharp criticism this year.

Now, he’s talking one-on-one with Eyewitness News reporter Joe Holden.

Abernathy, the mayor’s right-hand man says he’s out. The managing director tendered his resignation Tuesday, effective Sept. 4.

He says it was a decision reached jointly with Mayor Jim Kenney.

“It started out as a conversation about challenges, these are hard jobs. We ended up with me leaving,” Abernathy said.

Welling up with tears, Abernathy got personal about the killing of a child and an image he says was seared in his mind.

“I’ll never forget looking up at her neighbor, who was a little girl about my daughter’s age, with her head on her hands, her chin on her hands, just looking out the window, watching all the activity. And all I can think about is how we have failed that little girl,” Abernathy said.

Abernathy says he’ll now be able to spend more time with his wife and two daughters.

His two-year tenure has been punctuated by tough times — a refinery explosion, police officers shot, a pandemic and protests. Abernathy says while it comes with the territory, it also hasn’t been easy.

Working, he says, for three months straight, Abernathy has weathered sharp criticism, even blame for reduced police resources during protests on the weekend of May 30 and 31. Police commanders have argued that allowed civil unrest to explode.

CBS3 earlier this month first reported there was never any pressure exerted from City Hall to dilute police staffing that weekend.

“I have overseen the police department now for five-and-a-half years and I think any commander there will tell you I have never intervened in how they do their work because I’m not a law enforcement official,” Abernathy said.

Abernathy has been arguably the most visible member of the Kenney Administration. The mayor granted Abernathy wide latitude to run the day-to-day operations. At times, critics have said he was granted too much power.

“I led the best way I know how and certainly the mayor’s style is to allow me to run the operations and I’ve always appreciated that. But ultimately, by allowing me to do that, those are the mayor’s decisions as well,” Abernathy said.

As for future work, it’s a big question mark.

Abernathy says there have been good days in the office he runs up on the 14th floor, but the bad days have been many.

“This work is hard and we don’t do it because it’s fun — this isn’t a fun job, we don’t get paid all that much money, you get beat up a lot. We do it because it’s a calling,” Abernathy said.