The music mogul who is judging this year's singing contest alongside Demi Lovato, Kelly Rowland - who was a judge on the UK 'X Factor' in 2011 - and Paulina Rubio, fired Britney as she felt "uncomfortable" in the role.
Simon says the 'Ooh La La' singer was unable to give honest, "negative" feedback to the pop hopefuls so he had to replace her.
Speaking on 'The Ellen DeGeneres Show', he said: "I kind of booked someone who couldn't talk, which is a bit of a problem when you want someone to judge!
"She wasn't comfortable being negative, she didn't like it when the contestants spoke back to her, so she was just uncomfortable. I liked her. So we decided to change it this year and I thought I'll work with three girls because I like working with girls."
Britney was a mentor to the junior contestants on last year's show - on which she was joined by Simon, L.A. Reid and Demi.
Simon - who is expecting his first child with girlfriend Lauren Silverman - also revealed he made Britney, 31, a judge because she is such a huge name in the pop world.
When quizzed by Ellen - who was a judge along with Simon on 'American Idol' in 2010 - on why he hired Britney, he replied: "I think if I'm being honest with you, Ellen, there was a time when I think everyone was trying to up themselves in terms of who could find the biggest known star to be on a panel."