Simon Cowell offered Olly Murs £3million to be an X Factor judge but dropped him when Robbie Williams agreed to join the show.
Olly, 34, had accepted a return if he could also remain on rival contest The Voice.
But Robbie and his wife Ayda Field then agreed to join Simon as judges, meaning the fourth needed to be female.
He has just days to fill the last spot on his panel.
An X Factor source said: “We’re all huge fans of Olly’s, but Robbie is one of the biggest stars in the world.
“When the opportunity came for him to join, we couldn’t pass it up.”
Simon axed ex-contestant Olly as a host on ITV’s show in 2015, but was so keen for him to return as a judge that he offered him the huge fee.
An insider said: “Olly said he’d do X Factor on the condition he could stay on The Voice, which he loves.”
Though Voice bosses made it clear he would be dropped if he did, fearing viewers would tire of him.
But then Robbie joined Simon. Voice chiefs were said to be “high-fiving” at Olly staying.
X Factor’s London auditions begin on July 18.
Related: Olly Murs sings Tina Turner ‘What’s Love Got To Do With It’ in concert
MSN Loneliness Campaign: Could you go a week without seeing anyone? We’re helping three charities raise funds for the 9 million people affected by Britain’s ‘silent epidemic’. Find out more and please donate now.