In the hastily-arranged party leadership ballot, Mr Abbott, who had been plagued by poor opinion polls, received 44 votes to Mr Turnbull’s 54.
Liberal MPs also voted for Julie Bishop to remain deputy leader of the party.
Mr Turnbull is expected to be sworn in after Mr Abbott writes to the Governor General and resigns.
Earlier on Monday, at a press conference in Canberra, Mr Turnbull said if Mr Abbott remained as leader, the coalition government would lose the next election.
He said he had not taken the decision lightly, but that it was “clear enough that the government is not successful in providing the economic leadership that we need” and that Australia needed a new style of leadership.
Mr Turnbull will be Australia’s fourth prime minister since 2013.
Labor Prime Minister Julia Gillard was ousted by rival Kevin Rudd in a leadership vote in June 2013 – months before a general election that Tony Abbott’s Liberal Party won.
Ms Gillard herself ousted Mr Rudd as prime minister in 2010.