Payments to the non-existent employees had been costing the government more than $2m (£1.4m) a month, according to the prime minister’s office.
The authorities say they are continuing to audit the public payroll and expect to find more phantom workers.
President John Magufuli, who was elected in October, has promised to cut wasteful public expenditure in office.
He ordered the audit in March, calling for the money saved to be used towards development.
Nicknamed the bulldozer, Mr Magufuli has announced a range of cost-cutting measures since coming to power, including cancelling official celebrations for independence day.
Tanzania spends more than $260 million a month paying the salaries of its estimated 550,000 public workers, Reuters news agency reports.
“We intend to have workers in government who are honest, accountable and hardworking. This is our priority and it is a non-stop initiative,” Prime Minister Kassim Majaliwa told Tanzanians living in the UK, according to local newspaper The Guardian.
The prime minster was speaking after attending a major anti-corruption summit in the UK capital, London, last week.
Tanzania is ranked 117 out of 167 nations by Transparency International on its perception of corruption index.
Many countries across the continent have been affected by the scam of so-called ghost workers.
In February, the Nigerian government removed 24,000 workers from its payroll after an audit revealed they did not exist.
In September 2014, Kenya began biometrically registering all civil servants after unearthing 12,000 similar cases.