He told supporters that it was shameful that some top officials were earning $18,000 (£13,000) a month, while others were paid as little as $140.
Officials not ready to accept the new $7,000 monthly wage limit should “start looking for alternative jobs”, he said.
Mr Magufuli has promised to cut wasteful public expenditure in office.
He has announced a range of cost-cutting measures since coming to power last October, including cancelling official celebrations for independence day.
The changes would come into effect in time for the start of the next financial year, which starts on 1 July, the president said.
He likened the gulf in wages to a few “angels residing in heaven”, while the majority “languished as if they were in hell”, Tanzania’s The Citizen newspaper quotes him as saying.