The huge red crane crashed into a part of the Grand Mosque as it was filled with worshippers.
Officials say strong winds and heavy rains caused the crane to fall.
Correspondents say there have previously been concerns about safety records on Saudi construction sites.
The Grand Mosque, known as the Masjid al-Haram, is the largest mosque in the world and surrounds Islam’s holiest place, the Kaaba.
At least 230 people were injured in the incident. It is unclear how many people were hurt by the collapse or the stampede that followed it.
The incident happened at 17:23 local time (14:23 GMT) on Friday.
Videos posted online showed the moment the structure collapsed, with a loud crash followed by panic and shouting. Bodies and blood could be seen on the floor of the mosque.
The head of Saudi Arabia’s civil defence agency, Lt Sulayman Bin-Abdullah al-Amr, said an investigation was being carried out to assess the damage, and the “extent of the safety of these sites”.
Essam al-Ghalib, a Jeddah-based journalist, said the crane fell through the outer ring of the Grand Mosque and punctured through the roof sending down tonnes of cement and debris onto people either praying or walking through.
Mecca is currently preparing for the annual Muslim Hajj pilgrimage. Up to two million people are expected to arrive in the Saudi city from all over the world later this month.
Irfan Al-Alawi, from the Islamic Heritage Research Foundation, told the BBC that the Grand Mosque is currently surrounded by 15 large cranes amid major redevelopment work.
“The entire area is like a salvage yard,” Mr Al-Alawi said.
“Saudi Arabia has to re-think its health and safety strategy,” he said, “as there were 800,000 people in the mosque area at the time of the accident.”
The original parts of the Grand Mosque date back 1,400 years. Consisting of a large square surrounded by covered prayer areas, the building has since been extensively modernised, notably from the mid-20th Century.
Saudi authorities began a major expansion of the site last year to increase the area of the mosque by 400,000 square metres (4.3 million square feet), to allow it to accommodate up to 2.2 million people at once.