More than 180 people were injured in the accident, the country’s civil defence authority said on Twitter.
Images from the scene showed part of a huge red crane crashed through the mosque’s roof. The cause of the collapse is not known.
Mecca is currently preparing for the annual Muslim Hajj pilgrimage.
Hundreds of thousands of people are expected to arrive in the Saudi city from all over the world later this month.
Reports suggested the crane fell following high winds and rain. The Arabian peninsula has been has been hit by strong sand storms over the past week.
A video posted on YouTube, which could not be independently verified, appeared to record the moment the crane fell, with a loud crash heard in the background followed by panic and shouting.
Images circulating on Twitter appeared to show numerous bodies and blood on the floor of the mosque.
Saudi authorities began a major expansion of the site last year to increase the area of the mosque by 400,000 square metres (4.3 million square feet), to allow it to accommodate up to 2.2 million people at once.
The crane that collapsed on Friday was one of a number dotted around the site as part of the construction project.
The mosque is Islam’s most sacred site and the destination for millions of Muslims undertaking the Hajj pilgrimage every year.
At the heart of the mosque is the Kaaba, a black cube-shaped building which Muslims all over the world face when they pray.
Islam requires that every Muslim capable of doing so performs a pilgrimage to the site at least once in their lifetime. Once at the mosque, pilgrims perform “tawaf” – walking seven times around the Kaaba in a counter-clockwise direction.