Mr Modi admitted defeat after results showed Arvind Kejriwal’s Aam Aadmi Party (AAP) heading for a landslide in the 70-member assembly.
Mr Modi congratulated the AAP leader and promised him complete support.
Correspondents say his win marks a remarkable comeback for the campaigner who is a former tax inspector.
Mr Kejriwal’s party was routed by the BJP in May’s general elections, months after the AAP made a spectacular debut in the 2013 Delhi elections.
His win in Delhi is the first major setback for Mr Modi who has enjoyed huge popularity since taking office last year, winning a string of local elections and wooing international investors and world leaders.
At 13:00 local time (07:30 GMT), the AAP had won 16 seats and was leading in another 50, while Modi’s Bharatiya Janata Party (BJP) had won two seats and was ahead in one more. The Congress party, which has ruled Delhi for 15 years, has failed to win even a single seat.
Mr Kejriwal told cheering party supporters on Tuesday that the “people of Delhi have achieved something spectacular”.
“With the help of people, we will make Delhi a city which both poor and rich will feel proud of,” he said.
As supporters showered him with rose petals, Mr Kejriwal said the huge mandate was “very scary and we should live up to people’s expectations”.
The BJP’s campaign was essentially anti-AAP and the party leaders often criticised Mr Kejriwal at their rallies and road shows.
Mr Kejriwal, meanwhile, conducted an energetic campaign which proved popular with working class and underprivileged voters who make up 60% of Delhi’s population.
The BJP fielded former policewoman Kiran Bedi as its candidate for chief minister.
“We never expected that the results would be so depressing. We never expected that we would suffer such a setback. It is certainly not our day today,” BJP leader Praveen Shankar Kapoor told BBC Hindi.
The Congress party, which ran the Indian capital for 15 years until 2013, has suffered a complete rout and is not even leading in one seat.
Delhi has been without a government since Mr Kejriwal resigned last February as chief minister, angered that his anti-corruption bill was blocked.
Since then the state has been governed directly by the federal authorities.
Ms Bedi and Mr Kejriwal worked together during the anti-corruption campaign led by social activist Anna Hazare, but the two have since developed an intense rivalry.
During weeks of hectic campaigning in Delhi, both candidates promised to bring in good governance, end corruption and make Delhi safe for women.
In the previous Delhi election held in December 2013, the BJP won the most seats but fell short of a majority, leaving the AAP – which came second – to form a coalition with the Congress party.
But Mr Kejriwal resigned on 14 February, after 49 days in office, when opposition politicians blocked a bill that would have created an independent body with the power to investigate politicians and civil servants suspected of corruption.