The militants said they have killed 70 AU soldiers at the Janale base, 90km (55 miles) south-east of the capital.
AU forces say they are back in control after taking a “tactical withdrawal”.
Al-Shabab, part of al-Qaeda, is battling the AU-backed government for control of Somalia.
Residents said the attack started with a suicide car bombing at the base’s gate, followed by sustained gunfire which lasted more than an hour.
They told the BBC Somali service that AU forces were seen leaving the base, which is run by the Ugandan troops.
They said they counted the bodies of 20 AU soldiers and later they saw more troops arriving at the base after the militants had left with weapons.
The AU mission in Somalia (Amisom) says the situation is complex and it does not yet have casualty figures.
The BBC’s Mohamed Mualimu in the capital, Mogadishu, says the situation in the area is very tense and few residents remain there.
The militants had earlier damaged a nearby bridge with a massive bomb to prevent troops from escaping, Col Ahmed Hassan from the Somali army told the Associated Press.
This is being seen as an attempt by the militants to isolate AU bases in the area, our reporter says.
In June, the group overran another African Union base in Lego, a small town in the Lower Shabelle region in southern Somalia, killing dozens of soldiers and seizing military equipments.
Despite losing most of its key strongholds in south and central Somalia, al-Shabab continues to carry out attacks on the government and African Union troops across the country.
The militants also stage frequent suicide attacks inside the capital, Mogadishu.