The 6.4-magnitude quake struck in the middle of the night and as day dawned it revealed the destruction left behind in the city of Tainan, which bore the brunt of the damage.
An entire residential complex containing almost 100 homes toppled to the ground, lying on its side with twisted metal girders exposed.
A 10-day old baby girl and a 40-year-old man were pulled dead from the building, National Fire Agency officials said, with reports that around 30 more were still trapped inside.
A third woman died after being hit by a falling water tank at another site with no details immediately available of the other two deaths, the fire agency said.
Residents at the 16-storey felled Wei-kuan Building told of their terror as the quake hit.
“I saw buildings shake up and down and left and right,” said one resident.
“The first and second floor just collapsed and I smelt gas and water was leaking,” he told local channel SET TV.
Another man tied his clothes together to create a rope and lowered himself from his home on the ninth floor to the sixth floor below, Apple Daily reported.
One woman told how she had fought her way out of her home.
“I used a hammer to break the door of my home which was twisted and locked, and managed to climb out,” she told local channel SET TV, weeping as she spoke.
Rescuers have freed more than 200 people from the Wei-kuan building, with over 40 of them hospitalised.
Interior minister Chen Wei-jen said he feared there may be more people in the building than usual as family members would have returned to celebrate the Lunar New Year holidays next week.
“Exactly how many people were there when the quake hit was not immediately clear,” said Chen.
“We are concerned that most members of those families may have returned for the coming new year holiday.”
– ‘Dazed and exhausted’ –
Officials said there were 256 residents registered as living in the building, which contained 96 apartments.
Dazed and exhausted residents stood outside the toppled building, watching rescue workers take survivors from the rubble and carefully hand them down ladders.
Television footage showed two small children wrapped in blankets being plucked from the building.
As Premier Chang San-cheng visited the disaster zone, one elderly woman wept, saying her son, daughter-in-law and grandchildren were still trapped on the 15th floor, Apple Daily reported.
Cranes towered over the disaster zone with diggers trying to remove slabs of concrete.
Officials were unable to give an estimate of how many were still trapped as they scoured the building.
Separately, at least 30 people were earlier freed from another residential seven-storey building.
Across Tainan, 316 people had been injured, with more than 60 hospitalised, officials said.
The shallow quake struck at a depth of 10 kilometres (six miles) around 2000 GMT Friday, according to the US Geological Survey, 39 kilometres northeast of Kaohsiung, the second-largest city on the island and an important port.
The quake was initially reported as having a magnitude of 6.7, but was downgraded to 6.4.
The Pacific Tsunami Warning Center said a destructive Pacific-wide tsunami was not expected.
Taiwan lies near the junction of two tectonic plates and is regularly hit by earthquakes.
A strong 6.3-magnitude quake that hit central Taiwan in June 2013 killed four people and caused widespread landslides.
A 7.6-magnitude quake struck the island in September 1999 and killed around 2,400 people.