The pull-out from Yarmouk leaves al Qaeda-linked Nusra as the main group inside the camp.
The sources said hundreds of fighters of the hardline Islamic State had returned to their stronghold in neighboring Hajar al Aswad, from where they had launched their attack earlier this month.
“Most of them have withdrawn in mostly to-and-fro skirmishes that took place between them and their adversaries,” resident Abu Ahmad Hawari said.
Alongside seeking to capture the camp, they sought to defeat their rival, the Hamas-linked Aknaf al Maqdis, an opposition group that was ideologically opposed to them.
Islamic State’s arrival in Yarmouk had given the jihadist group a significant foothold a few kilometers from Syrian President Bashar al-Assad’s seat of power.
The group was still fighting some of the remaining Aknaf al Maqdis fighters in the northern entrance of the camp at the junction of the main Palestine and Yarmouk streets, two residents said.
The withdrawal leaves al-Qaeda offshoot al Nusra as the biggest force in the camp, many of whose residents have fled since Islamic State launched its offensive, they added.
The Palestine Liberation Organisation envoy to Damascus said that Nusra was now the main group in the camp.
“They and Nusra are one. They are changing positions,” Anwar Abdul Hadi told Reuters.
Nusra was accused by its rivals of facilitating the entry of Islamic State militants into the camp. Although they are rivals elsewhere in Syria, both share a loathing for Aknaf al Maqdis.
But Nusra, unlike Islamic State, was not ready to push its rivalry to a military confrontation and did not engage in the latest round of fighting in Yarmouk, according to residents.
The camp was home to some 160,000 Palestinians before the Syrian conflict began in 2011, refugees from the 1948 war of Israel’s founding, and their descendents.