The Saudi hajj ministry said that the delegation had “asked to go back home without signing the agreement on arrangements for the pilgrims” despite two days of extensive talks.
A statement said the ministry had offered “many solutions” to meet a string of demands made by the Iranians who had arrived on Tuesday.
Agreement had been reached in some areas, including to use electronic visas which could be printed out by Iranian pilgrims, as Saudi diplomatic missions remain shut in Iran, it said.
Riyadh cut ties with Tehran in January after demonstrators torched its embassy and a consulate in the Iranian capital following the execution of a prominent Shiite cleric in Saudi Arabia.
Earlier this month Iran had accused its regional rival of seeking to “sabotage” the hajj, saying “arrangements have not been put together” for its pilgrims.
Tehran said Riyadh had insisted that visas for Iranians be issued in a third country and would not allow pilgrims to be flown in aboard Iranian aircraft, which the Islamic republic rejected.
On Friday the Saudi hajj ministry said Riyadh had agreed to allow Iranians to obtain visas through the Swiss embassy in Tehran, which has looked after Saudi interests since ties were severed in January.
Riyadh also agreed to allow some Iranian carriers to fly pilgrims to the kingdom despite a ban imposed on Iranian airlines following of the diplomatic row between the two countries, the ministry said.
Talks on an agreement were the second attempt by both countries to reach a deal on organising this year’s pilgrimage for Iranians after they held an unsuccessful first round last month in Saudi Arabia.
The Saudi ministry said the Iranian Hajj Organisation would be held responsible “in front of God and the people for the inability of its pilgrims to perform hajj this year.”
The annual hajj is one of the five pillars of Islam which devout Muslims must perform at least once during the lifetime.
Another contentious issue has been security, after a stampede at last year’s hajj killed about 2,300 foreign pilgrims including 464 Iranians.
