It comes a week after US authorities said they would seek $14 billion from Deutsche Bank over similar allegations.
“The Royal Bank of Scotland Group has reached a final settlement with the National Credit Union Administration Board to resolve two outstanding civil lawsuits for $1.1 billion (£846 million),” the British bank said in a statement.
RBS said the “settlement amount is substantially covered by existing provisions”, although the bank noted it may need to set aside additional funds awaiting the outcome of other similar claims faced by the bank.
US authorities have accused major banks of misleading investors about the values and quality of complex mortgage-backed securities sold before the 2008 global financial crisis.
Much of the underlying lending was worthless or fraudulent, delivering billions of dollars in losses to holders of the mortgage bonds when the housing market collapsed, bringing down numerous banks.
The substantial fine for RBS is meanwhile the latest in a long line of financial blows for the lender since it was rescued with British state funds in the wake of the financial crisis.