Australia, who were chasing a record 289 for victory at R Premadasa Stadium, were bowled out for 206 in 47.2 overs with Matthew Wade top-scoring with 76 runs.
Left-arm spinner Amila Aponso claimed four wickets while medium-pacer Thisara Perera took three wickets to choke the Australian run chase.
But it was Mathews’ two wickets and a half-century while batting that made Sri Lanka register their biggest win against Australia in terms of runs. Their previous best was a 79-run triumph in Sydney in 2003.
Australia’s James Faulkner recorded his first ODI hat-trick but Sri Lanka, who chose to bat first, still managed to post 288 in 48.5 overs.
In reply Australia, who won the first of the five ODIs by three wickets, were dealt early blows after losing their openers to Perera.
David Warner was caught behind for 1 while Aaron Finch, who scored a half-century in the first ODI, was bowled for four with Australia left struggling at 16 for two.
Skipper Steve Smith tried to fight back with his 33-ball 30, his innings laced with five boundaries, but his wicket off Aponso spelt more trouble for the visitors.
George Bailey and Wade put together a 61-run fourth wicket partnership to keep Australia in the hunt but the required run-rate just kept mounting.
Bailey, who scored 27 off 46 balls, finally succumbed to the pressure of scoring quick runs after being bowled by Aponso, who registered impressive figures of 4-18 in just his second ODI.
Leg-spinner Seekkuge Prasanna soon joined forces with Aponso to get Moises Henriques, who scored four runs, stumped as Sri Lanka gained control of the match.
Matthew Wade then staged a gritty fightback with support from Travis Head as the duo managed 64 runs between them to keep Australia’s hopes alive.
Wade’s 88-ball knock was studded with three boundaries as the wicketkeeper-batsman finally became Perera’s third victim and the Australian resistence wilted.
Head, who scored 31, soon got out to Mathews as the visitors lost their way into the chase, losing their final five wickets for 24 runs.
The action now shifts to Dambulla for the third ODI on August 28.