It also meant United had failed to win any of their last 10 league and cup meetings with Chelsea over the past three years.
Jesse Lingaard scored a fine goal for the visitors after an hour’s play but in the first minute of added time Costa coolly beat Spain team mate David De Gea to snatch a draw for a side unbeaten since Hiddink returned as interim manager in December.
Chelsea, who lost defender Kurt Zouma with a serious looking knee injury, were left in 13th place in their unexpectedly feeble title defence.
United stayed fifth, six points behind neighbours Manchester City and the Champions League places.
“They started well but as the game grew on we grew into it,” Chelsea captain John Terry told Sky Sports. “It was a great response, we’ve shown that character since the manager came in and that’s good to see.”
Costa’s equaliser was gut-wrenching, however, for United goalscorer Lingard.
“To concede late on it is like a defeat but we have to move on to the next game,” he said. “We are playing a lot better now. As a collective group we are doing well.”
The equalising goal apart, Van Gaal had reason to be satisfied with another improved performance to add to recent wins by 3-1 over Derby County in the FA Cup and 3-0 at home to Stoke City.
The media speculation about Van Gaal’s future will not go away, with former Chelsea manager Jose Mourinho regularly linked to Old Trafford, but United were the brighter side early on.
Chelsea took half an hour to get going before making a series of chances with Costa, playing against United for the first time, driving just beyond the far post before Oscar shot over the bar.
There were also strong appeals for a penalty as John Terry’s effort hit Danny Blind’s arm.
Before that the visitors forced Thibaut Courtois into a fine save as Anthony Martial cut inside to shoot and United were on top again early in the second half, when Courtois had to save from Wayne Rooney and Lingaard.
The young Englishman then beat him by taking down a cross from Cameron Borthwick-Jackson with one touch and shooting high into the net with his second.
It was his fourth goal of the season and second in successive games.
Only then was United’s David de Gea forced into his first real saves, from Branislav Ivanovic and Cesc Fabregas.
Costa put a late header over the bar but made amends by taking Fabregas’s pass as Blind slipped and sliding in the equaliser.
He was even close to a winning goal in the sixth and last minute of stoppage time but was denied by the goalkeeper.
