Both sides had sights of goal in the opening two minutes as Joel Campbell missed his kick from Theo Walcott’s cross before Oscar – recalled in place of Pedro – had a shot tipped behind at the near post by Petr Cech.
The former Chelsea goalkeeper was called into action again in the 15th minute as Willian fired straight at him after Arsenal struggled to clear.
And Willian was heavily involved in the sending-off three minutes later as his through-ball released Costa, who looked to break away only to be brought down by Mertesacker, leaving referee Mark Clattenburg little option but to brandish the red card.
That dismissal prompted Arsene Wenger to send on Gabriel in place of Olivier Giroud, but Arsenal were almost immediately behind as Ivanovic’s superb cross from the right was turned in by Costa from close range at the near post in the 23rd minute.
Costa went close to doubling his tally shortly before half-time only for Cech to save sharply, and from the resulting corner Ivanovic’s header was cleared off the line by Nacho Monreal.
For all Chelsea’s superiority, Arsenal had a chance to level with the final act of the first half but Mathieu Flamini volleyed Aaron Ramsey’s clever pass over the crossbar.
In a quieter second half, Chelsea felt they should have had a penalty shortly before the hour when Koscielny body-checked former Arsenal skipper Cesc Fabregas, but Clattenburg was unmoved.
A frantic scramble in the Chelsea box raised Arsenal hopes of an equaliser, but the hosts could not force the ball in as Hiddink’s side stood firm to claim back-to-back away wins and move above West Brom into 13th place.