Intimation of tax ascertained as being payable under section 74(5)

Reversal of ITC - disputable, from tax officer's perspective it reversible since section 16(2)(c) states that ITC is eligible only if tax is paid to the government. On the other hand, courts have held (though in the context of pre-GST) that buyer cannot be punished for non-compliance of seller. Whether to fight this matter or not depends on amounts involved. 

Payment of interest - can be fought since you have enough ITC balance. Interest under section 50 applies on reversals under the matching concept (GSTR1/2/3), which is currently on hold. But this may not be accepted by tax officers at first level. Suggested to pay under protest and appeal further, if amount is significant. 

Payment of penalty - can definitely be fought, as you rightly mentioned, buyer should not be punished for seller's mistake unless they can prove that you were involved with seller in the mistake. 

Based on past experience, officers are unlikely to any/all of the accept the above - depending on amounts involved, better to prefer an appeal at higher forum where our arguments will be appreciated better.