Excise duty exemption limit against turnover

We are a manufacturing company dealing in variety of Minerals which are all non-excisable.

We are doing a small time trading activity in excisable material.

Our Total Annual Turn Over is Rs.20 Crores and we are registered under SSI. But the excisable goods turnover is only Rs.50 Lakhs. 

We are under impression that, if Excisable Goods sales cross over Rs.1.5 Crores, then only we come under excise. but we have not crossed till now or even in any of previous financial year.

Now Excise Dept is asking us to pay penalty of Rs.8 Lakhs for year F.Y.2013-14, since we purchased Excisable goods with paying excise duty, but sold without charging excise duty to our clients.

We are getting dual opinions from lots of experts. We do not know exactly what to do.

Can somebody please help us in this matter. Whether we can claim exemption or we have to pay excise penalty.

Thank you.