The Delhi High Court has cautioned both the GST Department and the Income Tax Department to exercise "utmost caution" when relying on Artificial Intelligence (AI) tools such as ChatGPT for legal citations in show cause notices (SCNs) and assessment proceedings.
The warning came from a division bench comprising Justice Prathibha M. Singh and Justice Shail Singh, while hearing a petition seeking to quash a GST show cause notice. Although the court declined to interfere with the SCN and directed the petitioner to respond through the proper legal process, it issued a strong observation on the risks associated with AI-generated legal references.
Court Flags Fake Case Laws, AI Hallucinations
During the hearing, the bench noted discrepancies and inaccuracies in the judgments cited by the GST authorities-raising concerns that some citations may have been generated by AI tools without proper verification.
The court referenced its earlier judgment in Christian Louboutin SAS & Anr. vs Shoe Boutique, where it had clearly stated that AI chatbots, including ChatGPT, cannot be relied upon to determine legal or factual issues due to the possibility of:
- Incorrect interpretations
- Fabricated case laws
- Imaginary or inconsistent data
- Responses influenced by training data or user queries
The bench reiterated that: "Accuracy and reliability of AI-generated data is still in the grey area. At the present stage of technological development, AI cannot substitute human intelligence or the humane element in the adjudicatory process."
It emphasised that AI may be used only for preliminary understanding or research, not for citing authoritative legal precedent.
Departments Must Verify Judgments Before Issuing SCNs
Highlighting the dangers of "AI hallucinations," the court directed that all judgments used in SCNs and assessment orders must be independently verified before being incorporated into official documents.
The ruling stated: "The GST Department as well as the IT Department must take full responsibility in case the same is cited or generated by using AI software."
Expert View: Government Must Validate AI Outputs
Commenting on the judgment, a tax expert noted that the High Court's warning underscores a critical compliance gap:
"The ruling reinforces that AI can only assist in research or summarization. It cannot replace human verification. Departments must cross-check all AI-generated citations to prevent fake judgments and ensure procedural fairness."
The expert added that the judgment also clarifies that legal presumptions applicable under the Income-tax Act cannot automatically be applied to GST proceedings, especially when based on unverified AI-generated data.
CAclubindia
