The phrases ‘ChatGPT’ and ‘artificial intelligence’ (AI) have become buzzwords in recent months with the whole world marveling over the vast possibilities introduced by the popular AI chatbot. This excitement stems from the ease of use of the open access program which can create new applications and predictive reports, utilize third-party plugins and have the skill to scour the depths of the internet to provide structured, analyzed information in a clear, concise format to the user in a fraction of a second. The hype has also been seen to engulf various institutions around the world. While conversations are still underway regarding the ethics and privacy concerns around the usage of ChatGPT-4, on 29th March, 2023 a court in Phalia (Mandi Bahauddin) in pre-arrest bail proceedings under the Anti Rape (Investigation and Trial) Act, 2021 included an ‘experimental’ use of ChatGPT-4’s responses to a series of questions within its judgment.
The case in which this supposedly innovative use of ChatGPT-4 had been conducted was about an alleged attempted rape of a minor and an accused juvenile’s consequent pre-arrest bail hearing. It goes without saying that such an unprecedented and polarizing use of ChatGPT-4 in the matter of a sexual offense with such a sensitive factual premise is unsavory, to say the least. Further, it is hierarchically unsound for a lower court to employ such an exploration not just for the reasons mentioned below but also for the sake of respecting its traditional jurisdictional bounds. It is not for lower courts to set trends, that too without paying heed to the potential negative impacts of their public ‘experiments’. Ideally, proper permission should be sought from the National Judicial Policy Making Committee (NJPMC) and the Law and Justice Commission of Pakistan (LJCP) prior to using new approaches to judgment drafting rather than treating their direction as an afterthought. Simultaneously, the onus is on the bodies that regulate judicial conduct to circulate updated codes of procedure regarding emerging technologies and the engagement of judicial staff with the same.
While prefacing the use of ChatGPT-4 in the judgment, it is mentioned that the experimentation is intended to explore the possibility of the use of such software to develop ‘crisp’ and ‘smart judicial orders’ while citing the employment of automated judges in China and Dubai as aspirational models. This motive is flawed on two accounts; the first one being dependence on a large language model to write well-drafted judgments and the second one being the comparison to Dubai and China. Simply put, open-access large language models are not the answer to poorly drafted judgments. Staff training and secure editing software can be utilized to this end while also limiting privacy and cybersecurity risks. Furthermore, Pakistan does not have the relevant data protection mechanisms in place to counter violations of any fundamental rights or constitutional freedoms via the use of AI in decision-making. Pakistan’s institutional use of AI needs to be contextually relevant i.e. technology must complement the existing structures rather than transpose AI on a demographic that is neither legislatively nor sociologically prepared for a tech revolution. Additionally, both China and Dubai have used assistive AI within their decision-making, but only for civil disputes, such as online trading, e-commerce purchase liability, copyright cases and rental disputes. There is a reason why criminal cases, that too of the factual magnitude of the instant case, have been limited to the traditional judgment drafting procedures. This has been done to counter the intrinsic deficiency of AI with its inability to be empathetic or conscious of nonverbal cues and the causational changes in circumstances which a human judge may be naturally poised to construe better.
In the judgment, ChatGPT-4 has been prompted with various questions regarding the grant of pre-arrest bail. Its responses notify the user about its lack of access to jurisdictionally relevant case-law and incapability to provide legal advice. These responses go to the larger conversation of whether the Pakistani judiciary even has the utility of such large language models given that not all case-law is available in digital repositories for these programs to access in order to curate relevant responses. Even today, most judgments from as near as a decade ago are on paper. These documents are usually exiled to unventilated storage rooms in judicial buildings, with no soft copies available. It is impossible to create case records for large language models or predictive writing programs until and unless efforts are undertaken to digitize all case records. The responses by the chatbot have been vague and not much different than anticipated answers yielded by a simple Google search. A few days ago, ChatGPT was banned in Italy due to its lack of transparent data processing, failure to employ age verification checks and the discrepancy between the information provided and real verifiable data. Reliance on such a platform, by an institution which places its prestige in being the custodian of public trust and tasked with dispensing justice, is mutually exclusive to its function.
It is constantly mentioned that the experiment is intended to pioneer a way forward for evolution in judicial decision-making and judgment drafting, but even well-intentioned projects fail if they are launched without proper research and risk management. According to the judgment, the accused juvenile’s name had been anonymized since the bail order was going to follow a unique route and the drafters were expecting a larger audience of interested readers. Unfortunately, the metadata of the judgment file reveals otherwise. Such blatant disregard for privacy can create life-threatening situations for those involved in cases of a sexual nature, especially when both the alleged perpetrator and the victim are minors. As mentioned earlier, such a glaring faux pas makes it even more discernible that our courts are not yet prepared to harness the powers of AI and complex new smart programs when they cannot even undertake the simplest of clerical tasks. While the judgment does mention that the final order has not been influenced by the findings of ChatGPT-4 in granting the accused’s petition, which is not reprehensible in itself, the performative nature of the experimentation seems to be inspired by the false promises of technological solutionism.
The hollow optimism around technology shows how little effort has been put into researching the ongoing debates on manipulative AI and the attack on human rights which is a resounding fear amongst legal academics and tech experts alike. Lack of technological literacy can be blamed for this unnuanced approach to new technologies. There needs to be an understanding that jumping on the bandwagon and vying for relevance during trends of the 4th Industrial Revolution is not only dangerous but can also cast doubt on institutional integrity. Recognizing this, it is high time to conduct an internal assessment of Pakistan’s judicial architecture before imported technologies are forcefully used by impressionable public service officials which could result in potentially disastrous outcomes.
 OpenAI, ‘What is ChatGPT?’ <https://help.openai.com/en/articles/6783457-what-is-chatgpt>
 Catherine Thorbecke, ‘Don’t tell anything to a chatbot that you want to keep private’ <https://edition.cnn.com/2023/04/06/tech/chatgpt-ai-privacy-concerns/index.html>
 The State v. AM, 29th March 2023 <https://courtingthelaw.com/wp-content/uploads/ChatGPT-4-Abdul-Moaiz-v-State.-FIR-No.-15-2023.-Offence-376iii-511-P.S-Bhagat…-allowed.-29.03.2023.pdf>
 Law and Justice Commission of Pakistan, <http://ljcp.gov.pk/nljcp/#2>
 TechGig, ‘China launches digital courts with AI judges and mobile court system’ <https://content.techgig.com/technology/china-launches-digital-courts-with-ai-judges-and-mobile-court-system/articleshow/72468390.cms>
 Issac John, ‘Dubai to use AI for litigation without a judge’ <https://www.khaleejtimes.com/technology/dubai-to-use-ai-for-litigation-without-a-judge>
 Ryan Browne, ‘Italy became the first western country to ban ChatGPT. Here’s what other countries are doing’ <https://www.cnbc.com/2023/04/04/italy-has-banned-chatgpt-heres-what-other-countries-are-doing.html#:~:text=Italy%20has%20become%20the%20first,chatbot%20from%20U.S.%20startup%20OpenAI.>
 United Kingdom Judiciary Guidance, ‘Practice Guidance: anonymisation and avoidance of the identification of children and the treatment of explicit descriptions of sexual abuse of children in judgements intended for the public arena’ <https://www.judiciary.uk/wp-content/uploads/2018/12/anonymisation-guidance-1-1.pdf>
 KU Leuven AI summer school, ‘Open Letter: We are not ready for manipulative AI – urgent need for action’, <https://www.law.kuleuven.be/ai-summer-school/open-brief/open-letter-manipulative-ai>
 Future of life Institute, ‘Pause Giant AI Experiments: An Open Letter’ <https://futureoflife.org/open-letter/pause-giant-ai-experiments/>
The views expressed in this article are those of the author and do not necessarily represent the views of CourtingTheLaw.com or any other organization with which she might be associated.