view
Abstract: We developed and tested the performance of a retrieval augmented generation system for answering legal queries related to corporate insolvency in England and Wales. The Insolvency Bot relies on open-source legal information and HMRC forms to provide sound responses to a user’s query focusing on insolvency matters regulated by English law. We evaluated our bot head-to-head on an unseen test set against the unmodified versions of large language models (LLMs) gpt-3.5-turbo, gpt-4, or gpt-4o with a mark scheme similar to those used in examinations in law schools. The Insolvency Bot outperformed each unmodified LLM (p = 0.05%). An additional user experience survey suggested the need for creating two versions of the bot, one for lay people who expect practical and actionable advice and another for professionals with the relevant legal authorities. Our legal chatbot demonstrates the benefits of combining a generative Artificial Intelligence system with a trusted knowledge base and shows future promise to cover cross-jurisdictional and insolvency-related queries and could be further improved in its technical architecture.
Keywords chatbot; insolvency law (England); large language models (LLM); legal technology; natural language processing (NLP); prompt engineering; retrieval augmented generation (RAG); vector index
JICL welcomes full length articles (generally not exceeding 13,000 words inclusive of footnotes), shorter contributions in the form of notes and comments (generally not exceeding 8,000 words inclusive of footnotes) and book review articles of not more than 6,000 words.
We accept contributions for consideration on an exclusive submission basis. When submitting an article please certify that it is an unpublished article (that is, it has not been previously published in substantially similar form or with substantially similar content) and that it is not under consideration by any other journal.
To facilitate anonymous review, please give the names of authors and their short biographical information and acknowledgments in a separate page.
Authors retain copyright in the words used, but upon submission of material for publication, grant Sweet & Maxwell a licence to publish the submission in print and/or digital formats. Sweet & Maxwell retains copyright in the design, format and layout of all material published in JICL.
Once submissions are published, authors are entitled to one copy of the issue, 10 offprint copies and a PDF version of the submission.
Authors who send articles published in JICL to other publishers or media must include a reference to the publication of the article by JICL and Sweet & Maxwell.
Contributions and book reviews should be submitted in Microsoft Word format by way of email attachment to Professor Anton Cooray at Anton.cooray.1@city.ac.uk.
Authors should follow the OSCOLA citation system (http://www.law.ox.ac.uk/publications/oscola.php), except that we prefer authors to use indenting sparingly.
JICL uses the following heading levels: Main headings are in bold and preceded by a Roman numeral; second-level headings are in bold and italics and preceded by an uppercase alphabet; third-level headings are preceded by an Arabic numeral; and fourth-level headings are in italics and preceded by a lowercase alphabet.