Projects per year
Abstract
In this article, we explore the potential of ChatGPT to pass the Brazilian Bar Association exam, which consists of two parts. The first part includes 80 multiple-choice, single-answer, questions, with a maximum score of 80 points. The second part comprises a procedural document, worth 5 points, and 4 open-ended questions, worth 1.25 points each, and a human expert evaluates ChatGPT’s responses, in different domains of law. All three versions of ChatGPT performed well in the multiple-choice, single-answer, questions. GPT 4 ranks the highest, achieving a score of 70% of correct answers, followed by GPT 3.5 Default with 55%, then GPT 3.5 Legacy with 53%. However, when it comes to the second part the results are not as good. In the criminal exam, GPT 4 performs the worst, while GPT 3.5 Default performs the best, with GPT 3.5 Legacy coming in a close second. Regarding the business exam, GPT 3.5 Legacy had the worst performance, while GPT 4 achieved the highest score: 5.02. Overall, all ChatGPT versions performed well in the multiple-choice questions, but their responses to open-ended questions were underwhelming.
| Original language | English |
|---|---|
| Title of host publication | Progress in Artificial Intelligence - 22nd EPIA Conference on Artificial Intelligence, EPIA 2023, Proceedings |
| Editors | Nuno Moniz, Nuno Moniz, Zita Vale, José Cascalho, Catarina Silva, Raquel Sebastião |
| Publisher | Springer Science and Business Media Deutschland GmbH |
| Pages | 131-141 |
| Number of pages | 11 |
| ISBN (Print) | 9783031490101 |
| DOIs | |
| Publication status | Published - 2023 |
| Event | 22nd EPIA Conference on Artificial Intelligence, EPIA 2023 - Faial Island, Portugal Duration: 5 Sept 2023 → 8 Sept 2023 |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 14116 LNAI |
| ISSN (Print) | 0302-9743 |
| ISSN (Electronic) | 1611-3349 |
Conference
| Conference | 22nd EPIA Conference on Artificial Intelligence, EPIA 2023 |
|---|---|
| Country/Territory | Portugal |
| City | Faial Island |
| Period | 5/09/23 → 8/09/23 |
Keywords
- Artificial intelligence
- Brazilian bar exam
- ChatGPT
- Law
Fingerprint
Dive into the research topics of 'Does ChatGPT pass the Brazilian Bar exam?'. Together they form a unique fingerprint.Projects
- 1 Active
-
CEID-CRCFL: Católica Research Centre for the Future of Law
Pais, S. O. (Associate Professor), Costa, A. C. R. (Invited Assistant Professor), Afonso, A. (Assistant Professor), Ribeiro, A. T. (PI), Martins, A. A. (Invited Assistant Professor), Guedes, A. (Associate Professor), Sousa, A. F. D. (Assistant Professor), Triunfante, A. (Assistant Professor), Botelho, C. S. (Assistant Professor), Fraipont, C. D. (Student), Tavares, E. (Student), Ferreira, A. S. (Student), Antunes, A. F. M. (PI), Fonseca, A. (PI), Abrantes, A. M. (Student), Amaro, A. E. L. (Scholarship holder), Cortês, A. U. (Associate Professor), Rocha, A. (Assistant Professor), Baldé, A. (Student), Sardinha, C. (Assistant Professor), Lucena, C. (Student), Sequeira, E. V. D. (Assistant Professor), Ferreira-Pinto, F. A. (Assistant Professor), Calvão, F. (Associate Professor), Alves, F. C. (Invited Assistant Teacher), Ribeiro, G. A. (Assistant Professor), Matias, G. (Assistant Professor), Ferreira, H. P. (Student), Salinas, H. (Assistant Professor), Antunes, H. S. (Associate Professor), Sousa, I. P. D. (Invited Assistant Teacher), Costa, I. G. (Student), Duarte, I. T. (Scholarship holder), Dalhuisen, J. (Invited Professor), Geng, J. (Student), Vasconcelos, J. (Associate Professor), Gomes, J. S. (Student), Lopes, J. F. (Student), Silva, J. P. D. (Assistant Professor), Lopes, J. A. (Associate Professor), Moutinho, J. L. (Associate Professor), Antunes, J. E. (Associate Professor), Proença, J. C. B. (Invited Associate Professor), Estaca, J. N. (Student), Domingos, J. L. (Student), Cunha, J. D. D. (Associate Professor), Rodrigues, J. (Student), Taborda, J. (Student), Araújo, L. (Student), Fábrica, L. (Assistant Professor), Naia, L. R. (Scholarship holder), Campos, M. F. (Associate Professor), Queiroz, B. M. (Assistant Professor), Cunha, M. D. C. (Associate Professor), Baptista, D. (Assistant Professor), Ribeiro, M. D. F. (Associate Professor), Ferreira, M. E. (Assistant Professor), Quadros, I. (Assistant Professor), Tavares, M. I. (Assistant Professor), Campos, I. M. (Assistant Professor), Martins, M. D. (Assistant Professor), Faria, M. P. R. D. (Associate Professor), Xavier, R. (Full Professor), Faria, R. L. D. (Assistant Professor), Rocha, V. (Assistant Professor), Almeida, M. A. D. (Full Professor), Domingos, M. P. (Scholarship holder), Rosas, M. (Invited Assistant Teacher), Portocarrero, M. (Assistant Professor), Rouxinol, M. (Assistant Professor), Alija, N. (Student), Silva, N. S. E. (Invited Assistant Professor), Martins, P. F. (Assistant Professor), Pichel, P. (Invited Assistant Teacher), Cunha, P. O. (Associate Professor), Albuquerque, P. P. D. (Full Professor), Marques, P. G. (Assistant Professor), Machete, P. (Associate Professor), Gomes, P. C. (Invited Assistant Professor), Freitas, P. (Assistant Professor), Carvalho, R. (Associate Professor), Moreira, R. R. (Scholarship holder), Andrade, R. (Invited Assistant Teacher), Morais, R. D. (Full Professor), Medeiros, R. (Full Professor), Duarte, R. P. (Full Professor), Carvalho, C. D. O. (Associate Professor), Tavares, S. (Assistant Professor), Vasques, S. (Associate Professor), Venda, S. B. E. (Student), Maltez, S. (Scholarship holder), Rendas, T. (Invited Assistant Professor), Tavares, T. (Assistant Professor), Ventura, V. (Student) & Nogueira, J. F. P. (Assistant Professor)
1/01/20 → 31/12/25
Project: Research