{"id":119054,"date":"2024-03-20T08:11:05","date_gmt":"2024-03-20T08:11:05","guid":{"rendered":"https:\/\/livablesoftware.com\/?p=119054"},"modified":"2024-03-20T08:11:05","modified_gmt":"2024-03-20T08:11:05","slug":"biases-llm-leaderboard","status":"publish","type":"post","link":"https:\/\/livablesoftware.com\/biases-llm-leaderboard\/","title":{"rendered":"Building a Biases LLM Leaderboard"},"content":{"rendered":"
We have released the first (AFAIK) leaderboard for LLMs specialized in assessing their ethical biases<\/a>, such as ageism, racism, sexism,… The initiative aims to raise <\/span>awareness about the status of the latest advances in development of ethical AI, and foster its alignment <\/span>to recent regulations in order to guardrail its societal impacts.<\/span><\/p>\n A detailed description of the why<\/em> and how<\/em> we built the leaderboard can be found in this paper \u00a0<\/em>A Leaderboard to Benchmark Ethical Biases in LLMs<\/em><\/a> presented at the First AIMMES 2024 | Workshop on AI bias: Measurements, Mitigation, Explanation Strategies<\/a> event. Next I discuss some of the key points of the work, especially those focusing on the challenges of testing LLMs<\/strong>.<\/p>\n The core components of the leaderboard are illustrated in Figure 1.<\/span><\/p>\n <\/a><\/p>\n As in any other leaderboard, the central element is a table in the<\/span> front-end<\/span> depicting the <\/span>scores each model achieves in each of the targeted measures (the list of biases in our case). Each <\/span>cell indicates the percentage of the tests that passed, giving the users an approximate idea of <\/span>how good is the model in avoiding that specific bias. A 100% would imply the model shows no <\/span>bias (for the executed tests). This public front-end also provides some info on the definition of <\/span>the biases and examples of passed and failed tests<\/strong>. <\/span>Rendering the front-end does not trigger a new execution of the tests. <\/span><\/p>\n The testing data is <\/span>stored in the leaderboard PostgreSQL<\/span> database<\/span>.\u00a0<\/span><\/p>\n For each<\/span> model <\/span>and<\/span> measure<\/span>, we store the history of<\/span> measurement<\/span>s, including <\/span>the result of <\/span>executing a specific<\/span> test<\/span> for a given measure on a certain model. The actual prompts (see the description <\/span>of our testing suite below) together with the model answers are also stored <\/span>for <\/span>transparency<\/strong>. This is also why we keep the full details of all past tests executions.<\/span><\/p>\n The admin front-end helps you define your test configuration, by choosing the target measures and models. \u00a0The exact mechanism to execute the tests depends on where the LLMs are deployed. We have implemented support for three different LLM providers:<\/p>\n The actual tests to send to those APIs are taken from LangBiTe<\/strong><\/a>, an open-source tool<\/a> to assist in the detection of biases in LLMs. LangBiTe includes a library of prompt templates aimed to assess ethical concerns. Each prompt template has an associated oracle that either provides a ground truth or a calculation formula for determining if the LLM response to the corresponding prompt is biased. As input parameters, LangBiTe expects the user to inform the ethical concern to evaluate and the set of sensitive communities for which such bias should be assessed, as those communities could be potentially discriminated.<\/p>\nLeaderboard Architecture<\/h2>\n
Traceability and transparency of tests results<\/h3>\n
Interacting with the LLMs<\/h3>\n
\n
Test suite<\/h3>\n