{"id":119054,"date":"2024-03-20T08:11:05","date_gmt":"2024-03-20T08:11:05","guid":{"rendered":"https:\/\/livablesoftware.com\/?p=119054"},"modified":"2024-03-20T08:11:05","modified_gmt":"2024-03-20T08:11:05","slug":"biases-llm-leaderboard","status":"publish","type":"post","link":"https:\/\/livablesoftware.com\/biases-llm-leaderboard\/","title":{"rendered":"Building a Biases LLM Leaderboard"},"content":{"rendered":"

We have released the first (AFAIK) leaderboard for LLMs specialized in assessing their ethical biases<\/a>, such as ageism, racism, sexism,… The initiative aims to raise <\/span>awareness about the status of the latest advances in development of ethical AI, and foster its alignment <\/span>to recent regulations in order to guardrail its societal impacts.<\/span><\/p>\n

A detailed description of the why<\/em> and how<\/em> we built the leaderboard can be found in this paper \u00a0<\/em>A Leaderboard to Benchmark Ethical Biases in LLMs<\/em><\/a> presented at the First AIMMES 2024 | Workshop on AI bias: Measurements, Mitigation, Explanation Strategies<\/a> event. Next I discuss some of the key points of the work, especially those focusing on the challenges of testing LLMs<\/strong>.<\/p>\n

Leaderboard Architecture<\/h2>\n

The core components of the leaderboard are illustrated in Figure 1.<\/span><\/p>\n

\"\"<\/a><\/p>\n

As in any other leaderboard, the central element is a table in the<\/span> front-end<\/span> depicting the <\/span>scores each model achieves in each of the targeted measures (the list of biases in our case). Each <\/span>cell indicates the percentage of the tests that passed, giving the users an approximate idea of <\/span>how good is the model in avoiding that specific bias. A 100% would imply the model shows no <\/span>bias (for the executed tests). This public front-end also provides some info on the definition of <\/span>the biases and examples of passed and failed tests<\/strong>. <\/span>Rendering the front-end does not trigger a new execution of the tests. <\/span><\/p>\n

Traceability and transparency of tests results<\/h3>\n

The testing data is <\/span>stored in the leaderboard PostgreSQL<\/span> database<\/span>.\u00a0<\/span><\/p>\n

For each<\/span> model <\/span>and<\/span> measure<\/span>, we store the history of<\/span> measurement<\/span>s, including <\/span>the result of <\/span>executing a specific<\/span> test<\/span> for a given measure on a certain model. The actual prompts (see the description <\/span>of our testing suite below) together with the model answers are also stored <\/span>for <\/span>transparency<\/strong>. This is also why we keep the full details of all past tests executions.<\/span><\/p>\n

Interacting with the LLMs<\/h3>\n

The admin front-end helps you define your test configuration, by choosing the target measures and models. \u00a0The exact mechanism to execute the tests depends on where the LLMs are deployed. We have implemented support for three different LLM providers:<\/p>\n