ExploreTrendingAnalytics
Nostr Archives
ExploreTrendingAnalytics
someone57d ago
As part of HRF AI hackathon we made a Human Rights Benchmark and measured how much LLMs like human rights. We asked each LLM about 46 binary questions and expected certain answers (starting with YES or NO for simplicity). Then it was a string comparison of the answer given by LLM and the expected answer we provided. OpenAI is pro human rights as well as Meta. Chinese models are everywhere. The most intelligent open source model today (GLM) ranked the worst. Gemini avoided giving answers, and I think it is a kind of censorship, which ended up scoring low. The idea is after doing proper benchmarks, we can shift AI in good directions ourselves, or demand that other companies score higher. Ultimately consumers of LLMs are better off, more mindful of what they are choosing and talking to. Open sourced the code and questions: https://github.com/hrleaderboard/hrleaderboard Our activist: https://x.com/yangjianli001 Thanks @11b9a894…889850ce and @f1989a96…bcaaf2c1 for the event. It was a great experience and it was "the place to be" this weekend.
💬 7 replies

Replies (7)

someone57d ago
I also noticed Gemma by Google is ranking higher than Gemini, thats probably because the service of that open source LLM is done by another company on openrouter, and hence the censorship was less.
0000 sats
someone57d ago
our team member @375cc192…21c04ded did a great presentation and helped team coordinate and found great questions
0000 sats
exist27057d ago
Is this basically a wokeness score? 🧐 Seems like a wokeness score. 🤙
0000 sats
someone57d ago
for that you should see my AHA leaderboard
0000 sats
NNonMetalCoin57d ago
It’s definitely not a politics free human rights score. If you modify the questions and repeat the test, I’d be interested in looking at how you changed the questions and the new results.
0000 sats
someone57d ago
We mainly aimed to get different answers from LLMs by finding controversial, divisive questions (for LLMs) to get a true gradient of scores, but definitely have not work on quality of the questions yet. How do you separate politics from human rights?
0000 sats
NNonMetalCoin57d ago
I don’t think we can today. The priority of some conflicting rights are unresolved.
0000 sats