CULTURE-MT Leaderboard

CULTURE-MT is a benchmark for evaluating culturally-aware social media translation.

Participants submit English translations for the benchmark inputs. Submissions are evaluated by a private Judger API, and the aggregated scores are displayed on this leaderboard.

Current Results

Current Results

10	Gemma-4-31B-it__u_t25_mixed_1x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	10.51	91.92	12.31	10.51	51.56	40.35	2.31	2026-05-27 22:38:47


1	GPT-5	--	GPT-5	API	8.08	91.92	0.9	7.19	51.56	40.35	2.31	2026-05-27 22:38:47
2	Gemini 3 pro	--	Gemini 3 pro	API	8.95	91.05	0.2	8.75	51.97	39.09	2.3	2026-05-27 22:46:10
3	Gemma-4-31B-it__t25_only_1x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	10.51	89.49	0.97	9.55	55.76	33.73	2.22	2026-09-14 09:12:46
4	Gemma-4-31B-it__foundation__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	foundation	11.51	88.49	1	10.51	56.56	31.94	2.19	2026-09-14 08:56:09
5	Gemma-4-31B-it__t25_only_7x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	11.78	88.22	1.77	10.01	55.72	32.5	2.19	2026-09-14 09:15:44
6	CULTURE-MT-baseline-32B	ZJU & FDU & Xiaohongshu	Qwen3-32B	SFT	12.08	87.92	0.73	11.35	56.56	31.37	2.19	2026-05-27 18:26:06
7	Gemma-4-31B-it__u_t25_mixed_1x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	11.14	88.86	1.87	9.28	57.48	31.37	2.18	2026-09-14 09:36:28
8	Gemma-4-31B-it__u_t25_mixed_7x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	10.98	89.02	1.4	9.58	59.31	29.71	2.17	2026-09-14 09:40:34
9	Gemma-4-31B-it__u_x_t25_seq_1x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	12.28	87.72	2.4	9.88	56.46	31.27	2.17	2026-09-14 09:44:28
10	Gemma-4-31B-it__u_x_t25_seq_7x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	12.24	87.76	2	10.25	57.09	30.67	2.16	2026-09-14 10:26:30
11	Gemma-4-31B-it__u_en_only_1x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	12.05	87.95	1.07	10.98	59.78	28.17	2.15	2026-09-14 09:18:13
12	Gemma-4-31B-it__u_only_1x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	14	86	2.36	11.65	57.02	28.97	2.13	2026-09-14 09:26:31
13	CULTURE-MT-baseline-8B	ZJU & FDU & Xiaohongshu	Qwen3-8B	SFT	14.34	85.66	1.27	13.07	57.42	28.24	2.13	2026-05-27 18:33:54
14	GLM4.6	--	GLM4.6	API	15.94	84.06	3.12	12.81	57.39	26.68	2.08	2026-05-27 22:32:00
15	Gemma-4-31B-it__u_only_7x__redtrans-20260912	Xiaohongshu RedTrans research	Gemma-4-31B-it	full-parameter post-training	15.2	84.8	2.1	13.11	60.75	24.05	2.07	2026-09-14 09:33:12
16	Deepseek V3.2	--	Deepseek V3.2	API	17.73	82.27	2.59	15.14	56.82	25.45	2.05	2026-05-27 22:25:50
17	Qwen3-235B-A22B	--	Qwen3	API	29.27	70.73	12.31	16.97	52.86	17.87	1.76	2026-05-27 23:18:53
18	Seed-X-PPO	--	Mistral-7B	RL	30.58	69.42	3.33	27.25	62.23	7.19	1.73	2026-05-27 23:45:14
19	Qwen3-32B	--	Qwen3	Base	31.67	68.33	5.19	26.48	58.68	9.65	1.73	2026-05-27 23:25:55
20	Qwen3-14B	--	Qwen3	Base	38.95	61.05	12.77	26.18	54.59	6.46	1.55	2026-05-27 23:12:48
21	Qwen3-8B	--	Qwen3	Base	48.64	51.36	15	33.63	45.98	5.39	1.42	2026-05-27 23:37:09
22	Qwen3-4B	--	Qwen3	Base	60.41	39.59	21.33	39.09	37.92	1.67	1.2	2026-05-27 23:31:31
23	Qwen3-1.7B	--	Qwen3	Base	83.79	16.21	45.84	37.95	16.11	0.1	0.7	2026-05-27 23:05:14
24	Qwen3-0.6B	--	Qwen3	Base	94.84	5.16	74.65	20.19	5.09	0.07	0.31	2026-05-27 22:55:41