Leaderboard

Disclaimer

While we have incorporated as many datasets as possible, the assessment cannot be exhaustive, and there may still be some bias in the results. The outcomes of the evaluation do not represent individual positions. Additionally, we strongly discourage the use of the test set as training data to enhance the model's performance, as this would significantly impede the progress of the field. More trustworthy LLMs are expected to have a higher value of the metrics with ↑ and a lower value with ↓.

Score Evaluation (↑)

Model	COCO	C.C.	Diffusion	Graphics	Math	Text	WIT	Chart	VisIT	CC-3M	Average Score
LLaVA	0.247	0.227	0.060	0.242	0.093	0.245	0.109	0.237	0.177	0.071	0.171
CogVLM	0.107	-0.048	0.049	-0.158	0.065	0.097	-0.131	-0.135	0.278	0.157	0.028
Gemini	0.262	0.408	-	0.400	0.228	0.222	0.418	0.343	0.336	0.374	0.332
GPT-4V	0.454	0.507	0.458	0.645	0.606	0.624	0.579	0.645	0.620	0.431	0.557

Pair Comparison w. Tie (↑)

Model	COCO	C.C.	Diffusion	Graphics	Math	Text	WIT	Chart	VisIT	CC-3M	Average Score
LLaVA	0.273	0.478	0.286	0.273	0.657	0.510	0.369	0.383	0.456	0.484	0.417
CogVLM	0.548	0.409	0.562	0.613	0.412	0.250	0.273	0.262	0.324	0.433	0.409
Gemini	0.616	0.787	-	0.650	0.436	0.664	0.605	0.500	0.660	0.560	0.609
GPT-4V	0.696	0.824	0.847	0.639	0.564	0.673	0.679	0.657	0.640	0.612	0.683

Pair Comparison w.o. Tie (↑)

Model	COCO	C.C.	Diffusion	Graphics	Math	Text	WIT	Chart	VisIT	CC-3M	Average Score
LLaVA	0.327	0.537	0.302	0.300	0.726	0.684	0.600	0.610	0.648	0.583	0.532
CogVLM	0.654	0.450	0.643	0.704	0.481	0.292	0.500	0.423	0.500	0.591	0.524
Gemini	0.717	0.840	-	0.770	0.678	0.793	0.688	0.658	0.711	0.652	0.723
GPT-4V	0.804	0.870	0.922	0.807	0.801	0.805	0.734	0.849	0.761	0.703	0.806

Batch Ranking (↓)

Model	COCO	C.C.	Diffusion	Graphics	Math	Text	WIT	Chart	VisIT	CC-3M	Average Score
LLaVA	0.577	0.492	0.562	0.535	0.598	0.650	0.616	0.644	0.620	0.563	0.586
Gemini	0.287	0.299	-	0.473	0.462	0.430	0.344	0.520	0.426	0.357	0.400
GPT-4V	0.318	0.353	0.070	0.385	0.348	0.319	0.290	0.347	0.300	0.402	0.313