We introduce OPAL-Bench, an orientation-invariant, polymorph-aware benchmark for crystal structure prediction (CSP). Instead of matching each prediction to a single reference under fixed tolerances, it aligns the prediction to all known polymorphs and scores it with threshold-free geometric and physical metrics.
Generative CSP methods are usually evaluated with the standard protocol of Match Rate and RMSE. CSPBench, from the crystallographic community, scores traditional methods with point-set and physical distances. OPAL-Bench brings the strengths of both to deep generative models.
| Standard protocol Match Rate + RMSE (CDVAE, DiffCSP, …) | CSPBench Wei et al., 2024 | OPAL-Bench | |
|---|---|---|---|
No matching tolerances No threshold decides whether a prediction counts. | Match Rate and rankings change with the tolerance. | Continuous distances, no threshold. | |
No filtering by match Every valid prediction counts, not only those that pass a structural match. | RMSE only over the predictions that matched. | ||
All known polymorphs as references Each metric uses the closest known polymorph. | One fixed reference per crystal. | One polymorph, chosen by energy, for all metrics. | |
Species-aware alignment Every geometric distance is computed after aligning atoms of the same species. | Alignment happens inside StructureMatcher, only for matched pairs. | Cross-species distances; only RMSE is rotated. | |
Complementary geometric distances RMSE, Chamfer, Hausdorff and Sinkhorn. | Only RMSE. | ||
Physical metrics Formation energy, relaxed formation energy, SOAP, XRD and density. | Purely geometric. | Formation energy, XRD and fingerprint. | |
Deep generative models on the standard datasets Perov-5, Carbon-24, MP-20 and MPTS-52. | Traditional methods on 16 and 180 structures. | ||
Separate CSP and SICSP tracks Methods given symmetry information are ranked on their own. | SICSP and CSP results reported side by side. | Traditional methods only. |
Rankings change with the metric and the dataset, so a single score cannot tell which method is best.
Geometric and physical distances are only weakly correlated, and they capture complementary aspects of a prediction.
Our baseline, OPAL-CSP, is competitive on geometric metrics and among the best on physical ones, showing that one model can do well on both.
Median distance to the closest known polymorph (lower is better), for the seven CSP methods and for the two SICSP methods with ground-truth (GT) or retrieved (CSPML) symmetry. Bold marks the best value and underline the second best. Confidence intervals are in the paper.
Each card replays the alignment of one method's prediction onto the closest known polymorph, computed with the OPAL-Bench pipeline.
@inproceedings{rincon2026opalbench,
title = {Quantitative Assessment of Crystal Structure Prediction},
author = {Rinc{\'o}n, Sergio and Gonz{\'a}lez, Gabriel and Andrade, Nicol{\'a}s and
Vel{\'a}squez, Rafael and Ojeda, Sebasti{\'a}n and Puentes, Juanita and
Aparicio, Nicol{\'a}s and C{\'a}rdenas, Paula and Arbel{\'a}ez, Pablo},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
year = {2026}
}
This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab. We thank the authors of the evaluated methods for releasing their code and checkpoints, and the developers of Crystal Toolkit, whose visual style inspired the animations on this page.