Do Large Language Models Know Their Chemistry? - An Evaluation of Open-Source LLMs for Chemical Reaction Feasibility

Abstract

In drug discovery, a crucial task is determining whether two molecules can form a specified product. Large language models (LLMs) have recently emerged as promising tools for such tasks, but their ability to reason about molecular structures beyond surface-level pattern memorization remains uncertain. This thesis evaluates various open-source LLMs on binary chemical reaction feasibility classification. In this task, the models must determine whether a candidate product can be formed from two given reactants. For this evaluation, we tested ten open-source models ranging from 7 to 80 billion parameters. The best-performing model was further evaluated using randomized SMILES and few-shot prompting. The results show that most models perform above the random baseline, with the best-performing model achieving an accuracy above 0.9 while maintaining a balanced error profile. We also found that parameter count is not the only predictor of performance; the model developer and architecture appear to play a more significant role, although larger models generally performed better than smaller ones. Randomizing the SMILES resulted in only a small decrease in accuracy, indicating that the models possess some capability to reason beyond surface-level memorization. Surprisingly, few-shot prompting did not improve performance but instead led to a small decrease in accuracy. Together, these findings suggest that some open-source LLMs can serve as useful tools for reaction feasibility assessment.