Commit Messages in the Age of Large Language Models

Abstract

Writing commit messages can be tedious and inconsistent among developers, motivating the question of whether large language models can help. We evaluate ChatGPT against existing automated approaches specifically trained on commit data for generating commit messages, finding that ChatGPT substantially surpasses prior methods in performance, generally producing accurate and high-quality outputs. We also document failure cases and categorize them for further analysis.

Iris Ma

Software Engineering Ph.D. Candidate

My research focuses on AI for reliable software engineering, including LLM memorization, hallucination detection, retrieval-augmented generation, and verified program synthesis.