ChatGPT struggles with coding problems past its 2021 cutoff date

OpenAI’s ChatGPT was able to code better than humans in some cases, but they did carry security concerns

Updated - July 08, 2024 04:52 pm IST

Published - July 08, 2024 04:18 pm IST

FILE PHOTO: ChatGPT was good at solving coding problems before its training cutoff date of 2021, but the chatbot struggled to solve problems after that year.

FILE PHOTO: ChatGPT was good at solving coding problems before its training cutoff date of 2021, but the chatbot struggled to solve problems after that year. | Photo Credit: Reuters

OpenAI’s ChatGPT was good at solving coding problems before its training cutoff date of 2021, but the chatbot struggled to solve problems after that year, according to a study published by IEEE Transactions on Software Engineering. 

ChatGPT’s ability to solve coding problems dropped by a wide margin if the problem set was outside its training data as language model was unable to even understand the meaning of basic problems, the study noted. 

The AI chatbot’s success rates of 89% (easy), 71% (medium) and 40% (hard) for coding problems dipped after 2021 to 52% (easy). For hard problems, the rate fell to 0.66%. 

The study covered 728 coding problems from Leetcode testing website in five programming languages -- C, C++, Java, JavaScript and Python. The chatbot’s prowess for tested on functionality, complexity and security. 

(For top technology news of the day, subscribe to our tech newsletter Today’s Cache)

The research also highlighted that within the Leetcode dataset, ChatGPT was able to code better than humans in some cases, but carried security concerns. ChatGPT was also good at fixing compiling errors in a code but not very good at correcting its own mistakes. One of the researchers involved in the study, Yutian Tang from the University of Glasgow stated, “ChatGPT may generate incorrect code because it does not understand the meaning of algorithm problems, thus, this simple error feedback information is not enough.” 

Tang said it was important to understand exactly where these AI models were limited so coders using them could find ways to work around it. “For example, when encountering more complex programming problems, developers can provide relevant knowledge as much as possible, and tell ChatGPT in the prompt which potential vulnerabilities to be aware of,” he added.

0 / 0
Sign in to unlock member-only benefits!
  • Access 10 free stories every month
  • Save stories to read later
  • Access to comment on every story
  • Sign-up/manage your newsletter subscriptions with a single click
  • Get notified by email for early access to discounts & offers on our products
Sign in

Comments

Comments have to be in English, and in full sentences. They cannot be abusive or personal. Please abide by our community guidelines for posting your comments.

We have migrated to a new commenting platform. If you are already a registered user of The Hindu and logged in, you may continue to engage with our articles. If you do not have an account please register and login to post comments. Users can access their older comments by logging into their accounts on Vuukle.