Did Eugene Goostman really make the grade?

A chatbot, named Eugene Goostman and modelled to mimic a 13-year-old Ukrainian boy, is said to have passed the Turing test - making it the first robot to do so since the inception of the test over sixty years ago.

The Turing test was formulated by Alan Turing, the pioneer of artificial intelligence, in his famous paper of 1950, “Computing machinery and intelligence.”

To pass, a computer must convince its human interrogators that it is also human. Of course, the computer is hidden from the judges by a screen and the communication is through text messages.

In a contest held recently at the University of Reading, 30 judges were asked to interact with 10 contestants – five of whom were human and five were chatbots. The judges had to guess whether they were interacting with humans or robots through five-minute conversations.

Eugene Goostman, created by Vladimir Veselov and Eugene Demchenko, fooled one-third of the judges into thinking it was human, and this led to the announcement that it had passed the Turing test.

However, the announcement has provoked much criticism. Since the chatbot was given the personality of a Ukrainian who did not speak fluent English and who was, in addition, not an adult but a 13-year-old, this automatically gave the robot an advantage.

Even if it made mistakes in the conversation, these could be attributed to not knowing either the language or the context. So, in a way, the program was designed to “game the system.” Critics have also pointed out that the direct interpretation of Turing’s words in the implementation of the test.

Fooling the judges

What Turing may have meant by “fooling the judges into believing it was human” was perhaps that the quality of the “conversation” be close to that between two humans, whereas in this case it may have been the personality rather than the quality of the conversation that fooled the judges. This interpretation is quite valid, as the whole test was Turing’s way to quantitatively answer the question “Can robots think?”

“Turing never called it a test, he only called it an imitation game, in his 1950 paper. His goal was only to get away from arguments about whether or not a machine could think,” said Deepak Khemani, professor at IIT Madras. “It is not at all clear what intelligence is and whether we can say that a machine that has passed this test is intelligent.

“If we classify playing chess and checkers as intelligent behaviour, computers can already beat us at it. Perhaps we are taking this test too seriously.”