Japan’s exam-taking robot does alright on mathematic test run

An artificial intelligence system under development by the Japanese government and several technology companies successfully answered 4 of 10 math questions on a sample college entrance exam, according to a press release from project member Fujitsu on Monday. The grand goal of the project, known as Todai Robot for short, is to pass the University of Tokyo entrance exams by 2021 (currently between 80 percent and 90 percent for the first-stage exam, and then 30 percent to 40 percent for the second, more-difficult stage) using algorithms that are efficient enough to run on a laptop computer.

Fujitsu is leading the project’s efforts at passing the math portion of the exam, and it tested out the current version on practice exams administered by a popular college prep academy called Yoyogi Seminar. According to the press release, “[T]he artificial brain of Todai Robot automatically solved two out of four University of Tokyo math questions for humanities course, and two of six math questions for science course. Among all exam takers for humanities and sciences, the results represented a deviation value of approximately 60.”

The general Todai Robot method of breaking down a math problem.
The general Todai Robot method of breaking down a math problem.

Deviation scores are a uniquely Japanese method of scoring exams (and I have reached out to Fujitsu for further clarification on what that score means), but some online digging suggests it means the robot was one deviation above the mean, or in the 85th percentile.

Aside from proving its early capabilities, the text exam also highlighted some areas for future research. For starters, it still needs to be better at text analysis (in fact, humans actually helped with it on this test run and it didn’t fare tool well on some recent natural-language processing test runs) and faster at solving problems, and its mathematics dictionary needs to be expanded to include certain words it did not know.

The actual results from the practice test.
How the system broke down a problem on the practice test.

As I explained when reporting on the Todai Robot project last year, keeping it maintained to a single laptop will make the technology more relevant to users who don’t have access to a supercomputer or even a small cluster. As with other similar technologies, such as IBM’s Watson, the ultimate goal isn’t actually to pass a university exam but to empower people faced with similarly difficult computational problems in whatever fields they might be working. The more computationally efficient its algorithms and processing techniques are, the better.