While chatbots are turning more and more intelligent, they are often seen to miss the human sense or emotion. Google is trying to change this with Meena, a chatbot which can conduct conversations that are more sensible and specific than existing state-of-the-art chatbots.
In a Google AI post, posted by Daniel Adiwardana, Senior Research Engineer, and Thang Luong, Senior Research Scientist, Google Research, Brain Team, the researchers say that the chatbot was created specifically to address one big and critical flaw of chatbots – they often do not make sense.
Existing human evaluation metrics for chatbot quality tend to be complex and do not yield consistent agreement between reviewers. This motivated the researchers to design a new human evaluation metric, the Sensibleness and Specificity Average (SSA), which captures basic, but important attributes for natural conversations.
The researchers say, “We show that Meena can conduct conversations that are more sensible and specific than existing state-of-the-art chatbots. Such improvements are reflected through a new human evaluation metric that we propose for open-domain chatbots, called Sensibleness and Specificity Average (SSA), which captures basic, but important attributes for human conversation,” To compute this score, Google took the crowdsourcing route and asked human workers to rank every conversation with the chatbot.
The Meena model has 2.6 billion parameters and is trained on 341 GB of text, filtered from public domain social media conversations. Compared to an existing state-of-the-art generative model, Meena has 1.7x greater model capacity and was trained on 8.5x more data.
Also Read:
Adding Human Touch To Unchatty Chatbots May Lead To Bigger Letdown