Robot Conversation Visualizer

This project explores the visualization of dyadic conversations between a social robot and a child. The Haru robot is an embodied conversational agent that can engage with children, developed by the Honda Research Institute. The project aims to visualize the conversation between the robot and the child in real-time, using a web-based interface.

The interface can display conversations as Sankey diagrams, Arc diagrams, and as a standard chatbox. The interface groups utterances from both speakers into intents and topics, and displays them in a visually appealing and informative way. Additionally, the interface can display the sentiment of the conversation, as well as the sentiment of individual utterances.

The front end was built in next.js, and the back end was built in Python using a Flask server communicating via websocket connection for real-time conversation updates. To actually record conversation data I created a robot operating system (ROS) Subscriber node to receive data from the robot and send it to the server. An additional tool was created to emulate custom robot conversations for testing purposes without needing to use the physical robot. This tool was built with GPT-4o and huggingface models for sentiment analysis and emotion classification values.

The visualization tool has been used to visualize child-robot interaction in order to promote responsible AI use in child education and to provide caretakers with an accessible platform to monitor child-robot interactions. Furthermore, the platform has also been used by developers of the robot to monitor the robot's performance and to improve the robot's conversational abilities. The platform is currently being used by researchers in studies for how visual data literacy can help with AI literacy. The research is still ongoing.