AI Chatbot in Mixed Reality

This is a conversational AI chatbot built for mixed reality. It utilizes Meta's MRUK to scan your real-world environment before the experience begins. Press and hold the 'A' button on the right-hand controller to talk to the AI character. Enjoy!

Development process:
Setting up the Convai API was quite challenging, as Convai had recently switched to a newer, different pipeline for conversing with the Avatar. I had to download an additional package from Convai and install the plugin into my Unity project, then connect my API key to the Convai platform in Unity. The avatar is programmed to move to the object that it detects in your real scene. However, it does not detect the scene in real-time. You need to scan the environment first, which labels all the objects in the scene. Once that's set up, the character will compare the name you said with the object label in the scene and walk towards that destination. You can also ask the avatar what they see in the room. However, it will not provide a detailed reply, as it only sees the pre-scanned object prefab with labels. You will, however, get an overall understanding of the surroundings. During playtesting, I found that if you're in a room with a large bed, it's best to delete the scanned bed to make room for the avatar to walk around. Otherwise, it may clip into the bed and play its walking animation, but it won't move.

That said, one improvement that can be made is to decrease the capsule collider of the avatar so it can walk around if there's something big in its way. Another improvement is to make the avatar more aware of its surroundings. For example, when I ask about any information in the environment, I can access my Quest camera to snap a picture of the scene and send the image to an LLM alongside my verbal input. This will return a verbal output to the avatar, making the entire experience more realistic.

GitHub Repo:

https://github.com/DannyHYM/botbot

Video Demo:

Built by Danny Huang