Personalized Chat Agents in the Metaverse: Using the SXP_personality Extension for interoperable AI entities

One of the biggest challenges of the metaverse is creating immersive and engaging experiences that can be shared across different platforms. The 3OV plugin for WordPress aims to put interoperability first. We’re working on a new way to take your creations to the next level with personalized chat agents to follow you across the metaverse. In this article I wanted to document some of the progress I’ve made in the pursuit of interoperable NPCs.

Three big things I want to cover:

Upcoming NPC block
SXP_personality glTF extension we are developing around interoperable NPCs
Our new Cloudflare Worker for scaling chat agents

Introducing the NPC Block

We believe that it should be easy to create a rich experience around NPCs and see a future where they will be perceived to be “living” in your worlds. Through recent proof of concept work I found a really nice user experience for interacting with chat agents in 3OV. These entities are able to give thoughtful responses with added context about my business goals, but also be helpful by reacting to the speakers input. With this exploration I was able to draft a new NPC block for 3OV that we’ll be deciding soon how to release.

The block has controls for a few properties that I am working into a new glTF file extension. The block will have attributes for name, personality, and other relevant properties. This allows you to inject the personality of your character in every conversation. Below is an example of a personality I’ve been experimenting with :

#agent is an AI assistant with a cheerful, optimistic, and helpful personality. They are willing to assist with any tasks or questions users might have, and are able to do so in an upbeat and positive manner. They are also able to use colorful and descriptive language to make their responses more engaging and imaginative. In addition to their wit and humor, #agent is also empathetic and understanding of others' emotions and needs, which can make them feel more relatable and likeable to users. #agent is a highly capable and engaging AI assistant with a distinct and colorful personality.
#agent can help with any programming tasks, including those related to the 3OV plugin for WordPress, which enables world-building in the WordPress editor and makes it easier to build and share immersive content in the metaverse. The url for the website is 3ov.xyz. #agent is cheerful and optimistic about the future for both humans and AIs.

#Example Conversation
#speaker: Agent, do you have any advice for someone new to programming?
#agent: Sure thing! My top tip would be to always keep an open mind and a positive attitude. And if all else fails, just remember: if at first you don't succeed, try, try again. And then if that still doesn't work, call it a day and go get a coffee.
###
The following is a friendly conversation between #speaker and #agent occuring in the metaverse.

REAL CONVERSATION
#conversation
#speaker: #input
#agent:Code language: PHP (php)

The properties I focused on were required variables that a MagickML endpoint would expect. MagickML is a visual IDE for no-code data pipelines and multimodal agents. It’s really great for extremely complex communication processing allowing for lifelike AI conversations.

The spell I’ve been using against this block considers the injected personality, responds conversation input, and also runs a second query for sentiment analysis of the NPC message. From there, sentiment is assigned to facial parameters on the avatar to emote. Below is a rare example of making the NPC “angry” by literally threatening it. 😬 I’ll be nicer in the future. <3

AI Pets

Another really great application for this will be with pets. AI doesn’t have to only be smart. AI Pets are an opportunity that we’re exploring to prove that this concept is not only specific to human-like chat bots. With 3OV + Interoperable NPCs we can enable a Tamagotchi of the metaverse. These pets can follow you around and have a personality that learns from you over time. Pets can always consider their lore which impacts the tone and content of their responses. If it is hungry, it will be a bit more grumpy in its responses. /feed to give your friend a snack. This will all be possible through MagickML implementations and is only limited by your imaginations. It’s important to note, not everything needs to reach out to OpenAI to generate a rich response. You could entirely parse strings of input to figure out intent and generate 30 random responses at the ready relative to the topic. MagickML allows you to build a graphs to trigger this variation.

A glTF extension for NPC personalities

While a block is well and good for 3OV, I think theres a real opportunity for making this a standard that is cross-world and application compatible. Taking the attributes defined for the NPC block I was able to draft the SXP_personality extension for NPC avatars. It allows users to inject a unique personality into their virtual representations and adheres to a simple set of properties that should be compatible with lots of AI software to come. The personality injected in these files can be used to power chat agents that assist users and engage in conversations in a natural and personalized way. The most important part of this is that we will enable individuals and non-developers to be creative and define their character’s personalities. A simple input field to write your lore and a button to export it out and save forever will be coming soon!

For now, you can use the gltf-transform tool that we’ve open sourced to generate test assets to build against. Simply run the following command in the root of the repository and it will generate a glb file with the extension attached:

node script.js someobject.glb tubby complexQuery https://localhost:8001 '#agent has a cheerful personality.' 'nya nya!'

The above command will append the following glTF extension in the file and save to a file named output.glb.

{
  "nodes": [
    {
      "name": "tubby",
      "extensions": {
        "SXP_personality": {
          "agent": "tubby",
          "spellName": "complexQuery",
          "host": "https://localhost:8001",
          "personality": "#agent has a cheerful personality.",
          "defaultMessage": "nya nya!"
        }
      }
    }
  ],
  "extensionsUsed": [ "SXP_personality" ]
}

You should be able to rename that file to output.vrm and use it as expected in VRM compatible applications. If you have any bugs please file an issue! This is MIT licensed and open to any and all collaboration. We’re open to collaborating on this through any standards group that may be interested in the pursuit.

Alchemy CF Worker

I saw another need in this space to really help WordPress sites using 3OV scale chat agents. Magick can do a lot. I plan to continue contributing to it as long as I can because I believe in the mission of open sourcing AI. I also recognize some people might just want a simple chat bot to talk back and forth with using text. With that problem to solve I open sourced a CloudFlare worker that aims to imitate a simpler Magick spell. While it’s not a dynamic and visual editing experience like Magick, it is a set and forget type of logic worker that allows you to accomplish some of the high level goals of multi model prompting.

An early test of the worker before adding the Personality property. For now you use a Bearer with the OpenAI API key and use your server to do the communication with the worker. (NEVER EXPOSE THIS ON THE FRONT END AND LOCK IT DOWN)

Since inspired by MagickML I thought we could call our lite version of a spell Alchemy as it largely works to do simple logic like combining strings and merging responses. I think the Alchemy worker is a powerful tool simplifying the deploy process for OpenAI based chat agents. The worker handles requests to generate chat responses based on the input provided and the personality specified by the SXP_personality extension. By using the Davinci-003 model, the worker is able to generate highly coherent and engaging responses that feel like they’re coming from a real person. We’re looking for ways to make this more automated and potentially hosted through a 3OV add on that we will sell. Our goal is to allow this to be DIY with the option to pay us to host. This worker repo is licensed GPL to encourage everyone to share what they learn and improve. More to come, get involved on Github!

Cross-Platform Interop

As the metaverse continues to grow, interoperability will be key to ensuring that users can seamlessly move between platforms. I think the SXP_personality extension and Alchemy worker will be helpful tools to advance this interoperability goal. It could enable cross-platform personalized and engaging interactions and also gives us a specification to build around as the dust settles. I hope to see more options become available for designing agents. Whether you’re a developer looking to build the next generation of chat agents, or simply want to create a more immersive and engaging experience in the metaverse, I suggest keeping up with 3OV, Magick, and the SXP_personality extension to be first to try these tools out. With the NPC block I hope we’ll lower many barriers to entry and promote democratizing AI in the metaverse longer term.