In 2019, I wrote an article for WP Tavern titled “Possibilities of a CMS in the Spatial Computing Future” . I explored the potential role of WordPress in powering Extended Reality (XR) experiences. Since then, we’ve seen an exciting evolution in the metaverse landscape, with spatial computing now being embraced globally by the largest technology companies and impressive devices that eliminate the need for controllers. Today, I stand by the vision I shared four years ago, but with way more insight and firsthand experience bringing 3D WordPress to life.

Prior to my 2019 article, I was focused exploring how WordPress could facilitate data consumption in platforms like Unity apps and other 3D platforms using the REST API. While these experiments are still relevant today, it was completely missing much bigger opportunities for WordPress in XR.

Since then, with the development of the Three Object Viewer (3OV) plugin, my perspective has shifted to: “WordPress is the 3D platform.” It’s no longer just the data layer, it can be the entire stack start to finish with zero third party dependencies. Instant immersive publishing for all!

I always like to take a step back and frame why I choose WordPress as the platform to build around versus building my own 3D focused CMS from scratch. It comes down to the following:

• Proven post type data structure
• Extendable data structure for users
• Secure user system
• Rich content editor
• Extendability through plugins and themes 
• Community <3

One thing that struck me recently was the realization just how positively impacting the introduction of the block editor was to adapting to the future of the web. It is no exaggeration to say that without the new editor, I would not have been able to bring this plugin to life. 3D content for WordPress would have likely still been limited to consumption by and of “better” apps.

The recent news…🍎

Apple recently announced their first serious step into XR with the Vision Pro headset which focuses heavily on mixed reality content. Many, including myself, believe that this was the signal needed to show the world that the immersive web was not only viable, but the inevitable future of the web. I am very happy to report that with Apple’s added support for WebXR in Safari, the headset will be compatible with 3OV making WordPress fully ready to activate on this device! 3OV comes standard with hand tracking for movement, and based on what I have read from Apple, will be compatible with the new headset. 🥳

Let’s Take a Quick Look Back at 3OV

Looking back at the past year since the public release of Three Object Viewer, the plugin has evolved from a simple model viewer into a comprehensive metaverse building experience.(They grow up so fast!) When the plugin first released in March of 2022 I really had no larger ambition than making a 3D model viewer. The hope was to make a simple viewer block that would allow folks to select a single file and share things like avatars or wearables.

With the introduction of the 3D Environment Block and nine core 3D inner block components, 3OV has transformed into a feature-rich 3D builder. I’m quite proud of where it has ended up!

I think one of the best new blocks was the introduction of the NPC Block. This block utilizes language models and our open sourced worker for plug and play chat bots in your 3D worlds. I see great potential in AI driven assistants that can live in virtual spaces with knowledge of you and your website to communicate with visitors in your absence.

Where to next for 3OV?

Now, as I chart the course for the next year, I’m setting my sights on several areas of focus for the plugin:

1. Networked Experiences + Multiplayer Editing
   • Networking is very close to finished. Once this lands my focus will shift to collaborative frontend editing using this network layer.
2. Third Person Controller
   • Introducing a third-person controller will enable visitors to interact with the virtual environment using avatars. Focus will then go to VR compatibility with these avatars. VRM will be the file type used.
3. Mixed Reality
   • Building upon the existing mixed reality features in 3OV, I will add configuration options for setting the display type. This will open up opportunities for creating mixed reality-specific blocks, such as a “Screen Block” for defining areas as screens or monitors.

4. Further enhanced AI
   • Expanding AI and NPC features to include multi-modal or multi-action capabilities will empower users to create more interactive experiences. The ability to program pre-defined tasks for virtual characters will bring a new level of interactivity to websites. I think this could power AI assistants that follow you in your everyday life.

I’ll be doing blog posts about each of these focuses as time goes on but I think this will be the best direction forward for the most impact.

What about the spatial future for WordPress?

This is where I have been noodling on the last few days since the Apple headset release. I think it is almost certain that WordPress will adapt to a spatial computing future in more ways than just the front end of a website. The admin dashboard will likely have great opportunity in giving us truly distraction free writing. Imagine grabbing the sidebar of the editor and pinning it to a corner of your desk. An avatar embodied AI assistant is on the other side of the desk providing you paragraphs of data relative to what you are writing about on screen. You go down the street to get tacos and your AI friend is there still being rendered from your website ready to have a chat while you wait for your friends to arrive. You snap a photo and attach it to a blog post. (This is all mostly possible with 3OV today!)

So anyway, I’m rambling. It’s safe to say that I’m excited to see WebXR being supported by practically all XR headset makers. This builds a more open 3D web. The future of WordPress in this space is obvious now more than ever. Back in 2019, I imagined WordPress powering basic 3D data visualization. Today, I celebrate it as a beacon of the open 3D web, and I can’t wait to see how far we can take it in the years to come.

antpb

Artificial intelligence (AI) has come a long way in just the last year, and one area that has seen significant progress is natural language processing. This technology allows computers to understand and generate human-like language, enabling them to communicate with people in a more natural and intuitive way. With new technology comes new CMS-like software to enable anyone to participate. Enter MagickML, a multishot system builder that leverages the capabilities of GPT-3 and other OpenAI language models to create AI assistants and characters with complex communication patterns. It is, in a sense, a WordPress of AI in that it manages AI themes and content. In this post, we’ll take a closer look at Magick and explore its potential to revolutionize the way we interact with AI in virtual worlds, particularly in 3OV.

What is Magick?

MagickML is a JavaScript-based framework that allows game designers and developers to rapidly create powerful natural language systems and prototype NPC logic in any web based experience. It utilizes a visual coding style interface, making it easy for users to create “spells” that define complex logic and queries to generate responses. These spells are almost like WordPress themes for AI, allowing users to customize the behavior and communication patterns of their AI assistants and characters. Being JavaScript driven allows for a much wider range of developer to participate. And the icing on the cake, it is open source and Apache Licensed!

One of the key features of Magick is its ability to integrate with GPT-3 and any other OpenAI language models to logically respond to requests. This enables users to leverage the advanced capabilities of these models to create AI assistants and characters that can understand and generate human-like language. You can use any mixture of language models to generate a final response.

The framework also allows users to incorporate memory recall or factual biographical information into their responses, as well as any other logic they would like to include to make it feel more natural. Between the final output of an entity, users are able to utilize JavaScript logic to parse many responses from OpenAI to build a final query that generates a more human-like response. Where this differs from something like chatGPT is its ability to send multiple request before giving the end user final output. This gives you room to tweak all the funny bits of language models where a single query might not give you the response you want.

The Potential of Magick:

The potential of Magick facilitating the creation of complex communication patterns with OpenAI language models is vast. In the gaming industry, Magick could be used to create more immersive and engaging experiences for players using AI characters with personality traits, making the game world feel more real and alive.

I found the potential of chatbots and virtual assistants particularly interesting for 3D web applications. By using Magick to create AI assistants, businesses and organizations could provide more personalized and efficient customer service. Your AI assistant can sell for you within your website. And with 3OV this could even be in a 3D world!

I built a Magick driven proof of concept in 3OV that allows WordPress site owners to define a URL where their Magick instance lives and make post requests from in-world to display responses above an avatar’s head. This avatar is given a personality through “spells” and is able to remember facts about each person that talks thanks to the awesome work Magick contributor m00n has built and shared with their Eliza entity.

Example Integration

The following integration of Magick in 3OV very basically shows how you can integrate spells and responses in your apps.

function ChatBox(props) {
	const handleChange = async (event) => {
		event.preventDefault();
	};
	const handleSubmit = async (event) => {
	  event.preventDefault();
	  // Get the value of the input element
	  const input = event.target.elements.message;
	  const value = input.value;
  
	  // Send the message to the localhost endpoint
	  const client = 1;
	  const channelId = "three";
	  const entity = 11;
	  const speaker = "antpb";
	  const agent = "aiko";
	  const channel = "homepage"; 

	  try {
		const spell_handler = "four";
		const spell_version = "latest";
		const url = encodeURI( `https://localhost:8001/spells/${spell_handler}/${spell_version}` )
		const response = await axios.post(`${url}`, {
			inputs: {
			  Input: value,
			  Speaker: speaker,
			  Agent: agent,
			  Client: client,
			  ChannelID: channelId,
			  Entity: entity,
			  Channel: channel,
			},
		  }).then((response) => {
			const data = response.data;

			const outputs = data.outputs;

			const outputKey = Object.keys(outputs)[0];

			const output = outputs[outputKey];

			props.setMessages([...props.messages, output]);
		  });
		} catch (error) {
		console.error(error);
	  }
	};
   
	return (
	  <div style={{ marginTop: "-140px", position: "relative", bottom: "15%", left: "5%", width: "50%", height: "10%" }}>
		<form style={{display: "flex"}} onSubmit={handleSubmit}>
		  <input type="text" name="message" onInput={handleChange} onChange={handleChange} />
		  <button type="submit">Send</button>
		</form>
	  </div>
	);
  }Code language: JavaScript (javascript)

In this example implementation of Magick in the 3OV plugin, the ChatBox function handles the submission of user input and sends it to a localhost endpoint through an HTTP post request. (In this case I have Magick running locally but you could install this on any server.) The request includes a few parameters like the input value, speaker and agent names, client and channel IDs, and entity and channel values. These parameters are passed to the endpoint as an object in the request body and is used to process and store information about the context of the conversation in Magick.

Upon submission, the function sends the message to a specified Magick spell handler and version, which are defined in the URL of the request. The spell handler and version are set to “four” and “latest” respectively in my example. You could have many spells and route chats to different spell handlers to dynamically make the requests more cost efficient or more technical depending on your needs.

		const spell_handler = "four";
		const spell_version = "latest";
		const url = encodeURI( `https://localhost:8001/spells/${spell_handler}/${spell_version}` )
		const response = await axios.post(`${url}`, {
			inputs: {
			  Input: value,
			  Speaker: speaker,
			  Agent: agent,
			  Client: client,
			  ChannelID: channelId,
			  Entity: entity,
			  Channel: channel,
			},
		  }).then((response) => {
			const data = response.data;

			const outputs = data.outputs;

			const outputKey = Object.keys(outputs)[0];

			const output = outputs[outputKey];

			props.setMessages([...props.messages, output]);
		  });
		} catch (error) {
		console.error(error);
	  }
Code language: JavaScript (javascript)

Upon receiving the request, Magick processes the input and other parameters using the specified spell, and returns a response in the form of an object containing output data. The ChatBox function retrieves the output data and adds it to the list of messages using setMessages. This state is passed to a lower component where the AI assistant avatar file is handled. The message populates above the avatars head and closes the loop of conversation!

You may notice in the screenshot above there’s json being output with a tone property. That is GPT-3 deciding the tone of the message it is sending! You could easily map this to VRM avatar facial expressions so the AI can naturally interact with you in-world. I plan to get more granular and give the spell control of the default VRM emotion parameters.

Voice recognition and responses are 100% possible today thanks to the work of Magick contributors. Overall, this implementation of Magick within the 3OV plugin demonstrates simply how the framework can be used to create interactive and personalized communication patterns with AI in a 3D environment. This can go way further and is only limited by imagination.

Pricing and Caution

It’s important to note that OpenAI queries could potentially be expensive to use, depending on how many shots to the AI you need to make in a spell. For example, if you are using a spell that costs around $0.02 per query and you spell has multiple queries, this could add up quickly if you are making frequent requests. It’s important to be mindful of this when using Magick and to consider whether it is cost-effective for your needs. Adjusting language models to cheaper options is another way to ease costs. In my testing I have short circuited the AI bits of a spell to return static responses to mock against as I develop ideas.

I would encourage using Magick as a local personal AI assistant or implementing safeguards in your Magick spells to recognize when someone is abusing the system. This could include defaulting to basic responses rather than querying GPT when abuse is detected. These are very early days for this project so keep your instances locked down as auth and other gated features work their way into the framework.

Conclusion:

Magick is a powerful tool for creating AI assistants and characters with complex communication patterns. Its ability to integrate with GPT-3 and other OpenAI language models allows users to leverage the advanced capabilities of these models to create sophisticated responses. The potential applications of Magick are numerous, and it could revolutionize the way we interact with AI in a wide range of industries, from gaming to customer service. I can’t say for certain how we will officially launch integrations for this in 3OV, but the important part is that it is now possible and we will find the right way to launch support through a NPC block. I’m excited to continue contributing to this framework as it takes shape!