How Spatial Computing Powers AI Assistants: Practical Examples

Introduction: Why spatial computing matters for AI assistants today

This integration unlocks tangible opportunities: from more intuitive navigation on wearable devices to solving safety challenges in autonomous vehicles, and automating business tasks in three-dimensional environments. Below, you’ll learn how to create these scenarios using cutting-edge prompt engineering techniques and open-source tools like Perplexity’s Lily.

What makes a spatial AI assistant unique

A spatial AI assistant combines three key technological pillars:

• Spatial perception:Depth cameras, LiDAR, and inertial sensors that map the environment.
• Contextual processing:LLM models optimized to understand 3D references and object relationships.
• Multimodal action:

When these components come together, the assistant can answer questions like, “What’s the temperature of that device, and how can I insulate it?” directly in the user’s AR interface.

The role of prompt engineering in spatial contexts

Traditional prompts often fail when the context is three-dimensional. An effective spatial prompt must:

• Specify exact coordinates or object tags.
• Define the desired output format (AR overlay, 3D label, text instruction).
• Include safety and privacy constraints.

Example promptfor an assistant working on an industrial device:

Analyze the incoming video feed from the control panel camera. Identify the overheating component (coordinates: 0.2, 0.7, 0.9) and display an AR overlay with the exact temperature and a safe intervention recommendation.

Building a spatial AI assistant with Perplexity’s Lily

In September 2026, Perplexity open-sourcedLily

To integrate Lily into a spatial assistant:

1. Clone the repository and configure the Metal plugin for GPU acceleration.
2. Load a compact LLM model (e.g., Qwen3.6-35B-A3B) and a spatial object dataset.
3. Implement a prompt wrapper that converts AR/VR commands into text requests for the model.

Here’s a code snippet demonstrating the workflow:

// Initialize the local inference engine
let engine = new LilyEngine({
  modelPath: 'qwen3.6-35b-a3b.gguf',
  device: 'metal',
  threads: navigator.hardwareConcurrency
});

async function spatialAssistant(prompt, coordinates) {
  const context = JSON.stringify({
    modality: 'spatial',
    coordinates,
    instruction: prompt
  });
  const response = await engine.infer(context);
  return response;
}

This example shows how a single inference component can be called from an AR user interface to provide real-time answers without sending sensitive data to the cloud.

Real-world use cases in 2026

Explainable autonomous vehicles

Motional and MIT have introduced a system that allows cars to explain their decision-making processes in real time. Using rule-based prompts and spatial reasoning, the assistant can say, “I braked because the pedestrian crossed the safety line at position (x=12, y=4).” This transparency builds user trust and aids investigators.

Corporate AI copilots for 3D workspaces

M&T Bank has deployed AI assistants across more than 15,000 workstations. Employees can now ask, “Show me the sales report for the (3D) region and suggest an action plan,” and receive an interactive AR visualization and an AI-generated strategy draft.

Both cases rely on the same principle: a well-crafted LLM, powered by a local inference engine, delivers understandable information in spatial contexts.

Best practices and recommended tools

• Use local inference engineslike Lily to reduce latency and protect data.
• Design multimodal promptsthat include coordinates, output formats, and safety constraints.
• Leverage AR/VR APIs(WebXR, RealityOS) to transform text responses into 3D annotations.
• Implement a feedback loop: Record spatial interactions to refine models and improve prompt quality.

Tools likeSpatialPrompt Studio(a visual editor released this year) help build and test spatial prompts without writing code.

Conclusion: The future of AI assistance is three-dimensional

Whether you’re developing a copilot for a bank, an autonomous driving assistance system, or an AR tutorial for a field technician, the techniques described here provide a solid foundation for building, testing, and deploying successful spatial AI assistants.