How Spatial Computing Powers AI Assistants

Introduction: Why spatial computing matters for today’s AI assistants

How spatial computing integrates with AI assistants

At the core of this integration lies a three-step process:

• Spatial capture:Gathering point clouds, depth-sensor images, or GPS coordinates.
• Semantic interface:Transforming raw spatial data into LLM-readable descriptors (e.g., "at a distance of 2 meters, there is a wooden table").
• Contextual generation:Using a prompt that includes spatial embeddings to produce responses referencing location, objects, or actions.

Platforms like Voxel51 3.0 and NVIDIA Structured Data Models have made this process more accessible than ever. Voxel51 3.0 unifies search, annotation, and multimodal evaluation of sensor data for physical AI teams, while Structured Data Models offers pre-built base models for tables and structured data.

Practical example: An AI assistant for warehouse logistics

Imagine a warehouse worker wearing an AR headset and asking, "Where have the last 50 pallets been placed?" The AI assistant must combine:

• Real-time positioning data (GPS, inertial sensors)
• A 3D model of the shelf
• Historical movement records (structured tables)

Pseudocode workflow:

# 1. Capture point cloud (e.g., from a LiDAR sensor)
point_cloud = get_point_cloud_from_lidar()

# 2. Extract spatial embeddings using a pre-trained 3D model
spatial_embedding = encode_3d(point_cloud, model='nv-embed-v1')

# 3. Prompt an LLM with spatial embeddings
prompt = f"""
The following embedding represents the current geometry of the warehouse:
{spatial_embedding}

Provide a concise, single-sentence answer for the warehouse worker:
'Where have the last 50 pallets been placed?'
"""
response = query_llm(prompt)
print(response)

The result is a location-specific answer like: "The pallets have been placed on shelf level 3, aisle B, positions 12-61."

Concrete prompt examples for spatial computing-based AI assistants

When crafting a prompt for a spatial AI assistant, always include these three components:

• Spatial context
• Object reference
• Desired action

Prompt 1 (Augmented reality navigation)

Prompt:
"""
You are an AI assistant for an augmented reality wearable device.
User's location: lat 41.9, lon 12.5, orientation 45° (north-east).
Field of view: a bar counter, a human barista in a blue shirt, a beverage dispenser.

Ask the user: 'Where can I find ice water?'
Provide a concise, distance-based direction in meters.
"""

Prompt 2 (Robotic maintenance with Kyutai REKI)

Prompt:
"""
REKI, a 2.4B VLA model, observes a robotic arm attempting to grasp a loose cable.
Cable geometry (point cloud): an irregular line 0.8 m long, diameter 0.02 m.
Goal: recommend a sequence of gripper movements for a secure grasp.
Return a prompt for an LLM that coordinates robot control.
"""

Key takeaways

• Spatial computing adds 3D coordinates and context to traditional text-based LLMs.
• Tools like Voxel51 3.0 and NVIDIA Structured Data Models simplify embedding engineering.
• Prompts combining spatial embeddings with goal-based questions yield more reliable results.
• The fusion of VLA models (like Kyutai REKI) and text-to-action AI assistants is unlocking new possibilities in collaborative robotics.

How to get started today

Conclusion