Scene Semantics API

Scene Semantics is an ARCore feature that uses machine learning to classify each pixel in the camera feed into semantic categories like sky, buildings, roads, trees, and more. This enables powerful outdoor AR experiences that can understand and react to the real-world environment.

Overview

The Scene Semantics API provides:

Real-time semantic segmentation of the camera feed
12 semantic labels covering common outdoor scene elements
Per-frame label fractions showing what percentage of the view contains each element

Requirements

Platform Support

Platform	Requirements
Android	ARCore 1.31+ and a compatible device. Scene Semantics is included in the standard ARCore SDK.
iOS	ARCore SDK for iOS with Semantics extension. Requires API key with ARCore API enabled.

Device Requirements

Scene Semantics requires significant GPU/NPU resources
Not all ARCore-compatible devices support Scene Semantics
Always check isSemanticModeSupported() before enabling

Setup

Google Cloud Setup (Required for All Projects)

Before configuring your app, you need to set up Google Cloud:

Go to Google Cloud Console
Create a new project or select an existing one
Enable the ARCore API:
- Navigate to "APIs & Services" > "Library"
- Search for "ARCore API"
- Click "Enable"
Create an API key:
- Navigate to "APIs & Services" > "Credentials"
- Click "Create Credentials" > "API Key"
- (Optional) Restrict the key to your app's bundle ID/package name for security

Expo Projects

For Expo projects, the plugin handles all native configuration automatically. Scene Semantics is automatically included when you enable Cloud Anchors or Geospatial features.

1. Configure app.json or app.config.js:

{
  "expo": {
    "plugins": [
      [
        "@reactvision/react-viro",
        {
          "googleCloudApiKey": "YOUR_GOOGLE_CLOUD_API_KEY",
          "cloudAnchorProvider": "arcore",
          "android": {
            "xRMode": ["AR"]
          }
        }
      ]
    ]
  }
}

Note: Setting cloudAnchorProvider: "arcore" or geospatialAnchorProvider: "arcore" automatically includes Scene Semantics support.

2. Rebuild your app:

npx expo prebuild --clean
npx expo run:ios
# or
npx expo run:android

That's it! The Expo plugin automatically configures:

Platform	What the plugin adds
iOS	`GARAPIKey` in Info.plist, `use_frameworks! :linkage => :dynamic` in Podfile, `ARCore/CloudAnchors` and `ARCore/Semantics` pods
Android	`com.google.android.ar.API_KEY` meta-data in AndroidManifest.xml (Scene Semantics is included in standard ARCore SDK)

Bare React Native Projects

For bare React Native projects, you must configure native files manually.

iOS Setup

1. Modify your Podfile:

Add the following to your ios/Podfile:

# Enable dynamic frameworks (required for ARCore)
use_frameworks! :linkage => :dynamic

# Add ARCore pods
pod 'ARCore/CloudAnchors', '~> 1.51.0'
pod 'ARCore/Semantics', '~> 1.51.0'

Important: ARCore SDK requires use_frameworks! with dynamic linkage. This may affect other dependencies in your project.

2. Add API key to Info.plist:

Add the following to your ios/[YourApp]/Info.plist:

<key>GARAPIKey</key>
<string>YOUR_GOOGLE_CLOUD_API_KEY</string>

3. Install pods:

cd ios && pod install && cd ..

Android Setup

Scene Semantics is included in the standard ARCore SDK on Android, so no additional dependencies are needed. Just ensure you have the API key configured.

1. Add API key to AndroidManifest.xml:

Add the following inside the <application> tag in android/app/src/main/AndroidManifest.xml:

<meta-data
    android:name="com.google.android.ar.API_KEY"
    android:value="YOUR_GOOGLE_CLOUD_API_KEY" />

2. Rebuild your app:

npx react-native run-android

Semantic Labels

The API classifies pixels into 12 categories:

Label	Description
`unlabeled`	Pixels that couldn't be classified
`sky`	Sky, clouds, sun
`building`	Buildings, houses, structures
`tree`	Trees, bushes, vegetation
`road`	Paved roads, streets
`sidewalk`	Sidewalks, pedestrian paths
`terrain`	Grass, dirt, natural ground
`structure`	Fences, poles, signs, bridges
`object`	Cars (parked), benches, trash cans
`vehicle`	Moving vehicles
`person`	People, pedestrians
`water`	Rivers, lakes, pools, fountains

Quick Start

1. Check Support

Always verify that Scene Semantics is supported on the current device:

import { ViroARSceneNavigator } from "@reactvision/react-viro";

function MyARScene({ arSceneNavigator }) {
  const [semanticsSupported, setSemanticsSupported] = useState(false);

  useEffect(() => {
    async function checkSupport() {
      const { supported } = await arSceneNavigator.isSemanticModeSupported();
      setSemanticsSupported(supported);

      if (supported) {
        // Enable semantics
        arSceneNavigator.setSemanticModeEnabled(true);
      }
    }
    checkSupport();
  }, []);

  // ...
}

2. Enable Semantic Mode

// Enable semantic processing
arSceneNavigator.setSemanticModeEnabled(true);

// Disable when no longer needed (saves battery/resources)
arSceneNavigator.setSemanticModeEnabled(false);

3. Get Label Fractions

Retrieve the percentage of pixels for each semantic label:

// Get all label fractions at once
const { success, fractions } =
  await arSceneNavigator.getSemanticLabelFractions();

if (success && fractions) {
  console.log(`Sky: ${(fractions.sky * 100).toFixed(1)}%`);
  console.log(`Building: ${(fractions.building * 100).toFixed(1)}%`);
  console.log(`Road: ${(fractions.road * 100).toFixed(1)}%`);
  console.log(`Tree: ${(fractions.tree * 100).toFixed(1)}%`);
}

Or get a specific label:

// Get fraction for a specific label
const { success, fraction } = await arSceneNavigator.getSemanticLabelFraction(
  "sky"
);

if (success) {
  console.log(`Sky covers ${(fraction * 100).toFixed(1)}% of the view`);
}

API Reference

`isSemanticModeSupported()`

Check if Scene Semantics is supported on the current device.

const result: ViroSemanticSupportResult =
  await arSceneNavigator.isSemanticModeSupported();

Returns: Promise<ViroSemanticSupportResult>

supported: boolean - Whether semantics is supported
error?: string - Error message if check failed

`setSemanticModeEnabled(enabled: boolean)`

Enable or disable Scene Semantics processing.

arSceneNavigator.setSemanticModeEnabled(true); // Enable
arSceneNavigator.setSemanticModeEnabled(false); // Disable

Parameters:

enabled: boolean - Whether to enable semantic processing

Note: Enabling semantics uses additional GPU/battery resources. Disable when not needed.

`getSemanticLabelFractions()`

Get the fraction of pixels for all semantic labels in the current frame.

const result: ViroSemanticLabelFractionsResult =
  await arSceneNavigator.getSemanticLabelFractions();

Returns: Promise<ViroSemanticLabelFractionsResult>

success: boolean - Whether the operation succeeded
fractions?: ViroSemanticLabelFractions - Object with all label fractions
error?: string - Error message if failed

Example Response:

{
  success: true,
  fractions: {
    unlabeled: 0.02,
    sky: 0.35,
    building: 0.28,
    tree: 0.15,
    road: 0.12,
    sidewalk: 0.03,
    terrain: 0.02,
    structure: 0.01,
    object: 0.01,
    vehicle: 0.00,
    person: 0.01,
    water: 0.00
  }
}

`getSemanticLabelFraction(label: ViroSemanticLabel)`

Get the fraction for a specific semantic label.

const result: ViroSemanticLabelFractionResult =
  await arSceneNavigator.getSemanticLabelFraction("sky");

Parameters:

label: ViroSemanticLabel - The label to query ("sky", "building", "road", etc.)

Returns: Promise<ViroSemanticLabelFractionResult>

success: boolean - Whether the operation succeeded
fraction: number - Fraction of pixels (0.0 to 1.0)
error?: string - Error message if failed

Use Cases

Outdoor Detection

Determine if the user is outdoors by checking sky visibility:

async function isOutdoors(arSceneNavigator) {
  const { success, fraction } = await arSceneNavigator.getSemanticLabelFraction(
    "sky"
  );

  if (success) {
    // If more than 10% sky is visible, likely outdoors
    return fraction > 0.1;
  }
  return false;
}

Environment-Aware Content

Adapt AR content based on the environment:

async function getEnvironmentType(arSceneNavigator) {
  const { success, fractions } =
    await arSceneNavigator.getSemanticLabelFractions();

  if (!success || !fractions) return "unknown";

  // Urban environment: lots of buildings and roads
  if (fractions.building > 0.3 && fractions.road > 0.1) {
    return "urban";
  }

  // Natural environment: lots of trees and terrain
  if (fractions.tree > 0.3 || fractions.terrain > 0.2) {
    return "nature";
  }

  // Street scene: roads and sidewalks
  if (fractions.road > 0.2 && fractions.sidewalk > 0.05) {
    return "street";
  }

  return "mixed";
}

Safety Warnings

Warn users about traffic:

async function checkForTraffic(arSceneNavigator) {
  const { fraction } = await arSceneNavigator.getSemanticLabelFraction(
    "vehicle"
  );

  if (fraction > 0.05) {
    // Significant vehicle presence detected
    showWarning("Watch out for traffic!");
  }
}

Sky Portal Effects

Create effects that only appear in the sky:

async function updateSkyEffect(arSceneNavigator) {
  const { fraction } = await arSceneNavigator.getSemanticLabelFraction("sky");

  // Scale effect based on sky visibility
  setSkyEffectIntensity(fraction);
}

Best Practices

1. Check Support First

Always verify support before using semantics features:

const { supported } = await arSceneNavigator.isSemanticModeSupported();
if (!supported) {
  // Provide fallback experience or inform user
  showMessage("Scene understanding not available on this device");
  return;
}

2. Manage Resources

Enable semantics only when needed:

// Enable when entering a feature that needs it
function onEnterSemanticFeature() {
  arSceneNavigator.setSemanticModeEnabled(true);
}

// Disable when leaving
function onExitSemanticFeature() {
  arSceneNavigator.setSemanticModeEnabled(false);
}

3. Poll at Reasonable Intervals

Don't query every frame. Use reasonable intervals:

useEffect(() => {
  let interval;

  if (semanticsEnabled) {
    // Poll every 500ms instead of every frame
    interval = setInterval(async () => {
      const { fractions } = await arSceneNavigator.getSemanticLabelFractions();
      updateUI(fractions);
    }, 500);
  }

  return () => {
    if (interval) clearInterval(interval);
  };
}, [semanticsEnabled]);

4. Handle Errors Gracefully

Always check the success flag:

const { success, fractions, error } =
  await arSceneNavigator.getSemanticLabelFractions();

if (!success) {
  console.warn("Semantics unavailable:", error);
  // Use fallback logic
  return;
}

// Safe to use fractions
processSemantics(fractions);

TypeScript Types

// Semantic label type
type ViroSemanticLabel =
  | "unlabeled"
  | "sky"
  | "building"
  | "tree"
  | "road"
  | "sidewalk"
  | "terrain"
  | "structure"
  | "object"
  | "vehicle"
  | "person"
  | "water";

// All label fractions
type ViroSemanticLabelFractions = {
  unlabeled: number;
  sky: number;
  building: number;
  tree: number;
  road: number;
  sidewalk: number;
  terrain: number;
  structure: number;
  object: number;
  vehicle: number;
  person: number;
  water: number;
};

// API result types
type ViroSemanticSupportResult = {
  supported: boolean;
  error?: string;
};

type ViroSemanticLabelFractionsResult = {
  success: boolean;
  fractions?: ViroSemanticLabelFractions;
  error?: string;
};

type ViroSemanticLabelFractionResult = {
  success: boolean;
  fraction: number;
  error?: string;
};

Troubleshooting

"Semantics not supported"

Ensure you're running on a compatible device
Check that ARCore is up to date
Verify the app has camera permissions

Fractions are all zero

Make sure setSemanticModeEnabled(true) was called
Wait a few frames after enabling for ML processing to start
Ensure the camera has a clear view (not blocked/covered)
iOS: Verify the GARAPIKey is correctly set in Info.plist

iOS-specific issues

"GARAPIKey not found"

Add <key>GARAPIKey</key><string>YOUR_KEY</string> to Info.plist
Ensure the API key has ARCore API enabled in Google Cloud Console

"Failed to create GARSession"

Check that ARCore/Semantics pod is installed (pod install)
Verify the API key is valid and not restricted incorrectly

Semantics shows "supported" but fractions are always zero

Ensure you're outdoors - Scene Semantics is optimized for outdoor scenes
Wait 1-2 seconds after enabling for the ML model to initialize
Check Xcode console for any ARCore-related warnings

Performance issues

Disable semantics when not actively using it
Reduce polling frequency (500ms is recommended)
Consider using getSemanticLabelFraction() for single labels instead of all fractions
Scene Semantics uses ML processing which consumes battery - disable when not needed

Limitations

Outdoor optimized: Scene Semantics is trained on outdoor scenes. Indoor accuracy may vary.
Processing delay: There's a small delay between camera capture and semantic results.
Resolution: Semantic data is at a lower resolution than the camera feed.
Battery usage: Continuous ML processing consumes additional battery.
Device support: Not all ARCore-compatible devices support Scene Semantics.