Using Advanced VPS in Your App

This page takes a place from a recording to tracked content in your app: record, upload, wait for processing, then track the place with <ViroVPS> and pin content with <ViroVPSAnchor>. Set up your plan, keys and the Expo plugin first: see Set up Advanced VPS.

Everything here runs on iOS and Android. Every function and component is exported from @reactvision/react-viro.

1. Record the place

Recording uses the AR session recorder on the arSceneNavigator your scene receives. See AR Session Recording for the format.

const nav = props.arSceneNavigator;

// On Android this answers false until the AR session has started,
// so ask after the first tracking update.
const depth = await nav.isRecordingSupported({ depth: true });

await nav.startRecording(outputDir, { depth });
// ...walk the place...
const result = await nav.stopRecording();
// { success, error?, durationSec, frames, depthFrames, gpsFixes, bytes? }

outputDir is a plain filesystem path in your app's storage, not a file:// URI. It is created if it does not exist, and gets video.mp4 and session.jsonl, and with depth depth/NNNNNN.png (16-bit, millimetres) and depth/NNNNNN_c.png (8-bit confidence), at most one every 100 ms.
Depth is recorded only when the navigator was mounted with depthCaptureEnabled on a device that measures depth. Otherwise the recording is made without depth, and its header says "depth": false.
stopRecording() resolves once every file is complete. Check success before you upload.
A recording with more than about 54 minutes of depth cannot be uploaded (two files per frame, and the upload zip holds at most 65,535 entries). An upload may be at most 512 MiB.

Help whoever is recording

useViroTrackingReady(state, { normalMs, limitedMs }) says when tracking has settled enough to start: normal for 1 s, or limited for 4 s, by default. Pass the latest state from onTrackingUpdated. It returns { ready, getLimitedSeconds, resetLimitedSeconds }.
onTrackingUpdated on ViroARSceneNavigator gives you the current scene's tracking state without passing it up from the scene.
useViroCaptureGuidance({ enabled, sceneRef }) gives a live hint while recording: "fast" (turning too fast), "pitch" (pointing at the floor or ceiling), "close" (standing too close) or "blank" (pointing at nothing with detail). Pass its onCameraTransformUpdate to your ViroARScene. Hints are ids, so your app supplies the wording. VIRO_CAPTURE_RULES holds the default thresholds and rules overrides any of them. getState() returns the seconds spent on each hint and the count of tracking jumps.

2. Upload the scan

An app that uploads needs a user token: a client key cannot create scans. See Acting for a signed-in user for credentials.

import { uploadScanRecording } from "@reactvision/react-viro";

// A new location, created with its first scan:
const scan = await uploadScanRecording(
  { location: { name: "Front entrance", lat, lon } },
  outputDir,
  {
    credentials,
    meshQuality: "medium",
    outputs: ["mesh", "semantic"],
    onProgress: (sent, total) => setProgress(sent / total),
  }
);

// Another scan of a known location:
await uploadScanRecording({ locationId: scan.location_id }, outputDir2, { credentials });

uploadScanRecording(target, outputDir, options?) zips the recording natively and sends it straight to storage, so nothing passes through JavaScript. Call it after stopRecording() has resolved, with the same outputDir. The navigator has the same method: arSceneNavigator.uploadScanRecording(target, outputDir, options?).

target is { locationId } for a known location, or { location: { name, lat?, lon? } } to create one with this scan. lat and lon go together or not at all.

Option	Type	Default	Description
`outputs`	`("mesh" \| "splat" \| "semantic")[]`	`["mesh"]`	What to build besides the map, which is always built. `semantic` needs `mesh`.
`meshQuality`	`"low" \| "medium" \| "high"`	`"medium"`	`low` builds quickest and skips the HD mesh. `high` uses full-resolution depth, a denser HD mesh and twice the texture resolution, and takes about 1.75 times as long to process, which is billed.
`keepRecording`	`boolean`	`false`	ReactVision Spatial keeps the recording only while it processes the scan, unless this is `true`. Merging a location's scans needs it. A kept recording counts toward the team's storage. The recording's folder stays on the device either way.
`onProgress`	`(sentBytes, totalBytes) => void`		At most four times a second. If Android restarts the transfer, the count starts again from 0.
`scanId`	`string`		Resume into a scan still in `awaiting_upload`: the `scanId` of the `ViroVPSError` an earlier attempt threw.
`endpoint`, `serverEndpoint`, `credentials`, `appUserId`		See Setup	Where the upload goes and as whom (`ViroVPSConnection`).

What to expect:

It carries on in the background, in a background URLSession on iOS and a WorkManager job on Android, and finishes even if your app has gone by then.
It resolves to the scan, queued once ReactVision Spatial has the recording, or still awaiting_upload when the recording is stored but ReactVision Spatial could not be told. That scan is queued by itself within a minute.
It throws a ViroVPSError (with code and status) when the recording cannot be read, a request fails, or the server refuses. When the scan exists but its recording did not arrive, the error's scanId is set, so you can resume. A lost connection gives NETWORK_ERROR. Options or credentials that would be refused throw BAD_REQUEST, INVALID_APP_USER_ID or CREDENTIALS_UNAVAILABLE before anything is zipped. UPLOAD_IN_PROGRESS means the scan's recording is already being uploaded.
Server refusals you may see: 403 PAID_PLAN_REQUIRED, API_KEY_TYPE_NOT_ALLOWED (a client key), COMPUTE_SPEND_LIMIT_REACHED, COMPUTE_BILLING_NOT_SET_UP or COMPUTE_PAYMENT_FAILED; 429 VPS_USER_LIMIT_REACHED (10 scans a day per app user by default), VPS_USER_OPEN_SCANS_LIMIT (3 scans waiting) or VPS_QUEUE_FULL (50 of the team's scans waiting); 413 TOO_LARGE (over 512 MiB).
A user token that expires during a long upload makes the final call fail with 401 APP_USER_TOKEN_EXPIRED. The recording is stored by then, so the upload still answers the scan in awaiting_upload, and ReactVision Spatial queues it within a minute.
Your app's own credentials go only to safe addresses: https, or http on localhost, a private network or a .local host. A custom endpoint elsewhere, given without credentials, fails with ENDPOINT_NOT_ALLOWED.

3. Follow processing

Poll getScan(scanId, connection?) every few seconds while status is queued or processing:

import { getScan } from "@reactvision/react-viro";

let s = await getScan(scan.id, { credentials });
while (s.status === "queued" || s.status === "processing") {
  await new Promise((r) => setTimeout(r, 5000));
  s = await getScan(scan.id, { credentials });
}
if (s.status === "succeeded" && s.map_accepted) {
  // The location can now be tracked.
} else {
  // s.error explains a failed scan; s.map_rejected_reason a map that was not accepted.
}

status is awaiting_upload, queued, processing, succeeded or failed. A scan whose recording does not arrive within 24 hours fails. A failed scan is not billed.
map_accepted and map_rejected_reason say whether the map was accepted. Only an accepted map becomes the location's map, and only it is billed.
assets lists what was built: map, mesh, world_mesh, splat, poses, semantic.
quality holds the map's self-check, and quality.mesh_confidence is { level, reasons, ... } with level "good", "fair" or "poor". reasons explain anything short of good to whoever recorded. It is null when the scan built no mesh. ViroVPSScan, ViroVPSScanQuality and ViroVPSMeshConfidence are exported types.

4. Track the place

Put a ViroVPS and a ViroVPSAnchor in your ViroARScene, then call startTracking() once AR tracking is normal:

<ViroARScene onTrackingUpdated={onTrackingUpdated}>
  <ViroVPS
    ref={vpsRef}
    arSceneNavigator={props.arSceneNavigator}
    worldMesh
    onTrackingState={(e) => setState(e.state)}
  />
  <ViroVPSAnchor>
    <Viro3DObject source={require("./sign.glb")} type="GLB" position={[0, 1.5, -1]} />
  </ViroVPSAnchor>
</ViroARScene>

// Later, for example from onTrackingUpdated or a button:
await vpsRef.current?.startTracking(locationId); // or "auto" (GPS) or "visual" (camera image)

startTracking

startTracking(locationId | "auto" | "visual") resolves to { success, locationId?, error?, errorDetail? }, where locationId is the location actually tracked.

A location id loads that location's map.
"auto" reads the device's geospatial pose from arSceneNavigator.getCameraGeospatialPose() and tracks the nearest location with a map, within autoRadiusMeters (default 200). Geospatial tracking must be on and producing a pose (see Geospatial). Without a pose, or with no location nearby, it falls back to "visual" unless visualSearch is false.
"visual" sends camera images to the VPS server to find the project's location from what the camera sees. Point the camera at the scanned place, not a blank wall.

Calling it again replaces whatever was loaded before. stopTracking() unloads the map. A navigator holds one map, so only one ViroVPS may track at a time.

Device or server localisation

localization="device" (the default) downloads the map and matches on the phone. localization="server" sends a camera frame every serverFixIntervalMs (default 1000) to the VPS server until it converges, then every serverRefreshIntervalMs (default 10000). Server mode holds up much better across days and lighting, but needs a paid plan and a map built for server mode, and is billed at $0.60 a minute by default. A map without server mode ends tracking with NO_SERVER_MAP.

Occlusion and physics against the place

worldMesh also loads the location's scanned mesh (world_mesh), so virtual content hides behind real walls and can collide with them. true turns on both; { occlusion: true, physics: false } picks one. It needs no worldMeshEnabled and no depth sensor, appears when localisation converges and follows the map's corrections. onWorldMesh reports { locationId, success, missing?, error? }, where missing means the scan built no mesh. A world mesh failure never fails tracking.

Tracking states

onTrackingState receives { state, locationId?, error?, errorDetail? }.

State	Meaning
`idle`	No map requested, or `stopTracking()` has run.
`searching`	Resolving a location and loading its map.
`mapLoaded`	A map is loaded and matching runs, but the estimate has not converged yet.
`tracking`	Converged. Content in `ViroVPSAnchor` is placed.
`lost`	Was `tracking`, then stopped matching.
`error`	The map failed to load, no location was found, or tracking ended with a code.
`unsupported`	The navigator cannot do VPS, or not the mode asked for.

errorDetail is { code, status, message }. code is the server's own code when the failure is its answer (such as NO_ASSET for a location with no map yet, PAID_PLAN_REQUIRED or COMPUTE_SPEND_LIMIT_REACHED), HTTP_ERROR for an error answer without one, or one set on the device: NETWORK_ERROR, CREDENTIALS_UNAVAILABLE, ENDPOINT_NOT_ALLOWED, VPS_SERVER_NOT_CONFIGURED, INVALID_APP_USER_ID, UNSUPPORTED, NO_LOCATION, MAP_LOAD_FAILED, LOCALIZATION_UNAVAILABLE, SUPERSEDED or UNKNOWN. ViroVPSErrorCodes exports the device-side codes as constants.

Content at detected objects

For a scan built with semantic, vpsRef.current.getSemanticObjects() returns the objects found in the tracked location, in the map's frame: { id, label, confidence, position, bounds, observations }. Use position for children of the anchor.

ViroVPSAnchor stays hidden until localisation converges and hides again when tracking is lost. onAnchorStateChange gives { state: "tracking" } or { state: "hidden" }. See ViroVPS for every prop.

Acting for a signed-in user

To let your app's users make their own locations and scans, your backend mints a user token with its secret key (see Mint user tokens on your backend), and the app passes createUserTokenCredentials() as credentials. It keeps the token between requests, refreshes it before it expires, and replaces a refused token once:

import * as React from "react";
import { createUserTokenCredentials } from "@reactvision/react-viro";

const credentials = React.useMemo(
  () =>
    createUserTokenCredentials({
      projectId: PROJECT_ID,
      // Your backend's own route, which calls app-user-tokens with its secret key.
      fetchToken: async () => {
        const r = await fetch("https://api.example.com/rv-token", {
          headers: { Authorization: `Bearer ${await getSessionToken()}` },
        });
        return r.json(); // { token, expires_in, ... }
      },
    }),
  [userId]
);

Pass credentials to <ViroVPS>, uploadScanRecording(), getScan() and the functions below. Make one per signed-in user, or call credentials.clear() at sign-out. The user's locations and scans are theirs, and they also see the project's shared ones.

Managing locations and scans from the app

Each function takes an optional ViroVPSConnection ({ endpoint?, serverEndpoint?, credentials?, appUserId? }) as its last argument.

Function	API operation	What it does
`findNearbyLocations(lat, lon, radiusMeters?, limit?, connection?)`	`GET /vps/locations?lat=&lon=`	Locations near a point, nearest first (default 200 m, 20 results).
`findNearestMap(lat, lon, radiusMeters?, connection?)`	`GET /vps/locations?lat=&lon=`	The nearest location with a map.
`listLocations({ limit?, cursor? }?, connection?)`	`GET /vps/locations`	The locations the caller can see, newest first, a page at a time (`limit` 1 to 100, default 50).
`getLocation(locationId, connection?)`	`GET /vps/locations/{location_id}`	A location, with its active scan or `null`.
`createLocation({ name, lat?, lon? }, connection?)`	`POST /vps/locations`	A new location. Creating it with its first scan, through `uploadScanRecording`'s `target`, is usually simpler.
`renameLocation(locationId, name, connection?)`	`PATCH /vps/locations/{location_id}`	Rename a location (up to 150 characters).
`deleteLocation(locationId, connection?)`	`DELETE /vps/locations/{location_id}`	Delete a location with every scan of it.
`listScans(locationId, { limit?, cursor? }?, connection?)`	`GET /vps/locations/{location_id}/scans`	Every scan of a location, newest first (`limit` 1 to 200, default 50).
`getScan(scanId, connection?)`, `deleteScan(scanId, connection?)`	`GET`, `DELETE /vps/scans/{scan_id}`	One scan. Deleting a location's active scan makes its newest remaining accepted scan the active one, or leaves it with no map.
`getLocationFileDownload(locationId, kind, connection?)`	`GET /vps/locations/{location_id}/{kind}`	A signed link to one file of a location's active scan.
`getScanFileDownload(scanId, kind, connection?)`	`GET /vps/scans/{scan_id}/{kind}`	A signed link to one file of one scan.
`fetchAssetBytes(download)`		Fetches a signed link's bytes right away.
`getVPSServerAddress(connection?)`	`GET /vps/server`	The VPS server address, or `null`.

A client key may only call the location reads and file downloads (map, world_mesh, semantic), for shared locations. Signed links expire after 15 minutes, so fetch them right away. See Retrieve your scans with the API.

Full example: scan a place, then track it

One screen with two modes. In scan mode the user records the place and uploads it as a new location. In track mode the app finds the place again and shows a model in it. It uses a user token from your backend, because a client key cannot upload.

import * as React from "react";
import { Button, Text, View } from "react-native";
import {
  ViroARScene,
  ViroARSceneNavigator,
  ViroTrackingStateConstants,
  ViroVPS,
  ViroVPSAnchor,
  Viro3DObject,
  ViroVPSError,
  createUserTokenCredentials,
  getScan,
  uploadScanRecording,
  useViroCaptureGuidance,
} from "@reactvision/react-viro";

const PROJECT_ID = "YOUR_REACTVISION_PROJECT_ID";
// A plain path in your app's storage (not a file:// URI). startRecording creates it.
const SCAN_DIR = `${APP_DOCUMENTS_PATH}/scans/front-entrance`;

const HINTS = {
  fast: "Turn more slowly",
  pitch: "Point at the walls, not the floor or ceiling",
  close: "Step back a little",
  blank: "Point at something with detail",
};

function ScanScene(props: any) {
  const nav = props.arSceneNavigator;
  const { recording, onRecorded } = nav.viroAppProps;
  const sceneRef = React.useRef<ViroARScene>(null);
  const guidance = useViroCaptureGuidance({ enabled: recording, sceneRef });

  React.useEffect(() => {
    if (recording) {
      (async () => {
        const depth = await nav.isRecordingSupported({ depth: true });
        await nav.startRecording(SCAN_DIR, { depth });
      })();
      return () => {
        nav.stopRecording().then(onRecorded);
      };
    }
  }, [recording]);

  React.useEffect(() => nav.viroAppProps.onHint?.(guidance.hint), [guidance.hint]);

  return <ViroARScene ref={sceneRef} onCameraTransformUpdate={guidance.onCameraTransformUpdate} />;
}

function TrackScene(props: any) {
  const nav = props.arSceneNavigator;
  const { locationId, credentials, onState } = nav.viroAppProps;
  const vpsRef = React.useRef<ViroVPS>(null);
  const started = React.useRef(false);

  const onTrackingUpdated = (state: number) => {
    if (state === ViroTrackingStateConstants.TRACKING_NORMAL && !started.current) {
      started.current = true;
      vpsRef.current?.startTracking(locationId);
    }
  };

  return (
    <ViroARScene onTrackingUpdated={onTrackingUpdated}>
      <ViroVPS
        ref={vpsRef}
        arSceneNavigator={nav}
        credentials={credentials}
        worldMesh={{ occlusion: true }}
        onTrackingState={(e) => onState(e.state, e.errorDetail?.code)}
      />
      <ViroVPSAnchor>
        <Viro3DObject source={require("./sign.glb")} type="GLB" position={[0, 1.5, -1]} />
      </ViroVPSAnchor>
    </ViroARScene>
  );
}

export function PlaceScreen({ userId }: { userId: string }) {
  const [mode, setMode] = React.useState<"scan" | "track">("scan");
  const [recording, setRecording] = React.useState(false);
  const [hint, setHint] = React.useState<string | null>(null);
  const [status, setStatus] = React.useState("Ready to scan");
  const [locationId, setLocationId] = React.useState<string | null>(null);

  const credentials = React.useMemo(
    () =>
      createUserTokenCredentials({
        projectId: PROJECT_ID,
        fetchToken: async () => {
          const r = await fetch("https://api.example.com/rv-token", {
            headers: { Authorization: `Bearer ${await getSessionToken()}` },
          });
          return r.json();
        },
      }),
    [userId]
  );

  const onRecorded = async (result: { success: boolean; error?: string }) => {
    if (!result.success) return setStatus(`Recording failed: ${result.error}`);
    try {
      setStatus("Uploading");
      const scan = await uploadScanRecording(
        { location: { name: "Front entrance" } },
        SCAN_DIR,
        {
          credentials,
          outputs: ["mesh"],
          onProgress: (sent, total) => setStatus(`Uploading ${Math.round((100 * sent) / total)}%`),
        }
      );
      setStatus("Processing");
      let s = await getScan(scan.id, { credentials });
      while (s.status === "awaiting_upload" || s.status === "queued" || s.status === "processing") {
        await new Promise((r) => setTimeout(r, 5000));
        s = await getScan(scan.id, { credentials });
      }
      if (s.status === "succeeded" && s.map_accepted) {
        setLocationId(s.location_id);
        setStatus("Map accepted");
      } else {
        setStatus(`Not usable: ${s.map_rejected_reason ?? s.error ?? "unknown"}`);
      }
    } catch (e) {
      if (e instanceof ViroVPSError) setStatus(`Upload failed: ${e.code ?? e.message}`);
      else throw e;
    }
  };

  return (
    <View style={{ flex: 1 }}>
      <ViroARSceneNavigator
        key={mode}
        depthCaptureEnabled
        initialScene={{ scene: mode === "scan" ? ScanScene : TrackScene }}
        viroAppProps={{
          recording,
          onRecorded,
          onHint: setHint,
          locationId,
          credentials,
          onState: (state: string, code?: string) => setStatus(code ? `${state}: ${code}` : state),
        }}
      />
      <Text>{recording && hint ? HINTS[hint as keyof typeof HINTS] : status}</Text>
      {mode === "scan" ? (
        <>
          <Button title={recording ? "Stop" : "Record"} onPress={() => setRecording((r) => !r)} />
          <Button title="Track it" disabled={!locationId} onPress={() => setMode("track")} />
        </>
      ) : (
        <Button title="Back" onPress={() => setMode("scan")} />
      )}
    </View>
  );
}

APP_DOCUMENTS_PATH and getSessionToken() stand for your app's own storage path and sign-in. In a real app, keep each recording in its own folder, and let the user leave the screen while the upload carries on: it continues in the background, and you can look the scan up again later with getScan() or listScans().