Semantic Masking

1. Semantic Masking

Platforms: Android (ARCore) + iOS (ARKit)

Semantic masking lets a material show or hide itself based on what the AR session classifies each real-world pixel as. A per-frame 256×144 segmentation image from ARCore/ARKit is used to discard fragments at the GPU level - no CPU readback, no extra render pass.

Supported labels

"sky" "building" "tree" "road" "sidewalk" "terrain" "structure" "object" "vehicle" "person" "water"

semanticMask material prop

TypeScript

ViroMaterials.createMaterials({
  // Render only on pixels classified as sky
  skyOnly: {
    diffuseColor: "#ffffff",
    semanticMask: {
      mode: "showOnly",
      labels: ["sky"],
    },
  },

  // Hide model wherever a person is detected
  hidePeople: {
    semanticMask: {
      mode: "hide",
      labels: ["person"],
    },
  },
});

mode	Effect
"showOnly"	Fragment rendered only where pixel matches a label
"hide"	Fragment discarded where pixel matches a label
"debug"	Color-codes pixels: blue = unlabeled, teal→orange = classified

Applying to a model

Use shaderOverrides (not materials) on Viro3DObject. GLB models have no geometry on the root node — materials is not recursive and has no visible effect. shaderOverrides is always recursive and reaches every mesh in the hierarchy.

tsx

<Viro3DObject
  source={require("./model.glb")}
  type="GLB"
  shaderOverrides={["skyOnly"]}
/>

Colored camera overlay

To highlight pixels of a given label over the camera feed, add a full-screen ViroQuad with a semi-transparent material and semanticMask. Use your own state to toggle it:

tsx

ViroMaterials.createMaterials({
  skyHighlight: {
    lightingModel: "Constant",
    diffuseColor: "#4499FF88",
    writesToDepthBuffer: false,
    readsFromDepthBuffer: false,
    semanticMask: { mode: "showOnly", labels: ["sky"] },
  },
});

// In your scene component:
const [showOverlay, setShowOverlay] = useState(false);

{showOverlay && (
  <ViroQuad position={[0, 0, -5]} scale={[20, 40, 1]} materials={["skyHighlight"]} />
)}

Scene navigator props

tsx

<ViroARSceneNavigator
  // Show native camera-background semantic color debug overlay
  semanticDebugEnabled={false}

  // Android only — pixels with ARCore confidence below this value are treated
  // as unlabeled. Reduces boundary noise / blinking. Range: 0.0–1.0.
  semanticConfidenceThreshold={0.3}

  initialScene={{ scene: MyScene }}
/>

semanticDebugEnabled controls the native background overlay and is independent from any ViroQuad in the scene.

Enabling semantics at runtime

tsx

const arNavRef = useRef<any>(null);

useEffect(() => {
  const nav = arNavRef.current?.arSceneNavigator;
  nav?.isSemanticModeSupported().then((result: any) => {
    if (result?.supported) {
      nav.setSemanticModeEnabled(true);
    }
  });
  return () => {
    arNavRef.current?.arSceneNavigator?.setSemanticModeEnabled(false);
  };
}, []);

iOS setup (Expo)

Semantic masking on iOS requires the ARCore Semantics pod. Add includeSemantics: true to your Expo plugin config in app.json:

JSON

["@reactvision/react-viro", {
  "ios": {
    "includeSemantics": true
  }
}]

If you are already using provider: "arcore" or includeARCore: true, the pod is already included — no extra config needed.