UniPart brings open-vocabulary segmentation to 3D parts

UniPart treats 3D part segmentation as a zero-shot retrieval problem, allowing a robot to identify sub-components like a chair leg or jar lid from a point cloud using natural language. Developed for robotic manipulation, this model replaces the rigid, taxonomy-based segmentation systems that fail when encountering novel objects. Because it does not rely on a fixed list of classes, a developer can prompt the model for a 'handle' or a 'spout' on the fly, provided those concepts exist within the model's text encoder.

UniPart functions by projecting 3D point cloud features into the same latent space as CLIP text embeddings. The architecture uses a point-wise feature encoder to process the geometric structure of the scene, producing a dense feature vector for every point in the cloud. By calculating the cosine similarity between these per-point geometric vectors and the text embedding provided by CLIP, the model generates a probability map for each point. High similarity indicates that a specific point belongs to the part described by the language prompt, while low similarity rejects it. This projection allows the system to compute segmentation masks without having been trained on a fixed list of labels.

Data Synthesis and Multi-View Alignment

For developers, the mechanism is a shift from supervised classification to cross-modal alignment. The research team generated LangPart-1M, a synthetic dataset containing over 160,000 assets and 8 million text-to-part pairs, to provide the necessary supervision. To build this, the researchers rendered 3D objects from various camera viewpoints and used pre-existing 3D ground truth labels to assign categories to individual points in the resulting point clouds. By aggregating these labels across different viewing angles, the system reconciles point-level identity through a voting process that preserves geometric consistency, ensuring that a point identified as a 'handle' from the front remains consistent when viewed from the side.

Integration with Motion Planning

Consider a robotic arm tasked with picking up a specific section of a power drill. With a traditional system, the developer would need a pre-trained model capable of recognizing the specific drill chassis and the trigger. If the model encountered a new drill design, it would fail. With UniPart, the robot receives the point cloud of the drill, and the control system feeds the phrase 'trigger' into the model. The model outputs a probability map over the points that the motion planner uses to isolate the target region. Instead of receiving a fixed geometric primitive—like a pre-calculated bounding box for a 'drill'—the planner receives a continuous probability distribution from the model that defines the spatial extent of the target, which the robot then uses to compute grasp poses.

This architecture relies on the CLIP feature space to bridge the gap between language and geometry, meaning the model identifies parts by their descriptive attributes rather than category names. This allows for nuanced interaction, such as distinguishing a specific sub-component like a 'door handle' from a 'door hinge'. While the model demonstrates success in language-conditioned part grasping, one limitation is that the model's performance relies on the density and quality of the point cloud input; regions with high occlusion where point data is missing can lead to gaps in the probability map. Researchers still need to determine how robust this alignment is when moving from these high-fidelity synthetic training sets to the sparse, noisy point clouds captured by real-world sensors in uncontrolled environments.