Basic Concepts

We give a high-level overview of the main concepts in fVDB. Namely, how fVDB encodes sparse voxel grids with attributes, as well as how fVDB efficiently manages batches with non-uniform numbers of elements.

GridBatch: Voxel Grids with Attributes

At its core, fVDB provides means to efficiently encode mini-batches of sparse voxel grids where each voxel contains arbitrary vector or scalar attributes encoded as torch tensors. In practice, such an encoding is implemented via the GridBatch class.

A GridBatch is an indexing structure which maps 3D ijk coordinates to integer offsets which can be used to look up attributes in a tensor. The figure below illustrates this process for a GridBatch containing a single grid.

In the figure, the GridBatch acts as an acceleration structure which encodes the sparsity pattern of the grid (also known as the topology) and can translate grid coordinates into offsets into a data tensor.

By separating the grid topology and data tensors, the same grid can be used to operate on many different attributes without rebuilding.

Every operation in fVDB is built upon this kind of query (e.g. Sparse Convolution uses this query to look up features in the neighborhood of each voxel).

JaggedTensor and Batching

Each grid in a GridBatch can have a different number of voxels (e.g. in the mini batch of four cars above, each car has a different number of voxels). This means that unlike the dense case, fVDB needs to handle parallel operations over jagged batches. I.e. batches containing different numbers of elements.

To handle jagged batches, fVDB provides a JaggedTensor class. Conceptually, a JaggedTensor is a list of tensors with shapes \([N_0, *], [N_1, *], \ldots, [N_{B-1}, *]\) where \(B\) is the number of elements in the batch, \(N_i\) is the number of elements in the \(i^\text{th}\) batch item and \(*\) is an arbitrary numer of additional dimensions that all match between the tensors. The figure below illustrates such a list of tensors pictorially.

In practice, JaggedTensors are represented in memory by concatenating each tensor in the list into a single jdata (for Jagged Data) tensor of shape \([N_0 + N_1 + \ldots + N_{B-1}, *]\). Additionally, each JaggedTensor stores an additional jidx tensor (for Jagged Indexes) of shape \([N_0 + N_1 + \ldots + N_{B-1}]\) containing one int per element in the jagged tensor. jidx[i] is the batch index of the \(i^\text{th}\) element of jdata. Finally, a JaggedTensor contains a joffsets tensor (for Jagged Offsets) of shape \([B, 2]\) which indicates the start and end positions of the \(i^\text{th}\) tensor in the batch.

Similarly, each GridBatch also has jidx and joffsets corresponding to the batch index of each voxel in the grid, and the start and end offsets of each voxel index in a batch.

A simple example

To illustrate the use of GridBatchand JaggedTensor, consider a simple example where we build a grid from a point cloud, splat some values onto the voxels of that grid, and then sample them again using a different set of points.

First, we construct a minibatch of grids using the input points. These input points have corresponding color attributes.

import fvdb
from fvdb.utils.examples import load_car_1_mesh, load_car_2_mesh
import torch
import point_cloud_utils as pcu

# We're going to create a minibatch of two point clouds each of which
# has a different number of points
pts1, clrs1 = load_car_1_mesh(mode = "vn")
pts2, clrs2 = load_car_2_mesh(mode = "vn")

# Creating JaggedTensors: one for points and one for colors
points = fvdb.JaggedTensor([pts1, pts2])
colors = fvdb.JaggedTensor([clrs1, clrs2])

# Create a grid where the voxels each have unit sidelength
grid = fvdb.gridbatch_from_points(points, voxel_sizes=1.0)

# Indexing into a JaggedTensor returns a JaggedTensor
print(points[0].jdata.shape)
print(points[1].jdata.shape)

Next, we splat the colors at the points to the constructed grid, yielding per-voxel colors.

# Splat the colors into the grid with trilinear interpolation
# vox_colors is a JaggedTensor of per-voxel normas
vox_colors = grid.splat_trilinear(points, colors)

Finally, we generate a new set of noisy points and sample the grid to recover colors at those new samples.

# Now let's generate some random points and sample the grid at those points
sample_points = fvdb.JaggedTensor([torch.rand(10_000, 3), torch.rand(11_000, 3)]).cuda()

# sampled_colors is a JaggedTensor with the same shape as sample_points with
# one color sampled from the grid at each point
sampled_colors = grid.sample_trilinear(sample_points, vox_colors)