Implement gaussian splatting ply file saving with o3d

[MrNeRF]

Ply saving based on open3D. That's a quick way to generate ply files.

I think this is a highly requested feature. Should work with default and mcmc densification strategies.

[ichsan2895]

I think its better if we can load existing checkpoint, then save it to PLY:

    if cfg.ckpt is not None:
        # run eval only
        ckpts = [
            torch.load(file, map_location=runner.device, weights_only=True)
            for file in cfg.ckpt
        ]
        for k in runner.splats.keys():
            runner.splats[k].data = torch.cat([ckpt["splats"][k] for ckpt in ckpts])
        step = ckpts[0]["step"]
        runner.eval(step=step)
        runner.render_traj(step=step)
        save_ply(runner.splats, f"{cfg.result_dir}/point_cloud_{step}.ply")

[ichsan2895]

request:

save_ply(self.splats, f"{self.ply_dir}/point_cloud_{step}.ply")

[ichsan2895]

I try my own save_ply with plyfile version (Left) versus save_ply with this draft (Right). But, the color seems a little bit different between two method, despite I loaded it from exact same checkpoint.

Opened by mkkellogg gaussian splat viewer v0.4.4

[MrNeRF]

Interesting. The arm chairs are equally colored? 🙂

[ichsan2895]

Interesting. The arm chairs are equally colored? 🙂

seems the result is more washed out in open3D. The most noticeable is black chair (third image).
I will check for another scene later.

[ichsan2895]

simple_trainer_save_ply.zip

Here is it, the code comes from gsplat v1.3.0, then I modified it a little bit, so it saves PLY with plyfile library.

The command is easy

python3 gsplat/examples/simple_trainer_save_ply.py default --disable_viewer --data_factor 1 \
            --data_dir data/my_dataset \
            --result_dir output/my_dataset \
            --ckpt output/my_dataset/ckpts/ckpt_29999.pt

Then it will create splats.

[MrNeRF]

Should be fixed @ichsan2895.
I am also down to add a separate script so create a ply from the ckpt. It could be also done additionally. I don't mind. The maintainers should decide what they want if at all.

[ichsan2895]

Should be fixed @ichsan2895. I am also down to add a separate script so create a ply from the ckpt. It could be also done additionally. I don't mind. The maintainers should decide what they want if at all.

Well, creating PLY from checkpoints is just adding a single line of code.

if cfg.ckpt is not None:
        # load checkpoint
        ckpts = [
            torch.load(file, map_location=runner.device, weights_only=True)
            for file in cfg.ckpt
        ]
        for k in runner.splats.keys():
            runner.splats[k].data = torch.cat([ckpt["splats"][k] for ckpt in ckpts])
       .....................................................
       # add this line
       save_ply(runner.splats, f"{cfg.result_dir}/point_cloud_{step}.ply")

[MrNeRF]

It is not about adding a line of code. There is some reasoning behind why it writes the checkpoint in the way it does and I am not aware of it. The ply file should be directly saved during training and not evaluation. There could be a script added that converts old checkpoints to ply files if desired.

Beside saving the trained scene in an alternative format, the ckpts are pretty useless. They don't serialize the full training state (optimzer, scheduler) so it is not possible to resume the training properly. In fact, the ply file contains the same information.

Additional input from the maintainers is needed.

[bolopenguin]

Would this method work when appearance optimization is enabled? Since the sh0 and shN are not available.

[MrNeRF]

@bolopenguin Good catch. I bake the appearance into the shs so it can be opened by any viewer.

@liruilong940607 What does this pr require to go into gsplat?

[brentyi]

Not arguing for it to block this PR, but as an FYI this will break things for folks running Python 3.12, which the latest open3d release doesn't support. (afaik this is the main thing that's preventing Python 3.12 support in the main nerfstudio repo)

It looks like the open3d folks merged their Python 3.12 PR a few days ago though, so maybe this will resolved soon? isl-org/Open3D#6717

[MrNeRF]

Ah, interesting!

We could either wait until open3D is compatible with Python 3.12 or we write the ply file directly similarly as it is done in Nerfstudio. Any preferences?

[MrNeRF]

I removed open3D in commit 36e3df3 in case that's blocking.

[lambdald]

Is there some difference in implementation from the 3DGS official version? I tried to save the training result as a ply file and then view it in the web viewer SuperSplat. I observed some color artifact, but it is fine in the gsplat viewer.

SuperSplat:

gsplat viewer[gsplat/examples/simple_viewer.py]:

[MrNeRF]

I am not sure what supersplat is doing behind the curtains. I can compare the output of the last two commits. They should be identical and were tested.

[hecodeit]

Encountered a strange issue. Everything working when training on previous datasets. However, when training on a new dataset and when save it as a ply file, an error is displayed.

Saving ply to ./outputs/10-15/ply/point_cloud_29999.ply
RPly: Failed writing scale_0 of vertex 62134 (binary output: float)
RPly: Failed writing rot_2 of vertex 62134 (binary output: float)
RPly: Failed writing opacity of vertex 62134 (binary output: float)
RPly: Failed writing ny of vertex 62153 (binary output: float)
RPly: Failed writing scale_0 of vertex 62153 (binary output: float)
RPly: Failed writing scale_1 of vertex 62153 (binary output: float)
RPly: Failed writing y of vertex 62177 (binary output: float)
RPly: Failed writing ny of vertex 62177 (binary output: float)
RPly: Failed writing rot_1 of vertex 62177 (binary output: float)
RPly: Failed writing rot_0 of vertex 62348 (binary output: float)
RPly: Failed writing y of vertex 62349 (binary output: float)
RPly: Failed writing nz of vertex 62349 (binary output: float)
RPly: Failed writing f_dc_1 of vertex 63240 (binary output: float)
RPly: Failed writing rot_0 of vertex 63240 (binary output: float)
RPly: Failed writing z of vertex 63241 (binary output: float)
RPly: Failed writing f_dc_2 of vertex 63476 (binary output: float)
RPly: Failed writing f_dc_1 of vertex 63476 (binary output: float)
RPly: Failed writing scale_2 of vertex 63476 (binary output: float)
RPly: Failed writing rot_3 of vertex 303342 (binary output: float)
RPly: Failed writing f_dc_2 of vertex 303342 (binary output: float)
RPly: Failed writing f_dc_0 of vertex 303342 (binary output: float)
RPly: Failed writing nx of vertex 321440 (binary output: float)
RPly: Failed writing rot_3 of vertex 321440 (binary output: float)
RPly: Failed writing rot_2 of vertex 321440 (binary output: float)
RPly: Failed writing x of vertex 380727 (binary output: float)
RPly: Failed writing nx of vertex 380727 (binary output: float)
RPly: Failed writing scale_0 of vertex 380727 (binary output: float)
RPly: Failed writing opacity of vertex 412845 (binary output: float)
RPly: Failed writing x of vertex 412846 (binary output: float)
RPly: Failed writing ny of vertex 412846 (binary output: float)
RPly: Failed writing scale_1 of vertex 566691 (binary output: float)
RPly: Failed writing opacity of vertex 566691 (binary output: float)
RPly: Failed writing y of vertex 566692 (binary output: float)
RPly: Failed writing rot_1 of vertex 570065 (binary output: float)
RPly: Failed writing scale_1 of vertex 570065 (binary output: float)
RPly: Failed writing rot_0 of vertex 570065 (binary output: float)
RPly: Failed writing nz of vertex 590964 (binary output: float)
RPly: Failed writing rot_1 of vertex 590964 (binary output: float)
RPly: Failed writing f_dc_1 of vertex 590964 (binary output: float)
RPly: Failed writing z of vertex 836721 (binary output: float)
RPly: Failed writing nz of vertex 836721 (binary output: float)
RPly: Failed writing f_dc_2 of vertex 836721 (binary output: float)

Training cli:

python gsplat/examples/simple_trainer.py mcmc --data_factor 2 \
        --opacity-reg 0.001 \
        --sh-degree 0 \
        --use_bilateral_grid \
        --data_dir data/10-15  \
        --result_dir  ./outputs/10-15

But the viewer working fine with:

python gsplat/examples/simple_viewer.py --ckpt=outputs/10-15/ckpts/ckpt_29999_rank0.pt

The problem only happen at ply. Files of ply generated, but open with Postshot show “PlyReader: unexpected table size”. MeshLab: "Error encountered while loading file. Unexpected EOF".

[MrNeRF]

@hecodeit
I didn't filter for NaNs and Inf which might cause your issue. Please check again and confirm.

Moreover, I found that there was a slight misalignment in the order the attributes are saved which might have caused the slight color issues that @lambdald reports. Please test as well.

[hecodeit]

@MrNeRF Yes, pull the commit and the error don't show again. Thanks for the updating.

[ichsan2895]

Wow, one month later of inactive PR of gs-scaffold. But I am curious how to export Gs-scaffold to PLY? Is it compatible with supersplat or mkkellogg gaussian-viewer?

[MrNeRF]

Is it compatible with supersplat or mkkellogg gaussian-viewer?

No, it is not that straight forward. It would be best to support it directly.

I think currently is busy season in terms of getting the publications camera ready... likely there is not much time left for open source projects.