Point tracking: few occluded points are not labeled as occluded

[ZHAOZHIHAO]

Hi,

First, thanks for making the great tool for point tracking.

I see that almost all occluded points are correctly labeled as occluded, but visually I still obverse few point among 1200=50 (videos) x 24 (points per video) are occluded. I'm sure these points are occluded because the foreground object has moved a large distance and the ground truth still stays the original position, like a position of the ground. I looked into the code, but still have not idea what caused this problem. Could you please give some suggestions or even solution? Thanks.

Best,
Zhihao

[ZHAOZHIHAO]

The following strategy seems removing bad ground-truth where occluded points are not labeled as occluded, though it will not preserve any occluded point anymore. For a 24-frame video, if a point is occluded in any frame, I mark it as occluded in the whole 24 frames.

Best,
Zhihao

[cdoersch]

I'm sure these points are occluded because the foreground object has moved a large distance and the ground truth still stays the original position

I don't understand this statement. If the point is on a foreground object, and that object has moved, then the ground truth should not stay in its original position. This seems like a problem with the positions, not the occlusions.

Can you provide an image with some examples of occlusion errors? I'm unable to find any, although there is a known issue where points near occlusion boundaries can 'flicker' due to aliasing in the depth map.

[ZHAOZHIHAO]

Hi,

Thanks for your response. Here I attach 8 frames, where the point at (177, 117) in the first frame is such an example.

Best

[cdoersch]

This is most likely due to the fact that we use the depth map to estimate occlusion. The model currently just tests if the (interpolated) depth map is in front of the 3D point location by more than 1% of the distance to the camera. The test is performed here:

https://github.com/google-research/kubric/blob/main/challenges/point_tracking/dataset.py#L273

This test will produce the wrong result if the object is very thin--i.e., less than 1% of the distance to the camera. Unfortunately there's no way to set this threshold without producing a small number of false positives or false negatives, because we can't query the scene for the depth at an arbitary point. Therefore, the depth that we use to estimate occlusion will always have some error.

However, the recent bugfix #284 might mean that this threshold can be tightened; if you want to play around with it, feel free to do so and post the results. Furthermore, we could almost certainly reduce this kind of error by testing whether the segmentation map matches the original object id for the query point.

[ZHAOZHIHAO]

Thanks for your explanation and solution.

"reduce this kind of error by testing whether the segmentation map matches the original object id for the query point" should remove these wrong labels.

[cdoersch]

Going to reopen this because I may improve this myself in the near future. Contributions are also welcome.

[cdoersch]

@ZHAOZHIHAO I created a PR (#285 ) which should fix these issues (and also fixes a bug with the prior PR). If you have a chance, could you test the updated version and see if there's still any problematic cases?

Note that it's actually two separate improvements. Testing the segmentation map will work if it's e.g. a background point being occluded by a thin object, but it will still allow false negatives if it's a thin object being turned around in 3D (i.e. a self-occlusion). However, you can fix this by tracking the point's surface normal: if it's pointing away from the camera, then the point is guaranteed to be occluded. Turns out it's a bit tricky to actually do this because there's seemingly some weirdness with the kubric normal maps, but at least for flat objects it seems decently reliable.

[ZHAOZHIHAO]

Hi, thanks for the improvements. I'll test it in a short future.

Thanks.

[cdoersch]

I merged #285, as I think #284 actually made the occlusion estimation situation worse, and I don't want people training on a dataset with those issues. I'm still interested in getting your feedback on the latest version though.

[ZHAOZHIHAO]

I checked the new points. I don't find any point with wrong occlusion flag. Thanks for your new codes.