is there any script to export pose from the matedata?

[wutingfan]

is there any script to export pose from the matedata? And, is the pose Tcw or Twc?
what is the data format in *.exr file? I read it by opencv-python, it is 3 channels, I'm not sure how to use it, or trans it to png/jpg.
Thank you very much.

[marek-simonik]

There is no example script that would show how to read the poses from the metadata file; it is an ordinary JSON file. Please see #27 (comment) and #33 (comment) for a description of how the poses are stored in the JSON file. The poses can be considered as T_{wc}.

Regarding EXR: see #53 (comment). You might have more luck trying to read the EXR files via OpenEXR (OpenCV struggles with it).

[wutingfan]

that's a great help, thank you.
The poses in metadata exported from face(camera type 0) are all [0 0 0 1 0 0 0], I test it on iphone11 and iphone13 pro max, they have same result, Is that normal?
The depth map from lidar is 192*256(the poses look normal here), I want use these depth maps to reconstruct the scene, is this a good idea to up sample the depth map? or what operation do you recommend? Thank you again.

[JulienGaubil]

Same issue as above, the poses obtained with a sequence recorded using the front camera of an iPhone13 mini are all [0 0 0 1 0 0 0]

[marek-simonik]

Sorry for the late reply, I haven't noticed that you wrote a new comment until today.

Only LiDAR videos have valid camera pose data. The selfie FaceID camera videos do not come with pose estimates, so seeing [0 0 0 1 0 0 0] is the expected behavior.

[wutingfan]

Sorry for the late reply, I haven't noticed that you wrote a new comment until today.

Only LiDAR videos have valid camera pose data. The selfie FaceID camera videos do not come with pose estimates, so seeing [0 0 0 1 0 0 0] is the expected behavior.

That's ok, I have sent you a serious app account problem by email for days, maybe you should check email acount([email protected] ). By the way, I will try to up sample depth map for fusion, is this a good idea?

[marek-simonik]

That's ok, I have sent you a serious app account problem by email for days, maybe you should check email acount([email protected] ).

I apologize for this, but after double-checking that email account, there are no unanswered emails. I have also checked the Spam folder, but I did not see your email. It seems like either your email did not reach my inbox or my reply did not reach yours.

Can you please re-send your email (and send it as a new email)? Most of the time, I reply to emails within 24 hours. Please resend your email and if I would not reply within 24 hours, send me a DM on Twitter @record3d. In case I would still not reply, please let me know here, in this Github issue and I will try to find another way how to connect with you. Thanks!

By the way, I will try to up sample depth map for fusion, is this a good idea?

I am not sure, I do not feel qualified enough to answer this question.

[marek-simonik]

@wutingfan I have just replied to your email.

[takeyamayuki]

@marek-simonik
Hello.
Thanks for the great project.

Only LiDAR videos have valid camera pose data. The selfie FaceID camera videos do not come with pose estimates, so seeing [0 0 0 1 0 0 0] is the expected behavior.

Is it possible to calculate the camera_pose from the iPhone IMU?
I would also like to get the proper camera_pose from the FaceID camera.

[marek-simonik]

Hello @takeyamayuki,

Is it possible to calculate the camera_pose from the iPhone IMU?
I would also like to get the proper camera_pose from the FaceID camera.

It would be possible to estimate device pose using IMU dead reckoning, but due to noise in IMU measurements, the estimated device pose would be unusable as it would drift.

[takeyamayuki]

@marek-simonik

It would be possible to estimate device pose using IMU dead reckoning, but due to noise in IMU measurements, the estimated device pose would be unusable as it would drift.

I see, but when acquiring a depth image, I really want to get the smartphone's orientation.
In my application, I need to tilt the smartphone, and the coordinate axis of the smartphone is tilted from the real world coordinate axis when the smartphone is tilted. This fact is not a problem if the angle can be known, but it is a problem if it cannot be known.

This is a problem not only for my application, but also for SLAM.
Therefore, I strongly request that record3D iOS apps have the ability to get the correct camera_pose even when using a FaceID camera.

If there is anything I can do to help, I would love to.

[marek-simonik]

@takeyamayuki I apologize for the delay in my reply.

Therefore, I strongly request that record3D iOS apps have the ability to get the correct camera_pose even when using a FaceID camera.

I will definitely add a better estimate of camera_pose in case Apple provides camera pose estimates for that camera type in the future. However, implementing camera_pose estimation for Record3D right now would take me quite a lot of time, which I am unfortunately unable to afford at this time.

[takeyamayuki]

@marek-simonik

CMMotionManager enables accurate acquisition of iphone posture in the form of euler angles, quaternions, and rotation matrices.
https://developer.apple.com/documentation/coremotion/cmmotionmanager

I would like to have a function to output the posture in the Record3D application using this.
If you don't have time, could you let me implement this function myself?

[takeyamayuki]

Currently, I've been able to acquire the smartphone's posture in the background using these two methods.

https://github.com/dachscafe/MotionSensorBackground
https://github.com/libimobiledevice/libimobiledevice/tree/master

[marek-simonik]

@takeyamayuki I apologize for the delay in my answer. Do I understand correctly that you would be interested in having stored the rough device's orientation estimate (provided by CMMotionManager) for each frame specifically for the FaceID camera?

[takeyamayuki]

@marek-simonik Yes