Web Based Virtual Sunglasses Try-on Demo

This project aims to create Real-time Web based Virtual Sunglasses Try-on experiences.

It is built with React, using MediaPipe, ThreeJS - react-three/fiber and react-webcam.
This project was bootstrapped with Create React App.

Demo

Sunglasses Try-on Demo

Try on Desktop and Mobile browsers to check the performance differences.

Under the Hood

This demo app currently captures a video stream via react-webcam, feeds the stream into MediaPipe FaceMesh which tracks 468 3D face landmarks in real-time, which then provides real-time locations to ThreeJS - react-three/fiber render the 3D object accordingly.

Limitations

• Unlike MediaPipe Hands, MediaPipe FaceMesh performs well (+20fps) both desktop and mobile browsers. Currently, no limitations have been discovered. Check Performance Characteristics

Work in Progress (WIP)

• The 3D model currently currently tracks the landmark location between the eyes, but 3D object rotations and scaling are WIP.
• GLTFLoader has a problem loading 3D objects from the public folder. We believe it is related to headers. It is currently WIP.

Available Scripts

In the project directory, you can run:

npm start

Runs the app in the development mode.
Open http://localhost:3000 to view it in the browser.

The page will reload if you make edits.
You will also see any lint errors in the console.