I mentioned in the first post that there were three major tasks remaining to complete the proof-of-concept:
- Improve performance by overhauling the 3D UI layout engine.
- Finish porting the code to the Nvidia Jetson AGX Xavier, which I’m using as the computer unit of my portable computer for my proof-of-concept.
- Switch to hand tracking code running locally on the Jetson to fulfill the promise of a truly untethered XR solution.
The performance improvement was completed in April, as it was critical for the demo video I last posted. The remaining two goals still need to be implemented, but I’ve decided that it makes more sense to switch the hand tracking code first, so that’s what I’ve been working on recently.
Currently, the project uses the Leap Motion device to do hand tracking, with the actual hand tracking code running on a tethered second computer. This is because the better versions of the Leap Motion driver do not run on Linux and do not run on the same processor architecture that the Nvidia Jetson uses. Unfortunately, having to be tethered to a traditional PC doesn’t make for a good proof-of-concept that’s intended to be portable. So, I needed to find a hand tracking solution that would run on the same device as the rest of Kros.
Thankfully, I came across Google’s MediaPipe, a toolkit that includes a hand tracking solution, which offers several advantages:
- It is free for commercial use.
- It is open source, so it can be ported to any operating system and processor architecture including the Jetson’s.
- It can use any basic camera, so, instead of a special device, it can use the same camera used for video passthrough.
If you’re interested in MediaPipe, you can try out its single-hand tracking demo within your web browser by going to https://viz.mediapipe.dev/demo/hand_tracking and clicking the run button in the upper right corner to track your hand in front of your web camera.
Although the integration of MediaPipe’s build process into Kros’s build process has been more vexing than I initially anticipated, I have made progress on this task and hope to complete it by the end of the summer. At that time, I’ll provide more details on the switch and any changes to the hand tracking functionality.
Thanks for reading, and if you haven’t already, subscribe to the blog.
Kros Development Log 