A developer discovered a intelligent solution to entry Quest 3’s passthrough cameras to run customized laptop imaginative and prescient fashions.
Meta doesn’t enable third-party builders to entry the cameras of Quest headsets, however Michael Gschwandtner, the co-founder and lead developer of the hand tracking fitness app XRWorkout, posted a video to social media platforms displaying the outcomes of an attention-grabbing workaround to this restriction.
The demo video exhibits the open-source MobileNet V2 object detection mannequin working natively on Quest 3 at 0.75 Hz. This was chosen for practicality, slightly than for efficiency causes.
Gschwandtner despatched UploadVR a check construct and we independently verified that that is actual and works. He additionally solely revealed to UploadVR how precisely he pulled it off.
How It Works
Gschwandtner’s app construct creates a webview, a Chromium internet browser occasion embedded throughout the app, which mechanically hundreds the oculus.com/casting URL which Quest headsets can solid their view to.
You then have to undergo the total technique of logging into your Meta account on this webview. When you’re executed, you utilize the Quest system menu to provoke internet casting. Uniquely, you are casting from the headset to an app inside the identical headset, to not a unique system.
Gschwandtner’s app then samples the pixel content material, the RenderTexture, of the webview receiving the casted view, and from there can do with it no matter he likes with it. Within the demo instance, he is feeding it to Google’s MediaPipe on-device SDK.
A key limitation right here is that the view will embody digital objects and interfaces too, not simply the passthrough digicam view, that means it can solely work when a lot of the picture is passthrough. However Gschwandtner claims he has a “plan” for a workaround for this.
What Occurs Subsequent?
It is unclear how Meta will reply to Gschwandtner’s discovery. It is doable the corporate may make this system inoperable in a future model of Quest’s Horizon OS. Gschwandtner’s social media posts embody a public request to “hop on a name” with Meta CEO Mark Zuckerberg or CTO Andrew Bosworth to “present them” the method.
Final month Bosworth gave his ideas on the thought of deliberately giving Quest builders entry to the passthrough cameras during an interview, saying Meta will “hold ” the problem.
Meta “Looking At” Giving Quest Developers Raw Camera Access
Meta CTO Andrew Bosworth said the company will “keep looking at” giving Quest developers raw camera access.
As compared, for its Imaginative and prescient Professional headset Apple solely provides uncooked entry to enterprise companies, and just for personal inner apps.
