Apple’s Persona Technology Uses Gaussian Splatting to Create 3d Facial Scans

If I'm doing work at my desk and I get a Zoom call, there is a 0.00% chance I will go plug in my Vision Pro to answer it. I'm just going to open the app and turn on my webcam, spatial audio be damned.

A lot of people here work with text all day every day and we would rather work with text that looks like it came out of a laser printer than out of a fax machine.

I often think how stupid video call meetings are. Teams video calls are one of the few things that make every computer I own, including my M1 MPB, run the fans at full tilt. I've had my phone give me overheat warnings from showing the tile board of bored faces staring blankly at me. And yeah, honestly, it feels like a solution looking for a problem. I understand that it's not, and that some people are obsessed for various reasons (some more legitimate than others) with recreating the conference room vibe, but still.

And with monitors? This is a far more "spicy" take, but I think 1280x1024 is actually fine. Even 1024x768. Now, I have a 4K monitor at home, so don't get me wrong: I like my high DPI monitor.

But I think past 1024x768, the actual productivity gains from higher resolutions begins to rapidly dwindle. 1920x1080, especially in "small" displays (under 20 inches) can look pretty visually stunning. 4K is definitely nicer, but do we really need it?

I'm not trying to get existential with this, because what do we really "need"? But I think that, objectively, computing is divided into two very broad eras. The first era, ending around the mid 2000s, was marked by year-after-year innovation where 2-4 years brought new features that solved _real problems_, as in, features that gave users new qualitative capabilities. Think 24-bit color vs 8-bit color, or 64-bit vs 32-bit (or even 32-bit vs 16-bit). Having a webcam. Having 5+ hours of battery life on a laptop, with a real backlit AMLCD display. Having more than a few gigabytes of internal storage. Having a generic peripheral bus (USB/firewire). Having PCM audio. Having 3D hardware acceleration...

I'm not prepared to vigorously defend this thesis ;-) but it seems at about 2005-ish, the PC space had reached most of these "core qualitative features". After that, everything became better and faster, quantitatively superior versions of the same thing.

And sometimes yeah, it can feel both like it's all gone to waste on ludicrously inefficient software (Teams...), and sometimes, like modern computing did become a solution in search of a problem, in order to keep selling new hardware and software.

Because it's not. Facial expressions and body language carry gigantic amounts of information.

So many misunderstandings arise when the channel is audio-only. E.g. if a majority of people in a meeting are uneasy with something, they can see it on each others' faces, realize they're not alone, and bring it up. When it's audio-only, everyone thinks they're the only one with concerns and so maybe it's not worth interrupting what they incorrectly assume to be the general consensus over audio.

Why do we have video calls? Because a webcam costs $1-5 to put into a laptop and bandwidth is close enough to free.

Why do we have 4K monitors? Because they only cost a small amount more than 1080p monitors and make the image sharper with not a whole lot of downsides (you can even bump them down to 1080p if you have a difficult time driving the resolution). I paid $400 for my 4K 150Hz gaming monitor so going with 1080p high refresh rate VRR would have only saved me $200 or so.

Serviceability for purpose is a spectrum and the Vision Pro is at the wrong end of it.

For more than the price of three 4K OLED 144Hz monitors, you get to don a heavy headset that messes up your hair, makes you sweaty, screws up your makeup, and you get less resolution and workspace than the monitors. Your battery lasts an hour so it’s inferior to a laptop with an external portable monitor or two. It’s actually harder to fit into a backpack than a laptop plus portable monitors since it’s not flat.

Then you have to use some complicated proprietary technology [1] to make a 3D avatar of yourself to overcome the fact that you now have a giant headset on your head and look like an idiot if you were to go on camera.

You can’t do a bunch of PC stuff on it because it’s basically running iPadOS.

This is not the same as “why are we bothering with 4K?”

[1] What will you do if Apple starts charging money for this feature?

On the other hand, video calls are worse and less comfortable than audio calls.

Thank you! Now I get it!

So it’s sort of a stopgap solution before the ar glasses are small enough to do actual video calls without looking silly?

The vision pro’s overall productivity solution is inferior to existing, cheaper technology, and it has to be supplemented by a solution to a problem created by its own design.

Essentially you’re saying that after putting on a double headband device that wrecks my hair, gets me sweaty, strains my neck with weight, and fucks up my makeup, I now have to use a workaround fake avatar because the tech bros who made this product had to say “oh shit, if you have a headset on you can’t be on camera!”

For $3500 I can be in real reality and be surrounded by higher resolution professional monitors and just show my real self on camera instead.

The “when you’re using the headset” part is the issue. Why are we using the headset? What are the benefits? Why am I making these tradeoffs like messing up my hair, putting a heavy device on my head, messing up my makeup, etc.

This is like saying “The Segway had advanced self-leveling to solve the problem of how to balance when you’re on an upright two wheel device”.

But why are you on an upright two wheel device? Why not just add a third wheel? Why not ride a bicycle? Why not ride a scooter?

The solution is really cool and technologically advanced but it doesn’t actually solve anything besides an artificially introduced problem.

They could never cut the price down because of it. The knockoffs used much simpler ways to balance yourself, including just changing the form factor to something more conventional that doesn’t even need balance correction (scooters and e-bikes).

Human-only Internet: why choose this implementation over something simpler? Surely there’s a simpler way to prove you’re human that doesn’t involve 3D avatar construction on a head-worn display that screws up your hair and makeup. [1] E.g., an Apple Watch-like device can verify you have a real pulse and oxygen in your blood.

Court: solution is already in place, which is showing up to a physical courtroom. Clearing backlogs can be done without a technological solution, it’s more of a process and staffing problem. Moving the judges from a court to a home office doesn’t magically make them clear cases faster.

Medical checkups: phone selfie camera

Bureaucratic appointments: solution in place, physical building, or many of these offer virtual appointments already over a webcam.

Web browsing without logins: passkeys, FaceID, fingerprint

[1] yet another male-designed tech bro product that never considered the concerns of the majority of the population.

wtf apple, indeed.

My take is like, make me tether with usb-c, reduce resolution and increase latency if I go over what the connection can handle. Use foveated rendering. All I want is more screens.

For now, I'm working with Virtual Desktop on my Quest 3. It's not ideal - pixel density at the edge sucks and even in center it's not quite good enough for text unless I enlarge my screens to be the size of barn doors, but I get 3 very large screens out of my m1 and that makes me happy enough. It's also lighter than an AVP, which after test driving I assume multi-hour sessions would become a literal pain in the neck.

Whatever the tradeoffs are, though, if apple offered infinite screens with text-readability I'd gladly throw money at them for the privilege.

Tinfoil hat moment - I do wonder if the AVP devs got a visit from a bat-wielding gang of monitor engineers. Apple screens ain't cheap.