Okay, this might be the most impressive Tech Demo I've ever seen. It's also one of the hardest to explain on video, but I'm gonna try. So there's a thing called Google Projects Starline You might have heard of. it might have not, but Google has showed a few video examples of this Live super immersive like 3D video calling Booth where you sit down in front of it and on the other side someone else sits down and through the power of the internet, it looks like an extremely realistic version of them is actually in front of you.

It's not just a normal video call, but it's not a hologram. it's just it's something different. Either way, we got the first look at it at Google I O A Couple of years ago, they showed this clip in the keynote and nobody got to record it. Nobody even got to try it.

But now they're back with a better looking, more compact, more simplified version and I got to try it and record it now. Just as a warning, this first clip, it doesn't actually do it justice because the effect does not translate very well on camera. We'll get to that. but I was lucky enough to get my first reaction to this.

my actual first time seeing it on camera. All right, let's see project start line. Oh wow, oh hey, how are you great? Okay, welcome to Starline Thank you! This is right. Yeah, it's a lot more.

Yeah, you and I just saw each other right outside, but we're in two different rooms and we want to make it feel like we left that space and we came back together in, uh, virtual space. It feels like we're completely in the same spot. It really looks like I can. actually.

uh, it looks like you could drop something on this table right here. Yeah, yeah. I've got a little apple over here, so let me show you that too. So you know you and I are in the same room.

Do you want me to pass this over into your space too? It looks like you can do it. Thank you! We're meant to feel like this room is just one room instead of two separate locations. It could be anywhere in the world. So this is, uh, something we think is the glimpse of what communication could look like in the next few years.

That's incredible. Okay, yeah, I gotta learn more about this. Okay, so what exactly is happening here? So I'm sitting in front of the new Starline Booth is what I call it now. The first star line they ever made was much bigger.

They had these like full room setups essentially with a bench and computers, the display, but also a bunch of cameras and depth sensors placed around you. This new one feels actually much more refined. It looks much simpler and it kind of feels like a reasonable product. It's a 65-inch display on a stand with a smaller barrier over the bottom bezel, and then there are actually some lights on the back that point at the wall behind the display that serve as a key light for me, the person on the call.

and then there are no more depth sensors. it's just a tidy array of color cameras and micro phones and speakers. so that's at the top, the left, and the right. and they use AI to create a depth map of me and the room I'm in and then the magic of this is twofold.

It's the display and the Computing happening so the display. the part giving you the actually impressive immersive 3D depth effect but then also all the Computing happening to turn me, the person on the video call, and the person on the other side into this realistic 3D model just with the camera information that you can actually look around with head tracking. Now, clearly this doesn't translate well on camera, which is why Google has understandably been super protective over anyone trying it even or or getting any photos or videos of it because it just won't look right. But I was able to convince Google to let me try something you see basically once you first sit down in the project.

Starline Booth This system has to identify where your face is and then you can move around and look at stuff with head and body tracking. and of course that's not going to work if you just put a camera in there. but if you do show it a face and a camera at the same time like say, maybe a cardboard face where you can stick a camera lens somehow through that cardboard face, then it would track and you could, ah, see what I'm talking about. Check this out! So what you're seeing right now is as if you were on a video call with me in the newest version of Google Starline This is the first time anyone outside of Google has seen the sort of visualization which is super cool, so it feel pretty reasonable right now, right? I can hold up an object, you can see the colors and the shapes and the lighting and textures on it, and this is all information compiled from the array of cameras around me.

and the depth information comes from these regular cameras. and that's all quite cool. But as you start moving around, that's when it starts getting a little more interesting because I can hold things out and you start to create this Parallax effect with the head tracking where you can start to literally look around and inspect the object and look underneath things as if you're literally in the room with me. It's super convincing to the actual eye when you see it on this screen.

Even the background behind me is not real. it's being composited in, but I'm casting a shadow on it and that's being rendered in real time too. It's a lot. We can even flip a switch to show a sort of a topographical map of specifically the depth information coming from these cameras.

So again, no depth cameras specifically being used here. But the array of color cameras has all this depth information being mapped and that's what all the information from the color cameras is being sort of projected onto. There's a lot of processing. Happening Here So it all comes together to form this really impressive real-time I mean I use the word immersion very lightly usually, but I will I Want to stress that it's very immersive when you actually get to use it? so hopefully this helps you visualize.

so that is pretty cool. But even that doesn't translate perfectly because the feeling of depth actually comes from the distance between your eyes. So a camera lens is one eye. but if you've ever tried to catch something with one eye closed, you know that humans perceive depth by the difference between what you see with your left eye and what you see with your right eye.

Your brain stitches them together and figures out depth that way. So even that demo doesn't quite get the full realisticness of what I saw on camera. but it's pretty close. The lights will display in Project Starline is literally showing a different image to my left eye and to my right eye.

So taking advantage of that biological fact and then letting me actually compute depth on the Fly while doing all the head tracking in real time, It's actually kind of crazy like if you've ever seen 3D glasses. you know how when you go see a 3D movie like one of the eyes is blue, one of the eyes is red. That's literally because your left eye will see something map for that and you're right I will see something that for that and that's how it creates the depth effect. But this is way better than a 3D movie.

It's so much smoother and the head tracking and everything that goes with it and it's it's so good. But then the craziest part. The the most impressive part is the Computing that's happening which is rendering the people using Starline in real time into these 3D models. So this is not like the metaverse thing or the VR headset tracks face movement and then you can have a eye contact face-to-face conversation with someone's Cartoon Avatar But no, no, this is this is taking the actual Imaging the lighting the the way you actually look what you're actually wearing and rendering that out in real time in 3D space and making it look like I'm talking to them through a window.

It's kind of amazing. The eye contact is so one-to-one real looking and it really just comes from the fact that I'm sitting here looking forward and the person on the other side is also looking forward and so we're actually looking into each other's eyes. So anyway, yeah it it is. The point is it's super realistic I promise you.

when I was when I first did my demo the first time I ever saw it. even knowing that it's 3D I still felt like I could reach out and like high five him or like fist bump him. He held that Apple out actually and to my eyes it looked like he could just drop the apple he was holding right on the table in front of me. Which of course meant that I looked stupid from this angle because I'm just reaching out at nothing.

But we got a bunch of reactions from some other Studio team members who got to join me for this demo. I'll link a video that has some of their reactions below. We should definitely check that out. So it was really only the like slight glitching, a little bit of edges and fading and stuff like that that kept out of Uncanny Valley territory.

Like obviously the eye contact is one thing and the 3d effect even. It had spatial audio so if I leaned over here it would sound like more audio came into one of your ears and that was responsive. But you know things like in between the fingers or like the edges of certain Fabrics or my hair especially. Some of that stuff could kind of break it a little bit and you could tell.

but honestly I wasn't thinking about that at all so you might be thinking kind of the same thing I was after I Finished this demo which is okay, this is super cool, but what is this for? Like what is this sort of a tech demo actually useful for? And the answer is we don't really know yet. So at this point in time Google has worked with a few companies Salesforce Wework and T-Mobile are some examples who are literally using some of these booths for meetings. basically. I Guess theoretically that's better than a zoom call, although it still has its limitations like the face tracking can only work with one person at a time, which means only one person can be in the booth at a time to get their realistic depth effect, so adding another person doesn't work.

I Think the question will be closer to being answered when the tech gets even better like we, it's already gone from the size of a room to the size of like an easel. It could just be the size of a TV with a backpack on it, but this will get more realistic, cheaper, simpler, and just better. But you know? here's the thing. I Actually don't know that regular people will actually want this like hear me out if I know anything about Bleeding Edge Tech which I've tried a lot of.

It's that regular people, the masses are not very fast to pay extra for higher Fidelity like better quality so the the cutoff for acceptable quality is surprisingly low for the masses. Like think about it, audio quality I Don't know. Streaming on Spotify over Bluetooth seems to be good enough for most people. Airpods are the most popular headphones in the world.

You think about cameras and how smartphone cameras are basically good enough for 99 of the taking pictures of your kids. you know. So convenience is King that like. that's why right now FaceTime and Zoom are it's just a square of a flat low-res video feed on a screen.

That's fine. It's fine for most people when you turn it all the way up to project Starline which is like this: incredibly realistic. like you can see micro expressions and textures and feel like you're in the room with the person that's over on one side here and FaceTime and Zoom are on the other side here. And this is where the masses live.

and this is like businesses, you know, getting a booth so they don't have to fly people out for lots of meetings overseas. but it's still too expensive and too difficult to get to for most people. but that's just for now. That's just for this version now.

I'm really looking forward to keeping an eye on how this Tech evolves from Google and even from others who are working on this whole 3D light field display technology stuff. We've actually seen some interesting stuff before. I Actually tried an Asus laptop not too many weeks ago that does the same thing. It renders a unique image for each of your eyes and does head tracking for an incredibly realistic 3d effect.

But of course for that you have to look at a certain file that was built in a certain model and work with certain software. This was like rendering real people and just having a conversation. It's crazy. it's crazy.

I'm gonna thank you for Google for letting us see this and I'm gonna hopefully be able to try future versions when they get finished because this is wild. Let me know what you guys think. Thanks for watching Catch you in the next one! Peace!.