Hologram Expert Daniel Smalley On Why We’ll Colonize Mars Before We See Displays That Really Are Holograms

I’m not sure why seeing all the product references lately to holograms makes me a little crazy, apart from the simple fact that none of them really meet the definition. It’s not like that’s the one term marketers abuse. We’ve seen bezel-less displays that had bezels. MicroLED displays that aren’t actually microLED. And on and on. 

I don’t entirely know what really does meet the definition, so I thought I’d ask an expert. Daniel Smalley is an associate professor of electrical engineering at Brigham Young University in Utah, and a genuine expert in the field. He’s working, his CV says, to make the 3D displays of science fiction a reality, using “waveguide-based modulators and optical tractor beam technologies.”

The short summary is that we’re not there yet, and in this conversation, we get into why that is – with the biggest reason being bandwidth and the immense computing power needed to genuinely make the holograms of Star Wars and Star Trek actually happen, and work.

We also get into a discussion of the various products already on the market that have co-opted the hologram term, and also talk about the real world, practical applications for holograms.

Daniel went to MIT and has his masters and a Ph.D, so he’s approximately a billion times smarter than me. This talk gets technical in spots, but I tried valiantly to keep up!

Transcript

Daniel, thank you for joining me. Can you explain your role at BYU and your interest in holograms? 

Daniel Smalley: Certainly, I’m an Associate Professor of Electrical Engineering here at Brigham University. My research primarily has to do with advanced 3D displays, including holographic displays and volumetric displays. 

Okay, and when you say you’re doing research, what does that mean?

Daniel Smalley: So it is our group’s manifest destiny, as we see it, to recreate the displays of science fiction, specifically the Princess Leia projector from Star Wars and the Holodeck from Star Trek, and so research in my mind is the steps we take to get from where we are to those places 

And where are we in those steps? 

Daniel Smalley: On the holography end, as we’ll talk about, I’m sure, the primary challenge now is that we can make little teeny tiny holographic video displays, but the bandwidth issues, the sheer computational power required to make big displays remain an obstacle. Some estimates have suggested that we will colonize Mars before we have the capacity to easily feed a big holographic display with all the pixels it’s hungry for and on the other side, on the Princess Leia projector side, we’re in a similar space, but with more hope. That is to say that we can make little teeny tiny Princess Leia projections, but I think we’re not far away from getting moderate and maybe even large-size volumetric images in the near future.

So let’s do a level set here. How do you define holograms and holographic visuals? 

Daniel Smalley: Yeah, that’s an excellent question. So there have been meetings of the minds where we’ve discussed and debated what these things mean, and I think the best way to think about the different display families is that there are three of them. So a trifecta of holographic display. 

The first is a “ray” family of displays, the second is a “wave” family of displays, and the third is a “point” family of displays. Now the ray displays are the displays we’re already familiar with. These are lenticular displays, stuff that you might see at Best Buy or in a magazine. These crisscross rays of light and space form an image point that we perceive, what we would call a real image point. A holographic display is a step up from that. Instead of taking rays and intersecting them in the air, what it will do is it’ll take its whole surface, so you’ll be gazing at a screen and this whole surface is focusing light, it’s curving away in front of a light, in order to focus at a point, and your eye perceives that focal. As a display point. Now the magic of holography is you can take that surface that’s shaping light and you can superimpose many such surfaces, one on top of the other, and focus on multiple points and in this way, build up an image in the air, and these images can be optically indistinguishable from real objects. 

So if you’ve seen a really good hologram in a museum, you may be tempted to pick it up and look behind the glass to see if there is a real object behind it. Even a seasoned holographer will occasionally mistake a hologram for a real object. Now it comes with the price of the fact that there is a glass, that you have to be looking through a screen of some type. But the reason for this is that wave shaping is being performed by a pattern of lines, a diffraction pattern, where there are three ways of bending: light, reflection, refraction, and diffraction.

And in a hologram, diffraction is the active ingredient in creating this wave shape. So you have to be staring into those lines. You gotta be staring into that pattern if you hope to see something, Now that said, imagery can be very deep. Looking into that hologram, that window, you can see imagery that comes out and tickles your nose or goes way back to infinity, back to the horizon. But you’ve always gotta be watching it like you watch a television set, even if what you’d prefer to do is watch it like a water fountain, right? Where the aperture is flat and then there’s content shooting up out. Then you can walk all around it and see it from every direction. Now, that type of display exists, but it’s not a hologram. It’s called a point display or a volumetric display, and unlike ray displays and wave displays that require screens, a point display can be screenless. 

In fact, maybe the best way to think about it is you take its screen and you grind it up into little pieces and you scatter them into the air, and then each time you’re looking at one of those little pieces, you’re looking at an image point as well. And that’s the technical definition of a point display is that every time you’re looking at an image point, you’re also looking at a group of atoms, a physical scatterer, which is to say, unlike the ray case, where you’re looking at an intersection of photons or the hologram case where you’re looking at the focusing of the wavefront, here we’re looking at physical atoms scattering light. So in some ways, a volumetric display is a lot like a 3D printer that just destroys the object it’s creating every 30th of a second and this endows it with some remarkable properties. So you can make images that you can see from every angle. It can be relatively low bandwidth images if they’re sparse and they have what’s called perfect accommodation, which means you can focus on them. Your eye believes even if you close one eye, you can focus really tightly on them and have really strong 3D cues. Now, the downside is that with these types of displays, it’s hard to achieve the same level of realism that you get with a holographic display, and the reason for this, is you can imagine if you had a jar of fireflies and you’re trying to make images out of these fireflies, no matter what, you’d always have this problem where you can the fireflies in the back of your image at the same time, you can see the fireflies at the front of your image and in the result is that everything looks like a ghost or a hole, right? So this problem of self-occlusion is a big one, and it’s one it’s part of the research we do is try to come overcome these issues so that it can be a complete display of the solution. 

In terms of array display, you were describing lenticular. So in the context of this stuff that people listening to this might relate to. Going back a number of years, there were what were called glasses-free 3D displays that were basically LCD displays with a lenticular layer over top of it and if you looked at it from different angles, you would see something was popping up from the screen. Is that basically what a ray display would be? 

Daniel Smalley: Absolutely, that’s exactly right.

The wave display when you were describing that, I was immediately thinking of that little company in Brooklyn called Looking Glass and the little loose-eyed blocks that they have. 

Daniel Smalley: So Looking Glass and I don’t want to misrepresent them or anything but Looking Glass, I think I will admit they are a ray display technology.

If you look at a Looking Glass display and you move left and right, you will see the image change perspective. But if you move up and down, you won’t. And that’s an indication to the viewer that you’re looking through a cylindrical lens as opposed to an array of circular or spherical lenses. Now the difference between them is that if it’s a lens-lit array as opposed to a lenticular array, then you can move up and down and you’ll also see 3D in that direction. But you can dramatically reduce the information you need by just making it horizontal, parallax only. They’re just providing information for the horizontal and your eyes for the most part don’t care. They’re horizontally separated. You don’t do a lot of bobbing up and down, so you get the most bang for your buck with just horizontal parallax. 

Yeah I’ve seen the Looking Glass stuff, I think I might have seen it at a trade show but I was underwhelmed. It’s like, I’ll shift to my right and I’ll shift to my left, and it does seem like the image is subtly different, but it’s one of these things where I’m going that’s nice, but so what?

Daniel Smalley: Yeah, that’s true. There is also some fatalism about three 3D displays that when you get really good, you’ve just now duplicating reality, which is something we’re very used to, and it just becomes suddenly banal. It just suddenly looks like everything. 

So what would be an example of a wave? Are there real-world examples of a wave family display?

Daniel Smalley: A wave display that you could go out and buy today, I don’t know, but there are certainly many good static displays. There are certainly commercial companies making an effort to create wave displays. Two approaches that are gaining traction commercially, I think, are holographic displays, which are a pattern of lines that refract light to form a wavefront or a nanophotonic phased array. There is a caveat, there’s a merging between the ray and the wave family at the moment when the rays come from emitters that are very small, smaller than a wavelength of light. If those emitters are super small, number one and number two, if all the emitters can see each other, that is to say, they have some fixed phase relationship with each other. The technical term for this is coherence. They act as a team. If all those things are true, then you can start shaping wavefronts with what would’ve been rays. So essentially if you have a big emitter, the ray comes out like a laser. But as your emitter gets smaller and smaller, the ray doesn’t come out like a laser. It comes out more like a, I don’t even know how to describe it, a spray, right? It defracts out more and more until now you’ve got a spherical emitter and all those spherical emitters see each other and they interfere with each other in ways that allow them to create arbitrary wavefronts. Any wavefront you want, you can create from a collection of spherical emitters, assuming they’re small enough and assuming they’re coherent with each other. 

So that’s another approach that some people are taking. But the problem is, in each one of these cases you’ve got just an intractable information problem. For example, any display could be made into a holographic display if its resolution was sufficiently high if it could achieve holographic resolution, which is roughly a thousand pixels per millimeter linear. So imagine taking all the pixels in your computer screen right now and squishing them into a 1:1 millimeter area and then refilling your computer screen at that density.

So that’s a million times more pixels than what you’re currently using to create a display the same size as what you’re currently using, and so you’re talking about if you wanted a meter-size holographic display updated, at a reasonable refresh rate you’re looking at in the neighborhood of hundreds of billions of pixels per second, maybe trillions of pixels per second to create that display.

So you’ve got challenges with computing power, with graphic processing, with bandwidth, and everything else? 

Daniel Smalley: Yeah, but primarily bandwidth. The feeling I think, broadly, is that optical electronics is a solvable problem. We might even be able to get pixel densities where we want them, maybe. But that compute power, that remains a big deal. 

Now there are shortcuts and workarounds. One particularly good workaround was by SeaReal back in the day, what they would do is they would look at the viewer’s eyeballs and they would only shoot light into the eyes, light that was diffracting in other directions they would ignore entirely. It wouldn’t compute any of that, so they could dramatically reduce the amount of the information they had to process and they could increase the pixel size because they only needed just a little bit of diffraction, just enough to cover your pupil, and then they were done. It’s unfortunate that we haven’t seen more from them.

They started out with a kind of mechanical version of the display that worked really well, and I think there was a struggle to make something that was solid state. But it was a pretty clever trick to reduce this bandwidth while still preserving the benefits of a wavefront-shaping holographic display and the realism that comes with it.

So where do light field displays fall into all this? Are those waves or points? 

Daniel Smalley: So this is the most controversial of all of this syntactic infighting that we have right now, because there are displays out there right now trying to commercialize light field displays, and they don’t want anyone thinking that they’re any less, that consumers are getting anything less than what they might consider being a holographic display.

And how they use the term and how we use the term are often very different. So those of us who’ve gotten together and agreed on this, say a light field display is a ray display. That is to say, it’s a pixelated display that’s shooting rays in different directions, and it’s those intersections that create image points that our brain perceives. Though I know there are displays out there, or at least they’re attempting to create coherent Wavefronts, that is to say, these nanophotonic phased arrays. They’re trying to create phased array wavefronts potentially, and I can’t be sure this is the case, but they do have wavefront shaping capabilities and that’s when you’ve crossed the bridge from ray display to a wave display. 

Are hologram and holographic Interchangeable terms or are they different things?

Daniel Smalley: So hologram as we see it, the way we decided to specify this term, we define a hologram as the surface with the lines on it that’s actually diffracting the light. So if you go to a museum and you see a hologram, the glass plate that you look into, the screen itself, that is the hologram, and the image that’s the holographic image. And then the process of creating that is holography. So we use holography to create holograms, and when we illuminate those holograms, they create holographic images. 

Is a spinning LED light stick that are these individual sorts of fan blade things and arrays of them that are being called holograms? Are they holograms?

Daniel Smalley: No. There’s nothing diffracting. So if there’s no diffraction, then it can’t be a hologram. Now it could be a volumetric image. What’s happening with most of these is there is a fan that spins in a single plane, however, if you just move that fan in and out, you just oscillate it in and out, or if you add a bunch of fan blades stacked on top of each other and spin them, now you’ve created a volumetric display. Now, every time I look at one of those image points, I’m looking at a physical object in a volume and I’m getting a volumetric image and it will have all of the benefits and all the deficiencies of that family of displays, of that point family, but not a hologram. 

So when you say it’s volumetric, it means if you went off to the side a little bit, it’s not just this single flat image, there’s a dimension to it or depth to it? 

Daniel Smalley: So when I say volumetric, I mean that If you look at an image point, you’re looking at a physical object, in this case, an LED. Of course, it’s just a flat screen, it’s just spinning in a plane.  If it wants to be qualified as a 3D display, then it needs to have pixels or voxels that exist off a plane. So you just need to stack these or move one of them in and out, and then you could achieve this effect of having a volumetric image. 

It’s yet more moving parts in these things, which would worry me even more. 

Daniel Smalley: That’s right. If they weren’t dangerous enough. 

Is a transparent LCD a hologram? 

Daniel Smalley: That is a good question. So that depends entirely on what are you displaying. So first of all, it could be a hologram if you’re displaying a pattern of lines on your transparent hologram meant to diffract light so that far away it’s converging to a point for somebody to observe. That kind of display would not be very useful unless the pixels of this transparent LCD were very tiny. Now, in the case of some microdisplays, for example, there are transparent LCD microdisplays for projectors, that could be a legitimate holographic display that would actually create an image that we would appreciate as a holographic image. 

Now, those microdisplays are micro, they’re small maybe an inch, maybe one or two inches on a side. So they’re not particularly well suited to humans. But they would make great pets or insect displays. The challenge now is to keep that same pixel, those teeny tiny pixels, those teeny tiny transparent LCD pixels, and then scale that size up while keeping the pixel small to something that a human would appreciate, something in the 20-inch diagonal range.

So these shower stall dimension displays that are transparent LCDs that are just nicely lit, white screen captured visuals of people who were standing in one place and it’s reflected on the transparent LCD inside the shower stall thing, that’s being described as a hologram, and when I’ve written about it I describe it as hologram-ish. But it wouldn’t qualify as a hologram, would it? 

Daniel Smalley: It would not. But I will say this, I think that the tradeoffs made there are actually pretty compelling. So when it comes to representing full-size humans, we have to recognize that humans are flat, especially if you’re looking at somebody standing on a stage, the six inches of depth from the front of their nose to the back of their head is not much in the grand scheme of things, especially if you’re looking at them from 50 feet away or a 100 feet away, which is why the two 2Pac “hologram” was so compelling, because the further away you get from an object, the fewer 3D cues your eye is able to use to determine. 

So when you go to a play, they can paint the background, the mountains, and the sun, because those things are so far away. The only 3D cues we get are occlusion. The fact that one is in front of the other, but it could be totally flat and those pictorial cues are all we need. As objects get closer, we start adding things like motion parallax. When you’re driving down the road, now you see these telephone poles moving with respect to each other, and then as things get a little closer, now you get left eye, right eye disparity, and it’s only when they get really close within a few meters does your eye start being able to focus on the near and far parts of that image and you get these accommodation effects, and then when they get within arms reach, you can touch them, and now you have keen aesthetic cues. So it’s really when things are up close, within arms reach that you get this rich set of 3D cues, but if you push imagery back far enough, you can really get away with a lot. Things get much cheaper, and much easier, and if the intention for these shower displays as you call them, which I think is a pretty accurate description, if it’s just to give the sense of the presence of another human being in a room, and if they’re a few feet away, that might be a reasonable trade-off, especially if they’re pushing all those resources into creating really high dynamic range, which they do, good color saturation, and high responsibility. 

Those things are gonna be much more compelling to a human viewer than those six inches of depth. We’re boring as far as 3D is concerned as humans. 

Yeah, I’ve seen light field displays at the SID trade show and I have seen the shower stall devices at different trade shows, and if I think of the two, the light field display is arguably closer to what people are thinking about as a science fiction hologram, but they’re also six inches tall, and I suspect that most people having to choose between the two would say, I like the life-size thing a lot more, even if it maybe isn’t quite as sophisticated in certain respects.

Daniel Smalley: Absolutely! 

When I talked to the guy at Portal, David Nussbaum, who founded that company, it used to be called Portal, and that’s the shower stall displays. He says, I know it’s not a true hologram, but we have to call it something and it’s something that consumers have their heads wrapped around so that’s why we use that. Is that a fair approach?

Daniel Smalley: Yeah, I think so. As I say, we’re all very defeated at this point on this. So I think that if you’re trying to communicate with humans and it’s already entered the vernacular in that way, unless we give them an alternative, then what else is a guy supposed to do? 

I’m curious longer term as this technology matures, what are the real-world applications for this? Because, if you’re replicating Princess Leia and Star Wars that’s a theme park attraction or a museum attraction or something like that. But are there practical business uses for holographic visuals? 

I did see a demo from a company up in Newfoundland, called Avalon Holographics and that was for energy exploration and shipping and so on, to show the depth of the ocean and all that, and I thought, that’s pretty interesting. So is that kind of the more, the real-world use of this going forward?

Daniel Smalley: That’s a very good question. I think we have yet to find the killer app for holography,  to be honest. So in any of the scenarios I’ve been approached with, it seems relatively straightforward to come up with something that’s almost as good for much, much cheaper. In the case of oil exploration, they’re trying to understand these complicated 3D shapes in the form of oil fields and where to dig and this kind of spatial stuff. But unless time is an important factor and it’s not in this case, you can use a really big, nice 2D screen, move your mouse around and rotate around enough to get a real good sense of the 3D shape. People are really good at abstracting from 2D to 3D, and I’m thinking of radiologists in particular who just make this second nature.

However, if you were a surgeon and you were trying to thread a catheter through the vasculature of the body, which can get very complicated in 3D, especially as you approach the heart and the brain it might be useful to have a really high fidelity 3D image that you can see as you’re pushing this catheter to avoid getting abrasions on the artery surface causing embolism, that sort of thing, and the reason for that is because time is important. You’re moving that catheter in time, you’re being able to capture the spatial information at the same time you’re moving is sensitive. Time is a sensitive part of this process and so maybe in that case.

Maybe if you’re doing aerospace surveillance, we’ve got all these extra satellites, thanks to Elon Musk and SpaceX to keep track of and the possibility of conjunction, which is the smashing together of satellites, I think it’s greater and greater all the time, and that’s more complicated than airplanes smashing into each other because you got these curved orbits and I’m sure there are all sorts of AI and computer analysis, but there’s still a human loop, I think in most cases, and they have to make a judgment call about whether these two complicated orbital paths are gonna result in the smashing together of two objects, and if you have that rendered in 3D, you’ve got this moving spatial situation. I think you could understand what’s happening much more viscerally than trying and abstract that from a 2D screen so I see those as two, clear and present applications for a really good holographic system. 

Is there a lot of business investment in this or is much of the work involving holography happening in environments such as yours, more on the academic side?

Daniel Smalley: Definitely more on the academic side. If you’re talking about the display, the real money in holography has never been in the display. It’s always been in things like security or photolithography or some of these other fields. 

So holography for currency counterfeiting?

Daniel Smalley: Yeah, that’s exactly right.

So I don’t imagine that’s going to change. My feeling is the display field is just fraught. It’s just a terrible market to be in, it is. If you think about the last century, we really only had two dominant display technologies. For the majority of this century, you had CRT displays, and then for the rest you had LCDs, and during this time, big companies were cannibalizing their own technologies. New things were coming on like miniature cathode ray tubes and all sorts of interesting OLEDs, just think how long it took OLEDs to take off even though they were superior in so many ways. It was just, you’ve got these multi-billion dollar foundries, and fabs, and you’re gonna squeeze every last drop out of those displays, and then the margins are so small and yeah, it’s just a rough business to be in. 

So thelast century in the early part of this one has just been littered with good technologies, good 3D technologies that just couldn’t get a foothold. In the 90s we had two excellent 3D displays. We had the Actuality display, which is the spinning paddle which was a very nice display, and then, it had a hundred million pixels, I think, per second, and then we had Sullivan’s Crystal display where he had these stacked liquid crystals that he would project on to form a volumetric image, are also excellent and solid state for goodness sake, and that both of those, about the 90s, both of those couldn’t quite find a foothold in the market. 

Is it the sort of thing that could be revived? 

Daniel Smalley: Oh, it has been revived. So there is a version of this type of display, which I called an enclosed volumetric display where you have a diffuser moving up and down inside, what I presume is an evacuated volume, and then you’re projecting on that and it looks beautiful, it looks great and they’re making a good try. They’re making a good effort to get out there and solve some problems. 

My feeling with most people who are doing 3D displays is that the targets they’re looking at are in entertainment, people who are trying to do VR or something like this, but need some collaborative platform to develop on that, where everybody can gather around and that becomes this volumetric display or in this case, Looking Glass is also good at this, and then I think Sony has another beautiful 3D display auto stereo for the same sort of thing, targeting that same sort of market. 

Yeah, I’ve seen that. Where do you think things will be in 10 years from now? Will there be commercial products out there, or is this still gonna be in the labs? 

Daniel Smalley: I guess we have to dig down a little bit on that question. What are we gonna have? Well, we’re gonna continue to have better and better displays for sure, and I think we’re gonna start making inroads on niche markets. I think we are seeing companies take this tack of hitting premium markets first. So oil exploration will be in there, entertainment will be in there, and hopefully, we’ll have a Tesla-like experience where they’ll get a nice premium product with lots of really inspiring features. They’ll identify a killer app and then the trickle-down will provide the rest of us plebians with a 3D display in the next little bit. 

Things are accelerating, lots of technologies are converging. I think it’s much more likely that you’ll see an everyday volumetric display before you see an everyday holographic display just because the information problem, and the bandwidth problem’s not going away. And I say volumetric displays. I should also say that displays like Looking Glass, these light field displays or more correctly, maybe these ray displays are also gonna get better and better, and we’ll have to make some decisions about whether we are willing to pay the premium to go from that excellent ray display to a much more expensive holographic display. 

This was very helpful, very technical, I even understood some of it. I appreciate you taking the time with me.

Daniel Smalley: Yeah, my pleasure. It’s my favorite thing to talk about.