A neat new demo showing Augmented Reality for a virtual dressing room comes from the nice guys from ARBUZZ. Artur Studzinski showed me their work using OpenNI, NITE and XNA – I must have missed it on vimeo before. It shows augmented clothing using the virtual mirror metaphor. But unlike other demos we have seen before, we now have a pretty good prototype showing the combination with a depth cam for registering the bones/body parts. This enables them for new spatial user interfaces and even more importantly: to make use of cloth physics simulation, so that the skirt, pants or sweatshirts moves along with you and is not stuck to one position and feel like made of cement or washed with way too much starch.
The demo video shows the big potential for this use case, though render quality and performance is still rough (funny score though!). But we are on the right track! :-)
Some of my Facebook followers and active AR browsing people may have seen these before, but I thought they are worth sharing here as well: for one, the nice butterfly demo of sensaa, that actually puts you in an augmented virtuality world. The demo is a bit rough as well, but the concept is fun:
Talking about great concepts…
…we (possibly) will have another nice way of killing time during long car rides for your (grand) children. Quoting on the Toyota youtube appearance:
The window can lets onlookers zoom in on objects they see outside, and it can translate and identify objects in another language. Augmented reality can help viewers gauge the distance of landmarks, and it can also help them identify constellations in the sky. Of course, these features are just a concept, but in the future windows could be the next computer screen.
… and if you want to experience the world like Geordi La Forge you can check out this neat HMD experiment! Additionally you get an uncoupled virtual position from your physical one letting you stumble through another representation of the world. Confused? Me too. :-)
Damn, busy days, so I thought I’ll swim with the wave and dig out some awesome Harry PottAR (omg) gags. But so far it has been only one demo from EA, that brings us some Harry fun to enjoy before we go to the movies (or don’t). Interaction between marker bound characters is always fun (like in the Star Wars Jedi fight), but nothing too crazy you’d expect with their creativity budget. Nevertheless, good fun for HP fans. I’d like to see more magic for this kind of franchise! :-)
E.g. for toy lovers: the AR.FlyingAce app for your AR.Drone! You can see the drone’s view on your ipad or iphone and shoot missiles towards your opponent. Below the fun marketing video and here another one that reminds you more of a paparazzi.drone. ;)
Short conclusion (e.g. as seen in German Computer magazine critics): bluetooth range sucks for this kind of game – you literally have to run after your drone not to loose the connection. But the idea is great, esp. that the loser automatically drops from the sky! Love it!
For the weekend some mixed news on the latest AR stuff this time for the early weekend on a Thursday. But before that I’d like to thank all, who gave me such a positive feedback on the interview-styled blogging (last weeks AR on previs). If you have your own idea or project you’d like to share, feel free to contact me directly.
Dreamy Future Cars
The 1st Prize at the Imagina Awards 2011 in “Best Design and Communication” went to Michael Harboun, who asked himself the question: “What if you could travel to exotic, far-away destinations while being stuck in traffic?” The Aeon Project explores the possibility of AR inside sci-fi dreamy transport vehicles. The concept video is a great watch, altough it get’s a bit freaky with the spherical vehicle and also spherical-dreamy music, when he suddenly turns the whole windshield into a VR world. Scary and beautiful!
Update: Video should work again! Sorry!
More of today’s world is…
…another neat video from Vuzix. I always like Magic Book styled things, where objects use the marker itself as a gateway/door to step into our world. Here it’s nicely done with the toon character (and his dog!). Also the integration looks pretty convincing, making me scream: let’s get the next generation of AR goggles already! :-)
Staying there with Pop-Ups…
… I liked this ipad2-demo, where we have a pop-up-pop-up-book with AR (you’ve heard me). It actually extends a regular pop-up book with a AR pop-up, instead of extending a regular book with an AR pop-up. ;-)
Also for people like me scanning the world for cool new ipad apps, there might be coming up in fall 2011, giving us a funny multiplayer game for iOS-devices! Hope to see a version with real occlusion with the objects lying on my table, though!
…is what you can do at César Mora’s open source project, he started to give a free framework for AR development. It’s multi platform and comes with its own markup language and allows extending scripting through goold old python interface. You can join him on sourcefourge!
…and in the end…
…you might be fed up with the looks of your robot vacuum cleaner in your house! So by then it’s time to (sorry) abuse the virtual furnishing real room demo and augment your vacuum robot with something cool and moving! :-)
This week, I promised you to post the interview I had some weeks earlier with the CEO of Lightcraft Technology: Eliot Mack. Eliot and me talked about their approach of using AR technology to support movie productions on set with a live preview. It brings the before mentioned previs and the post-production process together – live to the set. Their system Previzion has the claim of “Visual Effects to Go”. It’s a combination of high precision camera tracking, sophisticated rendering, and VFX quality keying that makes it ideal for on-set compositing of virtual backgrounds and CGI characters. They’ve been working on the science fiction TV-series “V” doing live previs and post-production!
Hi Eliot. Thanks for taking your time. I’ve introduced previs to my readers and read about your system. But to kick off from the other direction: is Augmented Reality a known term in previs and do you consider your work somehow AR?
Hi Tobias. Thanks for having me. In the previsualization community, the real time combination of virtual and live action worlds has quite a few names. ‘On set live preview’, ‘stage vis’, ‘live compositing’, and ‘Simulcam’ are a few of the names. It’s the same basic concept.
In a similar vein, ‘Augmented Reality’ is a fairly wide term, but in practice it usually means through the lens recognition of fiducial targets and features, and overlaying imagery on those targets.
Previzion definitely shares some technical origins with AR work in fiducial recognition and CG/live action graphics, but has a different development focus. For example, the target recognition camera is separate from the scene camera, as the targets can’t be visible in the scene.
We ended up driving the technology into the specific realm of production level VFX work by incorporating high accuracy lens calibration, custom built inertial sensors, photorealistic real time rendering, VFX quality keying, and complete data logging to create a system to handle the very large scale problems that are encountered in the production world.
I was introducing the reader to the previs concept before. When do customers use previs live on location?
We have two main areas: feature film/episodic previsualization, where we show a real time preview and record the motion data for use later in post production, and in camera finishing, where the keyer and photorealistic background rendering are of high enough quality to be used as the final product.
We get used anywhere there is CG mixed with live action (greenscreen, scene extensions, animated characters, vehicles, driving plates, etc.) where the speed and quality has to be very high.
Eliot gave me the permission to blog this video showing their work in action:
Your company worked on the production of the TV-series “V”, where you did on-set previs with your system. Could you describe shortly the setup we’ve seen in the video and how it works?
The video above gives a pretty good overview. The tracking sensors are mounted on the camera, the lens motors connected to the lens, and the optical fiducial targets are mounted on the ceiling. The Previzion system brings all of the raw tracking, video, and lens data together, renders the matching 3D background, composites it, and sends it out to the preview monitors with only a 5 frame delay.
Does it also work outdoors or only on stages?
For outdoor shooting, you can either mount the optical targets sideways, and point the tracking camera at them, or use encoded Technocranes, jibs, and dollys. We’ve done a complete integration with the General Lift encoding system, and adding other encoder systems is straightforward.
How long are the setup times?
Setting up the optical targets on a stage takes about a day. After the stage is calibrated, daily on set operations aren’t affected much by the system’s presence, except for the live preview’s presence.
Likewise, measuring an encoded crane takes about an hour on a pre-production tech day, and the daily setup time consists of powering on the crane encoders when the crane is in its default position, then making 2 measurements to locate the crane in the virtual space. It’s quite rapid.
Having set up all hardware – you mentioned before that you also do live compositing with e.g. 3D set extensions, that can be broadcast as is. With Augmented Reality tracking we often experience inaccuracies or tracking errors. Is the accuracy solid enough for live compositing?
The accuracy is good enough for live compositing in many cases. The floor contact is not subpixel accurate (it slides by a couple of pixels), but if you have a practical floor and can blend the CGI seam into the practical floor it’s fine. We’ve already been used to final some on the air children’s TV shows and infomercials.
Sounds great. So, let’s talk about the concept. What is the biggest advantage for whom on set? How does the system helps to visualize ideas and decisions?
When we started building Previzion, I thought the main benefit would be a big cost and time savings in post production, by automating much of the tracking process. [...] The main economic benefit is to the producer, as automating the tracking enables them to use hundreds of shots/week in episodic TV (like “V”). The automation definitely happened, but what surprised me was how much the live preview opened up the on set creative process:
The main creative benefit is to the director and DP, as they can see the whole shot come together and maintain creative control over what the show looks like. Without a preview, people tend to be conservative on greenscreen shoots, as they are worried about having someone run into a (virtual) wall.
When the camera operators had a live composited feed coming over the monitors, they could suddenly operate a virtual shot exactly like a normal live action shot. They could plan camera moves with the entire composition in frame, sweep around virtual geometry to maximize the dramatic impact of the shot, and make more powerful images.
The live preview simply transforms the VFX shooting process by bringing instant visual feedback.
…continuing with instant visual feedback – what is missing today in the AR approach and what do you want to integrate in the future to the system having collected a lot of experience with “V” and other projects?
We are continuously evolving the systems in the direction of better rendering, keying, tracking, stage usability, asset management, and data recording.
We are already seeing several customers using the system to finish shots in real time, as the keying and photorealistic background rendering are quite good.
We’ll be pushing the envelope of what is possible in real time very hard, as the cost and time savings made possible by finishing on set are simply enormous.
We’re also doing a lot of work in stereoscopic rig tracking and metadata handling, and working toward live stereoscopic preview. The optical lens calibration system that we invented is a natural fit for the precise optical matching required in stereo VFX.
Since we build the entire real time pipeline, we can work toward incorporating advanced VFX tools like 4:4:4 log space camera data input, HDRI lighting and rendering, scene referred linear compositing, color grading, and device corrected output color space transforms for realistic color previews. These are all standard tools in high end feature film VFX, but with our unique software and hardware architecture we can make them work in real time, which makes those techniques viable in a huge range of productions.
Talking about HDR lighting and high end visual effects… this seems to be too much for real-time today?
At a certain level of scene complexity, of course, it becomes something you have to render in post. Since the Previzion real time VFX pipeline is set up to mirror a post production VFX pipeline, the settings that were used on set to generate a director-approved look can be used as a perfect jumping-off point for the post process, without losing data or time in the process.
So you exchange lower quality place holders with higher quality later on. Do you more and more aim at live photorealism or what is the most important feature today?
Photorealism has obviously been critical in our in-camera finishing work. What has been more surprising is the desire for sophisticated rigging, models, and lighting in the on set feature film preview world. Everyone wants to see what the movie will actually look like as soon as possible.
Much of the virtual background 3D design work that traditionally happened in post production is being pushed into preproduction in the previsualization stage. This is a good thing, as it makes all the departments communicate before and during production instead of finding problems when it is too late in post to do anything about it.
In a nutshell to conclude. The great thing about your system and AR in movie production?
The benefit to post production is that Previzion takes care of a lot of the boring work – tracking, data recording, lens matching, etc. – and helps make sure that elements shot on the stage match the background lighting, have good greenscreen lighting, etc. It basically removes most of the unpleasant surprises that can cause post budgets to skyrocket.
Great! So, thanks a lot for all this insight, Eliot!
Hope you’ve enjoyed the interview. Please let me know in the comments or on facebook, if you’d like to see more of this format (or even have an interesting interview topic for me). :-) Cheers!
“Movie productions and AR – does that match?”, you might ask. Movies are produced offline and not real-time! Well, that’s true and I don’t want to get into the field of interactive “movie” experiences or games like in theme parks, either. But I want to focus on one special topic: helping movie productions with previsualization (previs) and real-time – but this time augmented!
For that reason I want to recap shortly the process and in the second part during next week I am very happy to be able to publish an interview I had with the CEO of a leading industry company in this field of live previs and “live post-production”!
Combining computer images with real actors and real sets for a movie production is a complex task. We all know the bad results from B movies and outdated movies, that used to look awesome, but fail to convince us today. When it’s done well, we will forget that it is actually a fake & combined image we see on the big screen. Visually great movies like Lord of the Rings, Avatar or (going back) Terminator or Tron are great examples of this combination of real and virtual. Especially since CGI blockbusters rule the cinema we are more sensitive to visual effects and defects: we won’t immerse until it’s done 99%+ right. We’ve reached a level of visual perfection, that is so scary, the images can make any dream convincingly true…
…but there remains one problem: the production process is a sequential more-step-process, that only integrates computer generated content after shooting the real actors and real sets. “What’s the biggy?” you want to ask. The huge problem with this approach is that neither actors nor directors nor the photographers/cameramen can actually see and experience the monsters, dinosaurs, flood waves, etc. This leads to blind acting and demands a high level of imagination, intuition, a lot of repetitions and a long post-processing period for integrating the CGI material right. Once the real actor was shot, there might be no chance to reshoot this actor later on… The cameraman won’t know how to plan the shot and where to point the camera (possibly cutting off parts, that would only appear later on during the post processing). This again leads to numbers of repetitions and makes it more costly in production. Acting blind leads to worse performances and to get the timing right between a not-yet-existing virtual character or prop and the real actor is very hard.
You see these problems every day during the news, when the weatherman stands in front of a green screen (see tech video below) pointing nowhere, while possibly checking his acting on an extra screen aside the stage. This usually leads to funny poses, when the weatherman tries to get a glimpse on that screen without doing it too obvious. Sorry, it always fails. ;-) Even though, it already is a live integration with a fixed camera!
But getting back to movie productions: as it is a complex task with a lot of variables and high costs, the producers started visualizing the story boards more accurate. They not only produce a shot list with a nice drawn story board, but also create complete sequences digitally rendered in 3D to get a previsualization – even before shooting the first piece of celluloid. This ranges from moving 2D cardboards through space to a madness of pretty-good-realism. The below video shows an example of a production with a “previs” before starting the actual production and the final result:
Previs comparison for “WANTED”, director: Timur Bekmambetov
This way it is possible to have a more accurate planning of the later-on shot. During my work at Berlin based company rise|fx we had the same tasks: previsualize in a 3D real-time engine to check for dolly setups, green screen sizes, camera angles, right lenses, etc.
… but it was still only done before shooting. On set the cameraman, director and actor didn’t get another chance to get an advantage out of the technology. The basic setup could be done better and planning the shot before was easier. But now it’s again every man for himself/every woman for herself…
The expensive previs stops here. Everyone is stuck with his intuition, a portion of luck and hopefully a pile of money to have enough time of reshooting it over and over again until it fits later on with the not-yet-existing CG.
But here we are today. Big studios like WETA with Peter Jackson shoot augmented steady cams, ILM does on-set integration of Caribbean Pirates… With augmented reality on set we can overcome the limits of the classic post-production way! How this works in detail, how it helps the overall movie production and the final quality… and even more will be the second part next week! Stay tuned! :-)
TDK just announced their latest OLED technology UEL476, bringing us neat transparent RGB colored displays closer! The tech specs are a field angle of 2.4 inches, transmittance of 40 percent, brightness of 150 cd/m². They claim to get pretty good viewing angles and fast response times by currently 240×320 pixels on 36.0 × 47.9 mm.
Could be another way to get closer to AR Head Mounted Displays…
Windshield HUDs
… or towards high quality HUD systems for our cars and navigational needs or wants. Company Autoglass has put up a marketing future vision video I wanted to share nevertheless, since it nicely illustrates the HUD idea again:
Will the car windscreen display key information about speed, fuel and any issues with parts of the car. Even combine visual sensors with augmented reality , online maps and GPS technology to provide drivers with live, visual information about the places and hazards around them.
People’s hopes and dreams are brought to life by the “Island of Hope” at the Venice Biennial AR Intervention 2011.
The island searches for the world’s most hopeful tweets.
It searches for the hashtag #hope and considers what the tweeter is hoping for. Only the very best hopes of that day are answered with physical objects. These objects erupt from the island volcano, floating upward and spread, settling across Venice.
At 5:00 p.m. each day the most re-tweeted #hope floats up into the sky at the Giardini and San Marco Square with the ringing of the bell. The Island of Hope is viewed on supported smart-phones.
The show runs from May, 31st to June, 12th! Be sure to tweet around, too! :-)
Good for early or late hours, when you definitely don’t want to see that yawning or pimpled face of yours… Additionally just convert yourself further to an Anime hero or any other cartoon character with the work of samahmuslima(?) ! See the virtual character follow all your mirrored moves! Full body/joints/facial tracking, keying, overlay, occlusion and fun included!
Well, obviously we don’t get that live yet. Damn! :-) I couldn’t get a hold of her/him, but it’s described as the final project in engineer study. A great and funny concept video showing us once again the cool wonders of Augmented Reality! Just love it! Well done!
If you, samahmuslima, read this, please drop me a line. Is there a HD-version? Or who did it? Any translators around? :-)
busy days down here in Munich! You are probably preparing yourself for ARE 2011 on May, 17-18th, 2011 in Santa Clara (2010 video remix here), while I’m setting up my stuff for our conference in Munich, 18th – 19th: the RTT Excite 2011 on photo-realistic real-time rendering, VR, digital prototyping and marketing. If you are in Munich: be sure to come over to say Hi in the FutureLab with some neat Mixed Reality demos, too! :-)
Android based Realism
Adding realism to AR by using real world surrounding information (HDRs panoramas, IBLs) is a thing we all want to have sooner than later! We have seen a cool demo of Suomi’s VTT on this, where they fake the surrounding information using the background and scanning a ping pong ball for lighting information.
Now we have another candidate porting a similar approach to our beloved mobile devices. The paper of the University of Münster (Germany) has been published already in 2004 (titled “Virtual Reflections for Augmented Reality Environments”), but now it’s time to hit the mobile world with it!
If you are interested in some tech talk, please read on! :-) Their approach can create CubeMaps for all reflective or lighting purposes in real-time using only the information from the background video frame: they not only flip and stitch the frame six times and mirrored together, but do a pretty smart estimation, that yields in plausible neat-looking reflections (although obviously fake): they project six regions of the background frame (see below) onto the new six sides of a Cube. The general idea is to have reasonable regions to map from. Parts from the left will be put into the left Cube-side, parts from the right into the right Cube-wall, lower parts for bottom, upper parts for the sky-part above. The center of the image will be used for the back-facing side. Well, but then the front cube face is still missing and we don’t get the easy way for it. They approximate the information needed by projecting outermost parts of the image onto the front face. This looks pretty convincing and the singularity in the middle can be neglected. Especially when having non-planar and more complex objects (rather a bunny than a flat mirror…).
This way they get to simulate glossy and diffuse reflections already. Lighting/shadow influence will probably be the next step, since the CubeMap is already there…
Two nice twists are further to it: first, they adjust the region (image below) for the background part by looking up, where the CG will be placed and taking the “behind”-pixels from that area and distorting the adjacent regions.
Second, inter-virtual reflections are made possible by multiple render passes: by setting a recursion depth, multiple objects can reflect each other (including the real-world reflections!), by capturing the rendered image including the 1st object and using this as the input for the CubeMap reflections for the 2nd object. Smart! :-)
Can’t wait to see more complex reflection and light calculations through two and more cameras (back-facing, front-facing, stereo…)!
More Kinect Demos
In other news, AR Door put together another fun virtual mirror dress-up demo for a virtual fitting room for Topshop:
It seems to me they only use the body’s center position for “rigging” the cloth to the person. It is pretty nice already, but I hope we can see even better integration (possibly when the official Microsoft Kinect SDK is there this summer) for the moving bones, also regarding occlusions and hopefully even some integrated cloth simulation, too!
Juggling with the Kinect is what Tom Makin does in his cute video (code here). It just made me smile, especially waiting for the European Juggle Convention 2011 (in August in Munich). So greetings to you, Tom!
Today I’d like to short-feature Keiichi Matsuda again, who brought us the great concept video of Augmented (Hyper)Reality. He has a 30 minute talk on his concept designs of an augmented world. To refresh your memory on his work, I’ll quote:
The architecture of the contemporary city is no longer simply about the physical space of buildings and landscape, more and more it is about the synthetic spaces created by the digital information that we collect, consume and organize; an immersive interface may become as much part of the world we inhabit as the buildings around us.
Augmented (hyper)Reality is an ongoing independent research-by-design project by Keiichi Matsuda, exploring scenarios for our future occupation of the city in the context of emerging technologies and ubiquitous media.
Unfortunately the sound is a bit off, but you might want to lean back and get some first hand thoughts on his AR. :-)
PS. Don’t forget the AR Stammtisch in Munich tonight!
The Royal Family does not only stay inside of the Westminster Abbey and the yellow press, but now hits your smartphone, too. You can’t escape! If you didn’t make it to shake Williams hand yesterday evening you might want to fake it with a cartoony version of Catherine or William on your or a friends head to post your encounter to your social hub. Aptronic came up with a face-tracking app to augment the royals for your gossip pleasure! Enjoy the big wedding day!
In other news…
Staying in London (on June, 16th) the Future is… Augmented Reality Summit. “Whether you are a Brand, Advertiser or Agency, Technology provider or Developer looking to meet key industry figures, keep up with the latest developments, technology’s & tools or simply to network with other from the world of AR – The AR Summit is an excellent platform for you.” You might want to check this Europe-based summit out, …
Qualcomm announces the version 1.0.0 commercial release of its Augmented Reality Platform. You might want to try out the official release here.
Architectural Projective AR
Projektil created an “augmented reality installation, a by 5 projectors recorded architectural model in scale 1:200. On the model can be shown various simulations (nightlife, sun and shadow, etc.), and also information (use zones, green areas, transport etc.) . Not only this installation is controlled interactivly by an ipad, but the whole space, which includes 44 dimmable dali lamps, 6 DMX lamps, 9 beamers, a touchscreen and the audio system.”
