augmented.org

Hey everybody,

two days of a great conference are over and now I’d love to give you guys an insight on what was going on at metaio’s insideAR conference 2011! Thanks metaio for the great show and the parties, it really has grown so quickly and all demos and the location has been so impressive. I guess the event gave quite a few headaches to the staff before (and later at the party to more). Let me wrap up the key facts for you, I won’t be able to go into all talks, though.

Keynote

The conference kicked off with an introduction of Thomas Alt (CEO) and Peter Meier (CTO) while being supported by a great moderating magician: to a non-tech guy AR still looks like magic and this as a red line was definitely a good fun connection. Thomas stressed their vision, that within the next three years AR will be a standard on everybody’s smartphone (and probably almost everybody will by then have a smart(er) phone). AR will grow for the consumer big time due to useful apps and solid use cases. There is more than one killer app.

His big announcement right at the start was the surprising news that they will now offer a free mobile SDK for all developers. Now there will be two SDKs: a pro edition, that includes their latest research results and tracking algorithms, and a free basic edition, that allows working with the same interface, but has some limitations (e.g. a watermark, no QR code reader, no face tracking). A great step for the developers out there!

Peter continued with some examples to explain their claim of “Making the digital a natural experience”: the whole interface concept and the way we interact with information needs to change and will be changed with AR and new approaches. The interaction needs to be easy, fun and more intuitive. AR can also influence the communication and social interaction of citizens on the long term. E.g. it even allows a direct dialogue of politicans and citizens, aiming at a more direct democracy (the used example was the Da-müssen-wir-ran-App from the greens party during the election in Berlin).

To allow faster creation of this content, Peter announced the metaio Creator. A new tool, that allows easy linking of markers and contents and speeds up the deployment of ideas. The shown alpha release already handled marker+feature tracking plus video and 3D content well (though the WiFi impeded the live demo).

He explained their latest tracking improvements like the GAFD algorithms (gravity aligned feature descriptor) to stabilize tracking on mobile/multi-sensor devices. Moreover we could see demos of a 3D outdoor tracking and a RGB and depth-camera based 3D-reconstruction approach, that made use of the Kinect to recreate the real world into virtual 3D (e.g. for occlusion handling and collisions). Impressive! The good news: the new technology clearly favors AR in multiple ways (from NFC to HTML5, LTE networks and stereo-camera smartphones (for 3D reconstruction)).

But the biggest claim was – introduced by an old school Wolfenstein3D shot – that a new technology state will be reached soon, comparable to the big steps computer graphics did during the early 90s, when the first person shooters hit the public (yielding in the big business of graphics hardware like nvidia and ATI). The same story will happen again – now to Augmented Reality. Especially hardware cooperations of metaio should foster this development and the vision of a natural digital interaction. For this, metaio closely works with different hardware developers, e.g. Vuzix and Sony (both showed their latest HMD prototypes and products) and esp. mobile platform developers (like ARM chipsets) to get the right speed, longer battery life for everyday AR and the best tracking and presentation.
Once these technologies mature enough Peter claims, we will have another technological discontinuity, that breaks away from the currently dominant design (like the touchscreen smartphone is the standard today) towards something completely new. A new era of human-machine and human-world interaction – possibly based on the smart combination of easier interaction methods and AR goggles. Can’t wait to see it happen! :-)

Exhibition

The demos were numerous and displaying a lot of different aspects of AR deployment. Be sure to check out the image gallery below for more to see. Mostly you could see use cases with the mobile SDK and junaio (enhancing posters, shopping situations, games, configurators, instruction manuals, etc.). Moreover they are well established inside the industrial realm (showing precise and fast tracking+overlay for prototyping and design reviews & planning). But to name only two explicitly, I’d pick:

The Augmented City, we’ve been talking about before, was shown the first time live at the conference. Their tracking on a Samsung Galaxy SII was impressively stable and the look of the CG is getting better and better. Judge by yourself above, in their offical video or in this live video, that has been put up before I could edit mine, so enjoy the one from Augmented Minds. Can’t wait for metaio to implement the whole concept of an augmented city in a 1:1 scale in the states!

Biggest fun was to see myself from the perspective of a toy car racing through the augmented real track outside the conference building! It’s weird to see yourself from below through your eyes through the camera of the car… ;-) Next time please with a combined AR:DRONE fight! :-)

By the way – we now have pictures up on our very own flickr account. So enjoy more of the conference online and conveniently embedded below:

Talks

The well fitting talks gave great new impulses on different sides of AR usage. John C. Havens from Transitional Media started on how AR will transform ads and your identity. He made good points regarding the unclear question of who owns augmented space: if an artists highjacks the Mona Lisa by overlaying it with his own art: can he be sued? Does the Louvre have the only right on the 3D volume of air around its building? Who owns the ad space? Especially: should the app developers get revenues from linked companies inside their AR browsers? If I as a user are being tracked: how much control do I have over my own person? Do I own myself? Inside this quite scary marketing vision he recommended to become your own brand and to turn the tables this way.

Josh Shabtai of Vertigore Games speaks about games as being the #1 driver of technology (before AR and with AR) and elaborates on the overall gamification process of our lives (e.g. like chore wars). He demands that AR games should trigger more feelings to the user and start our imagination, rather than trying to squeeze a bit of more polygons out of the tiny graphics chips. Altered Reality Games are the best example: the user converts much more of the real world around him into a gaming experience – than the graphics only ever could – by imagining much more than is presented. If we trick the brain through a great experience, audio and some visual hints, we can get the goose bumps much faster than by a single image. Lastly he envisions how future social AR games will demand for even specific real world physical spaces for the people to meet and play. (Like playing AR paintball.) Great talk!

Of course I can’t continue on all talks in this detail. Though there were great talks on AR example use cases (historical records, industry AR), story telling concepts and upcoming hardware for better AR-enabled experiences (scary: “You dont’ need to talk to nobody anymore, your phone is your best friend”) and more panel discussions, as you might have read here at metaio’s page anyway. But One more thing to mention: I loved the talk of Shelley Mannion (British Museum) on AR in culture and arts and especially her interest in augmented sound, I share by 100%. I must have missed the mentioned Inception Augmented Sound app, so check it out if you haven’t either. It blows you away and into the 13th dream level. :-)

metaio & the Future

The second day metaio’s researchers gave more insight on the latest development. The big bang was the progress in markerless RGB tracking on mobile devices and their approach of rgb+depth camera based tracking and meshing. To the latter we will see a presentation at this year’s ISMAR in Basel! Can’t wait. The live demo with a stereoscopic camera cell phone allowed direct and live-learning marker-free tracking, that will also be able to solve the scale problem (due to the stereo base). I hope I’m allowed to go into more details about all this soon.

Other news and plans include an online calibration for web-based apps, gravity-awareness to be used inside the SDK (allow more convincing experiences due to correct physics for virtual objects), continued work on tracking improvements (of course) and scalable mobile instant 3D tracking (indoor/outdoors) plus a mobile client-based fast recognition on large databases. This all will help to better localize the user on a much better level of accuracy and to create better applications such as outdoor AR-navigation or more detailed context-aware AR.
Also, cool to see, that they are always willing to integrate even more sensors for a better augmentation, if the consumer products/hardware offers it (e.g. thermal sensors, etc.).

Overall…

… metaio did it again. Great progress in tracking and 3D reconstruction and a complete porting to the mobile world is a great deal. Their clear vision that not only focuses on the core development of tracking but also thinks about the consumer connection and the hardware was impressive. They sure have a plan! Lets hope that their plan works out and that external developers will give us the needed killer app plus content and that metaio is not betting on the wrong horse, delivering “only” the awesome base for AR.

See you next year and hopefully also at the ISMAR11 in Basel!

Thanks, guys!

Arturo Castro shows a great video to the internet crowd, that amazes already thousands of people: he swaps his face that is held into his webcam in real-time for prominent faces. This face recognition algorithms plus replacement are of course well understood especially in the movie productions, but once again we can see how far free toolkits and standard hardware bring us today regarding real-time applications!

We can imagine lots of applications like avatar control (combined with altered-voice chat) or augmented video conferences to disguise yourself. This face substitution is not yet perfect, especially looking at the edges of the mapped skin without elaborate blending, but you get the idea and it should be sufficient for a big weekend “WOW” when seeing Michael Jackson chatting to you. ;-) There is more on real-time facial animation and to show another short video, I quote this one here, since it easily shows the underlying mesh and mapping process.

The demo uses openFrameworks, Jason Saragih’s c++ library for face tracking combined with Kyle McDonalds ofxFaceTracker, color interpolation algorithms from Kevin Atkinson to give full credits here. :-) Enjoy!

Faces from arturo castro on Vimeo.

… and if you were a little bit scared by this and on its possible future usage and scam potential, you can relax by having a cool drink with some augmented information and dancing girls. I guess the apps gives a bit of information about the product… or… not. :-)

Today a short update with my favorite videos of this week. The two clips show typical student situations: forgetting the lecture hall and playing FPS games. Well, basically there is not much more to being a student. ;-) Without further comments, I’d like to present these two funny videos showing AR goggles for the campus usage with bullying and for free time activities for sporty nerds! The first comes from AndyHyperIsland, the second via Freddiew. Enjoy!

A longer while ago I’ve been blogging on the company Azavea and their mobile AR prototype to support cultural heritage browsing and knowledge access. They teamed up with PhillyHistory, that lets you find historical photos and maps by date, neighborhood, address, and/or keyword in a collection of over 90,000 records.

Triggered by the Philadelphia Department of Records (DOR), Azavea started to research on the use of mobile augmented reality technology in displaying a large database of historic photographs. The app enables users to view historic images as overlays on the current landscape. After the research period Azavea published a white paper called “Implementing Mobile Augmented reality Technology for Viewing Historic Images”, that should answer the before asked questions like: Is augmented reality (AR) a useful method for showing the history of Philadelphia and helping users to see the connections between the past and the present? Is AR technology advanced enough to make this type of application possible? Are smartphone networks fast enough? Is the phone able to pinpoint a user’s location accurately enough to load images of that location even in a crowded urban setting?

That’s why I spoke to Deborah Boyer from Azavea to learn a little bit more first hand about the results. Deb allowed me to publish her answers for you all! Thanks again! It’s only some answers here, the white paper has a lot more. So, now: enjoy!

augmented.org: Was the GPS Positioning accurate enough for your purpose in the end?

Deborah: The GPS positioning was accurate enough although it could certainly be improved. Our biggest issue was with the jitteriness of the images. They often bounced or wobbled and simply didn’t stay in one place as well as we would have liked. Use of the gyroscope in newer smartphones should hopefully improve this in future apps. We talk more about the positioning issues on p. 22 of the white paper.

Are processing times fast enough for a stable and good experience?

When we chose to display nearly 90,000 points in the app, load time started to become an issue. We sped up the app by choosing to display only the four closest points as images. Other nearby points load as icons (which can be cached) and the user can then click to load the image as an overlay. We also only return a select number of POIs for any location. More information about processing times is available on pages 15-17 of the white paper.

Regarding input/output devices what are your learnings? what is missing today?

The relatively small screen size of a smartphone required us to pay careful attention to what we could display without overwhelming or confusing the user. We created an opening launch screen to help guide users through the app and used the Layar customization options as much as possible. The images were often very small in the AR view so we also built an option for users to load and view the image on another screen where it could be a larger size. This screen also contained additional descriptive information. While we tried to use the screen space as well as possible, it is still a fairly small space in which to show a lot of information. Tablets may be useful for their larger screen although the weight and size could make it difficult to easily use them with an AR app for long periods of time.

How do you prioritize POIs if there are too many? How do you guide the user?

We organized the POIs into three categories: twenty images with additional accompanying text, 500 images which were aligned in 3D space, and the remaining images (roughly 87,000) which made up the majority of the collection. The app prioritizes the images with additional text or the 3D aligned images over the other points. The prioritization also includes distance calculations to prevent POIs from appearing on top of each other. To guide the user, we created three different icons to indicate the different types of images. We also wrote a help page to walk people through the process of using the app. More details about prioritization are available on pages 16, 17, and 21 of the white paper.

Did you try prototypes with tablet devices?

We did not try the prototype with tablet devices. At the time of development, tablets with cameras were not readily available so we stuck to smartphones.

How was the user feedback, what are your results on usability/effectivity? E.g. does placing images in 3D space work better screen-aligned or “3D-space-aligned”?

In general, users were excited about the project and the research. When we released the prototype, we realized that users may have some difficulty viewing the aligned images due to the jitter that is often found in AR applications. We received some feedback that the images bounced around too much and made viewing difficult. Other users reported no problems. While the 3D alignment helped in some cases, other users found it confusing.

In the end: was the program useful and accepted? Will you continue in cultural heritage projects with AR?

The project was useful for investigating AR technology and its possible use in cultural institutions. We received a positive response to the white paper and hope that it will serve as a useful resource for other organizations investigating the use of augmented reality. We are interested in pursuing other AR projects and potentially expanding the PhillyHistory project but have no immediate plans.

Talking about a possible open source AR platform: what is most important for them and will you be forcing this, too?

With the variety of collections and media types available in cultural institutions, an open source AR platform could be useful to providing additional features and flexibility to organizations hoping to experiment with AR. The publication of open source AR standards and the creation of a community around open source AR would hopefully allow for more innovations and sharing of resources. We’re interested in the topic of open source AR but are not currently working on any open source AR projects.

The App was only a prototype and only available for a few month. But you can still try it using layar with a dedicated channel or read the full white paper for more research results! Thanks again, Azavea and of course, Deb for taking the time!

Have a nice week you all!