Augmented Humanoid - Project 2
This page details my experience building the second project of course CS 428 at UIC called “In the Land of Make Believe”. You can find also find a video demo of the application at this link.
The project involves displaying a humanoid version of myself in a themed scene. On weekends, I like to laze back in my couch and watch my favourite team, Arsenal FC, play in the English Premier League. The scene is a depiction of such a weekend and contains the following aspects -
- On the Astronaut image marker is the main playset, that depicts my “laid-back persona” on a couch watching the game on the TV. There are also some cool animations, like a burning candle, a bubble machine, and my imaginary pet cat, Javier
- On the Fissure marker, there is a virtual button that the user can play with
- The Drone marker contains a model of the table top that is used to simulate collisions
- The Oxygen marker displays my “workout persona” (wish this persona existed in real life) doing squats
- A life-size version of the scene over the Astronaut marker in a separate Unity scene that you can deploy on your mobile device
The scenes use 3D models to depict the theme, with the models being a mixture of freely available assets and 3D modelling done by me.
For running the code, you need the following on your computer -
Once you have satisfied the above requirements, go to the GitHub repo for the project and clone the project. Obtain copies of the above mentioned image markers used in the project. Open the project in the Unity editor. For the table top scene, go to
Assets -> Scenes and load the
TableTop scene in the editor. Connect your webcam as the camera source for Vuforia engine and click Play in the Unity editor. For the life-size scene, go to
Assets -> Scenes and load the
Lifesize scene in the editor. Deploy the scene to a mobile device such as Android or iOS by going to
File -> Build Settings, choosing the appropriate device, and clicking on
Build and Run. Make sure your phone is connected via USB cable, and make sure to enable USB debugging in the Developer settings of your phone. Once the app runs on the phone, give the correct permissions to the app and click on the reticle to initialize the scene at that location.
- Moving your hand over the virtual button on the Fissure marker makes the laid-back persona get up from the couch and fist-pump while saying “Let’s go!". This is a toned down representation of my behaviour when Pierre-Emerick Aubameyang, the no. 14 in Arsenal, scores a goal
- The Drone marker is positioned in such a way that the bottom edge of the image lines up with the bottom edge of the table top. So, the user needs to position the image marker on the edge of a real table surface to get the correct effect
- On clicking the left mouse button, red balls are generated on top of the Astronaut marker that fall over the objects in the scene. If the Drone marker is positioned appropriately, the balls bounce of the table top and roll away
- The workout persona can be brought out using the Oxygen image marker, and depicts me doing squats. When the Oxygen marker is moved really close to the Astronaut marker, the workout persona stops his squats to chide the laid-back persona, saying “Get off the couch and workout”. Laid-back me sometimes requires harsh motivations
- The life-size version of the playset on the Astronaut marker can be found in the scene labelled Lifesize in the Assets folder, and can be deployed to an Android or iOS device by going to
File -> Build Settingsand building it on your device. On the device, point your phone on the ground, and click on the reticle to initialize the scene.
The Use of Playsets in a Future with AR Glasses
- With the development of realistic humanoid models and animations, it seems likely that virtual humanoids will be adopted across different professions. Starting from playsets for kids, it seems like a cost-effective idea to have children interact with virtual objects rather than physical ones. It may seem annoying for parents to deal with boxes and boxes of toys, and hence, virtual playsets can solve the space limitations introduced by real-world toys
- There is a lot of educational potential to playsets, where existing learning methods and techniques can be augmented by humanoids. This is especially useful in a time like this, where everyone is working from home. It would be highly effective to have a humanoid version of your teacher pop up on your AR glasses and explain trigonometric identities to you. While this seems complicated to develop, we can quickly bridge the gap that exists between in-person learning and online learning
- What seems skeptical with kids is the amount of time that their parents would allow them to wear the AR glasses. For any AR app to be effective, the user needs to be able to interact with it for a good period of time, and in a future where many applications being shipped to AR glasses, just like how apps are available from the PlayStore, it maybe easy for a child to get addicted early. AR playsets can pose the same addiction threat as social media apps on phones, for example
- Having a similar version of you interact with you virtually does seem exciting as a kid, but at the end of the day, I would want my kid to have virtual interactions with humanoid playsets only where there is an opportunity to learn, and not “for fun”, but that depends on the parent
- There doesn’t seem to be a huge difference between the table top and the lifesize versions of me, primarily because I am still viewing the scene from my phone screen, but I imagine the effect will be more realistic through AR glasses, which opens up the avenue for a wide variety of collaboration apps
- Instead of Zoom meetings, we could have meetings like what is shown in Avengers: Endgame, where instead of holograms displaying the attendee, your vision is augmented with a humanoid version of the attendee, whose lips move in accordance with what they say
- Demos and tutorials can be made more realistic and dynamic with the use of humanoids. Imagine that, for a product like a washing machine, you have your favourite actor appear in their humanoid version and explain to you what every button does by pointing at it virtually. This could replace the instruction manual