Semantic AR

 

COMBINING CUTTING-EDGE

TECH IN NEW WAYS

We embedded Google TensorFlow into Unity to combine augmented reality with artificial intelligence in a real-time mobile application.

So why did we start exploring this technology? Our Chief Technology Officer, Marco, digs a little deeper:

What made you start this experiment?

Augmented reality is simply not enough on its own - we need context to the objects that we augment into the world.  So we started experimenting with object recognition to see how we could possibly enhance augmented reality experiences.  

"THE TRUE METHOD OF KNOWLEDGE IS EXPERIMENT"

William Blake

How did you go about it? What technology did you use?

Embedding TensorFlow into Unity for a mobile application wasn’t something that had been done before, so we had no theory or demos to work from. A lot of playing around later, we used the large-scale COCO dataset to embed object detection into our mobile application.

We created several AR assets based on what the AI could recognise. We invited our creative team to decide on the context for each object, playing around with different concepts and real-world scenarios.  

COCO is renowned for its common object dataset so it wasn’t difficult to imagine scenarios in which we use these objects: from a scanning a banana to bring up banana bread recipe to scanning a toothbrush reminding us to brush twice a day were some of the real-world applications we brought into the app.  *** will need to be changed if we go to the zoo

What were the challenges?

It was a very technical challenge.  We had two main challenges: using unity as a game engine and embedding TensorFlow in unity for mobile, neither of which had been done before.   

Next steps?  Are you planning on improving the current prototype?

We are not limited to just object detection with this kind of technology - we can apply any AI application that runs on mobile, so technologies like body tracking and image segmentation we are looking at implementing.  

We can also recognise two objects, making it possible to scan them together and augment information around them.  

 

Happy Finish
2018

Experimenter: Marco Marchesi

INTERSTED ? WE WOULD LOVE TO CHAT
GET IN TOUCH

To discuss your project, request a portfolio or learn more about Happy Finish, send us a message below or call our studios direct, we’ll be happy to help.