TECH IN NEW WAYS
We embedded Google TensorFlow into Unity to combine augmented reality with artificial intelligence in a real-time mobile application.
So why did we start exploring this technology? Our Chief Technology Officer, Marco, digs a little deeper:
What made you start this experiment?
Augmented reality is simply not enough on its own - we need context to the objects that we augment into the world. So we started experimenting with object recognition to see how we could possibly enhance augmented reality experiences.
"THE TRUE METHOD OF KNOWLEDGE IS EXPERIMENT"William Blake
How did you go about it? What technology did you use?
Embedding TensorFlow into Unity for a mobile application wasn’t something that had been done before, so we had no theory or demos to work from. A lot of playing around later, we used the large-scale COCO dataset to embed object detection into our mobile application.
We created several AR assets based on what the AI could recognise. We invited our creative team to decide on the context for each object, playing around with different concepts and real-world scenarios.
COCO is renowned for its common object dataset so it wasn’t difficult to imagine scenarios in which we use these objects: from a scanning a banana to bring up banana bread recipe to scanning a toothbrush reminding us to brush twice a day were some of the real-world applications we brought into the app. *** will need to be changed if we go to the zoo
What were the challenges?
It was a very technical challenge. We had two main challenges: using unity as a game engine and embedding TensorFlow in unity for mobile, neither of which had been done before.
Next steps? Are you planning on improving the current prototype?
We are not limited to just object detection with this kind of technology - we can apply any AI application that runs on mobile, so technologies like body tracking and image segmentation we are looking at implementing.
We can also recognise two objects, making it possible to scan them together and augment information around them.