TECH IN NEW WAYS
We embedded Google TensorFlow into Unity to combine augmented reality with artificial intelligence in a real-time mobile application.
So why did we start exploring this technology? Our Chief Technology Officer, Marco, digs a little deeper:
What made you start this experiment?
Augmented reality is simply not enough on its own - we need to contextualise the objects that we augment into the world. So we started experimenting with object recognition to see how we could possibly enhance augmented reality experiences.
"THE TRUE METHOD OF KNOWLEDGE IS EXPERIMENT"William Blake
How did you go about it? What technology did you use?
Embedding TensorFlow into Unity for a mobile application wasn’t something that had been done before, so we had no theory or demos to work from. A lot of playing around later, we used the large-scale COCO dataset to embed object detection into our mobile application.
We created several AR assets based on what the AI could recognise. We invited our creative team to decide on the context for each object, playing around with different concepts and real-world scenarios.
COCO is renowned for its common object dataset, so it wasn’t difficult to imagine real-life scenarios where we could utilise these objects: scanning a banana can bring up a baking recipe, whilst scanning a toothbrush could remind us to brush twice a day.
Were there any obstacles?
We had two very technical main challenges: using unity as a game engine and embedding TensorFlow in unity for mobile, neither of which had been done before. But we had a play around, did some problem-solving and got there in the end.
Next steps? Are you planning on improving the current prototype?
We are not just limited to object detection with this kind of technology - we can apply any AI application that runs on mobile, so we are also looking into implementing technologies like body tracking and image segmentation.
We can also recognise two objects, making it possible to scan them together and augment information around them.