Using Artificial Intelligence to find wine, with Microsoft Cognitive Services and Wine-Searcher

The core of Wine-Searcher’s business is a database and search engine that indexes more than 9 million prices, helping millions of users every month find something to drink at the best price, via a desktop and mobile website, a mobile app, and now a chatbot.

We work with Wine-Searcher on their data warehouse, and know their business pretty well, so thought they would be a good fit for a hackfest we were planning with Microsoft. Says Head of DevOps John Buckwell:

“When Theta came to us with the hackfest idea we knew that we had some projects in development and some interesting ideas in the AI space that we just hadn’t had time to explore before. Participating in the hackfest gave us time and space to do that, and figure out which ideas we should pursue further.”

Head of Theta Digital Andrew Taylor adds:

“That’s one of the great things about a hackathon. As well as access to cutting edge tech and the experts who know how to use it, there’s a real benefit in spending dedicated time on a problem. Ideas can be validated and creative solutions emerge. It’s a model we’re using more and more with customers who are looking to innovate.”

Hackfest projects

During the hackfest two teams looked at two sets of AI technologies to address specific goals.

Label recognition with computer vision and optical character recognition (OCR)

Wine-Searcher was interested in how AI could enhance the product’s label recognition process and accuracy. John Buckwell outlines the objective:

“At the moment, Wine-Searcher does a pretty good job of recognising a label in a picture of a single bottle. We wanted to find out whether you could take a picture of ten or a dozen bottles – say a shelf in a store – and have Wine-Searcher find them all.”

The team used the Microsoft Cognitive Toolkit to train an AI model to recognize wine labels.

Photos taken at local supermarkets were used for training. The wine labels in the training photos were marked out and used to obtain a convolutional neural network. We deployed the trained model to an Azure App Service and sent images to that service so that it could send labels found to Wine-Searcher’s existing label recognition service.

We also showed that it was possible to improve the accuracy of label matching – which can be a problem when several wines from the same producer have similar labels:

‍

By using optical character recognition (OCR) and handwriting recognition services, they delivered a useful boost in accuracy that increases the likelihood of a first match being correct – which of course makes the Wine-Searcher experience better and faster for its users.

The team also took a look into the future with augmented reality. How could AR technologies change the wine shopping experience - by integrating label searching with wine information from the search results?

‍

Chatbot technology, language and voice recognition

Wine-Searcher already has a simple chatbot, but was interested in exploring whether a Wine-Searcher bot could do more. John Buckwell explains:

“One of the complications with wine in particular is that there are so many product names – like 600,000 – and then many different ways that a consumer might express that name. So they might say they are looking for a 'Domaine de la Romanée-Conti', 'Romanée-Conti' or just 'DRC'. When I first looked at language understanding models it seemed that every variation on an entity had to be listed. I was keen to see if, a bit down the track, it was possible to use machine learning and LUIS [language understanding intelligent services] to be smarter about it, and enable the bot to do more.”

The team proved the concept during the hackfest, and this is now being built into the roadmap for the chatbot.

They also explored voice recognition, using a limited dataset of 50 wines and a range of different voices and accents. While Wine-Searcher don’t plan to implement this immediately, they do see possibilities longer term for voice-enabled search.

‍

Reflecting on the experience

We asked John Buckwell whether they would participate in something like the hackfest again. He said:

“Doing a hackfest is a big undertaking, but we got a lot out of it. We had defined clear tasks to investigate over the week, and I think that helped. It was great to spend the focused time, and to have lots of support available – from Theta and from Microsoft. It was pretty cool to be able to contact the engineers at Microsoft in the US working on the products – that’s not something you normally have access to. And even having access to Theta’s kit like the HoloLens – it may not be directly relevant right now, but it’s a glimpse of the future and sparks the imagination.”