Using Deep Learning to Find Beautiful Photos by Alex Miller [Mind the Product]

In this ProductTank San Francisco talk Alex Miller, one-time software engineer in the content understanding team at Yelp, gives us a case study of using machine learning (specifically deep learning) to provide a ranking system that surfaces the most beautiful photos of a business to the top of their page.

Basic Photo Ranking

You may know of Yelp as a site where users post reviews of businesses. But Alex tells us that the volume of photos is actually far greater than the number of reviews. Yelp wants to ensure the photos that best represent a business are featured at the top of a business’ page. So how do they do that?

Traditionally, Yelp used basic metadata to rank photos such as user feedback (like and votes), photo attributes (photo size, age), and clicks. While this worked up to a point, the underlying assumptions in the model were flawed. For example, a photo might get a lot of clicks because it was blurry and users wanted to see what it was. Or a photo might get a lot of likes because it was funny. But neither of these would be the best representation of the business.

Helping the Model to Learn

Yelp was looking for a model that could better understand the quality of photos: features such as depth of field, contrast, alignment, and other characteristics that are typically present in higher quality photos. So it started using a model called a Convolutional Neural Network (CNNs) that would provide a better predictor that photo would qualify as “beautiful”.

For any well-posed machine learning problem, Alex explains that you need a model and data to train that model. Yelp had to find “gold set” of photos that it could reliably say were good and bad to help the model learn. The content understanding team found a variety of signals that helped to determine a proxy: if a photo was taken by a DSLR camera, it would be labeled as “good”. If it was a non-DSLR camera, it would be labeled as “bad”. This allowed the model to learn the characteristics it was looking for as Yelp built its quality score.

Putting the Model to Use

Now that Yelp had a fairly reliable predictor of photo quality, the content understanding team had to decide how to use it in practice. Alex talks about other considerations they had to take into account: daily batch vs. streaming evaluations of photos, APIs that would be needed by other teams to use the ranking, and filtering and diversification logic to ensure the ranking would be a feed that would best represent a business on Yelp. In the end, they were able to evaluate the results with metrics and provide an impressive improvement in their feeds using a deep learning approach to evaluate photos.

The post Using Deep Learning to Find Beautiful Photos by Alex Miller appeared first on Mind the Product.

Source: Mind the Product https://www.mindtheproduct.com/2018/04/using-deep-learning-find-beautiful-photos-alex-miller/