1/8/2016 August Schiess, Coordinated Science Laboratory
Written by August Schiess, Coordinated Science Laboratory
Imagine finding an image of someone wearing an item of clothing, putting that photo in an app, and being instantly presented with dozens of images of garments with similar colors, patterns, and shapes. Each item is linked to a website where customers can purchase it.
This highly customizable shopping experience is within reach.
Advanced Digital Sciences Center (ADSC) researchers Vassilios Vonikakis and Siddhanta Chakrabarty and their colleagues have commercialized FashionMatch, a fashion-optimized tool that analyzes apparel in an image to instantly find visually similar items and where to purchase them from a large database.
The research for this technology began in the Coordinated Science Laboratory by a team of graduate students, led by Bernard Ghanem (MS '08, PhD '10), in Professor Emeritus and Research Professor Narendra Ahuja's Computer Vision and Robotics Lab. Ghanem handed the project—then called FashionLatte—to ADSC for commercialization.
“The algorithm has mastered many computer vision challenges. Even with shadows and cluttered backgrounds, the algorithm can identify which pixels of the image correspond to the person, to the garment, and to the background,” said Vonikakis, an ADSC research scientist. “It can then identify the skeletal structure of the person, which makes it possible to tell the fashion-related proportions of the garment, like how long the sleeves are, the proportion of the neckline to the waist, general shape, and more.”
The selections draw from a large database of images, and users can customize their results based on their preference for certain similarity attributes. For example, users can rate how important it is to them that selections match the garment’s color, pattern, or shape, and receive different results based on those selections.
With a $285,000 grant from A*STAR, an agency in Singapore that encourages research commercialization, eight engineers were hired and split into two teams to work in parallel on developing two versions of FashionMatch for the marketplace.
Vonikakis led a team to evolve FashionLatte, and the other team worked on a similar algorithm developed by former ADSC research scientist Wang Gang. Vonikakis’ team focused on making the algorithm robust for real-life conditions, while Gang’s group focused on the business development side.
After the grant ended, Chakrabarty, an ADSC software engineer, joined Vonikakis’ team to make the technology more user-friendly for administrators. The two teams have already sold the license of FashionMatch to five companies, including Jet Solutions, IQnet, and Image Science, and are looking to sell to more.
“We started with a good algorithm, and then we upgraded specifications to meet the needs of clients. They wanted the app to function on mobiles and ‘in the wild,’ which meant identifying the garment accurately even with complications like shadows and cluttered backgrounds,” said Vonikakis. “Our two groups delivered two complementary solutions to the growing demand for personalized and customizable fashion search and retrieval.”