Close Ad

The Spoon

Daily news and analysis about the food tech revolution

image recognition

Samsung Adds Food Image Recognition To Bixby Through Calorie Mama API

Search has come a long way since the earliest algorithms deployed by Google, Lycos, and Inktomi. After conquering basic and complex queries, search engines set their sites on images, video, and audio as frontiers that required new ways of looking at metadata to provide consumers with useful results.

Image recognition has been a focus of developers wanting to add value to the basic ability to capture and identify a picture of a peach or a fast food meal at Wendy’s. The key, as exemplified by companies such as Palo Alto-based Azumio, is to link image recognition to valuable datasets. For Azumio’s Calorie Mama AI-powered platform, the company offers an API available for third-party developers as well as a consumer download which allows users to track nutrition intake.

While Azumio faces competition from Google and Pinterest, a new partnership with Samsung may allow the folks behind Calorie Mama to separate itself from the pack. Samsung has announced a working relationship with Azumio to adds its Calorie Mama technology to Bixby, the South Korean giant’s AI personal assistant platform. Calorie Mama will be baked into the new Galaxy S9 and S9+ enabling users to obtain instant nutritional information about the food they eat.

“Our vision for the Calorie Mama API is to provide the best food image recognition technology to our partners,” Tom Xu, co-founder of Azumio said in a press release, “and to simplify nutrition tracking and food discovery for healthy living to their customers.”

While this announcement is a nice to have for Samsung smart device users, the true value goes far beyond the basic ability to count calories and set nutrition goals. Azumio’s work primarily is focused in areas related to health in such areas as diabetes and sleep disorders. The company’s Argus platform offers activity and diet tracking along with a social network focused on health and fitness. Argus powers its suite of applications such as Instant Heart Rate, Sleep Time, Fitness Buddy and Glucose Buddy.  The endgame of connecting food recognition with health-related applications, focusing on those in which diet management is essential, is where the real power lies.

Samsung and Azumio’s combined efforts began in 2013 when the Argus platform was offered in Samsung’s Gear line of devices. Samsung continues to work on its own proprietary health and fitness apps, but those have not been offered to consumers outside of Korea. The value of adding Calorie Mama to Bixby could show great promise. For example, a cook wanting to create a healthy meal could ask Bixby to recommend a substitute for a high-fat ingredient by tapping into its database of image data.

Pinterest—which is preparing for a possible IPO—will undoubtedly rise to the challenge to go beyond its image recognition work with Google (called Lens) which allows users to find recipe pins based on captured pictures. Pinterest recently has hired a new head of computer vision, Chuck Rosenberg, a 14-year Google veteran. Given the primary task of computer vision technology is to analyze images and tie those results to associated data, Pinterest is on an accelerated path in this space. Unlike Samsung, working in a somewhat closed ecosystem, Pinterest will make its application available to all takers.

Visual Search Holds Great Promise for the Future of Food

The fruits of advances in visual search technology offer great promise for the future of food in a wide range of applications.

Applying such technologies as computer vision, machine learning, artificial intelligence and the ability to delve deeply into tags and other metadata, innovators can build applications that can assist in identifying food properties. These can be valuable to help build healthy eating habits or even speed up the line at your average fast food restaurant. KFC in China is working on using a facial-recognition payment system geared to moving people through long waits at record pace. Ant, a financial subsidiary of Alibaba has the means to identify a person in two seconds and match them with a photo ID that lives on the Alipay network. Using a special algorithm, Ant says it is nearly impossible to spoof the system using a photo instead of a live shot, as well as being able to identify an individual even if he or she is wearing heavy makeup.

The long arm of visual search can touch the food industry in ways that use the power of the smart home with great utility. Amazon, for example, has its Rekognition technology which uses deep learning and a massive database to identify images, including pictures of food. Rekognition’s API is available to developers who could apply its power to many smart kitchen appliances, such as the family fridge. Take Samsung’s smart fridge as an example. Its ability to see what’s inside the appliance, take a recipe and identify what needs to be purchased can be married with a database using visual recognition. That list then is passed on to Alexa (which pairs with Samsung’s appliances) and within hours, everything a cook needs to make a special dinner is on hand.

While many of the culinary applications of advanced visual search are a bit “blue sky” in nature, especially as they apply to the smart home, there are enough efforts in market to show the concept’s promise. Pinterest and Google have individually developed what could be called “Shazaam for food.” Pinterest’s version of Lens allows users to snap a photo of food and bring up recipe images for that ingredient. As with other search products, Pinterest’s Lens still is in learning mode, not able to distinguish between a yellow squash and a banana.

Google’s visual search product, also called Lens, was announced at its May developers conference and is not yet available. Google plans to incorporate Lens into its smart home portfolio using Google Assistant.

And then you have an idea that looks good on paper but perhaps has too many moving parts to be viable on a mass scale. Smartplate, is a countertop device that works with a specially designed plate/tray combo that works with an app. The app weighs and identifies your food, and then tracks your dietary intake. The plate uses built-in cameras to take photos of the food and matches the contents to a database that lives in the cloud. The results from a search can be saved to a personal journal or synched to other devices such as a smartwatch.

The application of visual search into the core of the smart kitchen will depend on how easily the individual pieces (and who owns them) of this puzzle fit together. Marrying multiple technologies, image database, hardware and delivery mechanism are just a few of the elements that need to come together. Even Amazon, who owns much of the necessary components, won’t be able to crack the visual search code alone. The speed that valuable partnerships can be built will dictate how this market fares moving forward.

MIT’s Pic2recipe Uses AI To Match Photos to Recipes

The smart folks at MIT have come up with a new food-technology AI breakthrough that has some interesting applications when it comes to meal prep and curating recipes. Called Pic2recipe, the system identifies recipes based on images delivering a list of possible matches and their ingredients. While it is a long way from launch, Pic2recipe not only has value as a standalone app. It could also become a key component of guided cooking systems.

Pic2recipe uses computer vision, a technology long associated with video search. Using a dataset—in this case recipes from All Recipes and Food.com—images are annotated with information about the picture that could then be identified and correlated to the matching recipe. To accomplish this final step, a neural network is trained to find patterns and make connections between pictures and recipes. At this point, Pic2recipe has a database of more than one million recipes.

“In computer vision, food is mostly neglected because we don’t have the large-scale datasets needed to make predictions,” MIT’s Yusuf Aytar said in a recent interview. “But seemingly useless photos on social media can actually provide valuable insight into health habits and dietary preferences.”

A test of what looks to be a beta version of the Pic2recipe system was hit and miss. This enhanced search engine was unable to correctly match a salad and a vegetable stew to the correct recipe. On a third try, this a picture of homemade minestrone soup, Pic2recipe fared far better. Beef minestrone was fourth on the list with a 79% degree of confidence.

Moving forward, it’s easy to imagine Pic2recipe as an app that could feed into a guided recipe system. By showing the recipe ingredients, a user can make any modification necessary for a special diet. For example, the minestrone soup identified in the trial is actually vegan; the recipe shown in the search results can easily be modified by substituting ingredients. So, if you are in an Italian restaurant and snap a pic of your eggplant parmesan, once you make the recipe match, you can cut down on the salt and substitute a lower-fat cheese to make the dish healthier. Send that recipe to your guided cooking system and you are on your way to preparing it at home.

“This could potentially help people figure out what’s in their food when they don’t have explicit nutritional information,” Nick Hynes graduate student at MIT adds. “For example, if you know what ingredients went into a dish but not the amount, you can take a photo, enter the ingredients, and run the model to find a similar recipe with known quantities, and then use that information to approximate your own meal.”