Close Ad

The Spoon

Daily news and analysis about the food tech revolution

Google

Food Tech News Roundup: Google Groceries, Icelandic Drones, and More “Bleeding” Vegan Burgers

What a week! We just wrapped up our first ever Smart Kitchen Europe event in Dublin. It was a whirlwind few days jam-packed with content and networking as foodtech innovators from across the continent got together to forge the future of food tech. If you’re curious, get to know the winner of our SKS Europe startup showcase, and read up on some smart ktichen news that dropped on the show floor. And of course, if you want to connect with many of the innovators from SKS Europe as well as a whole bunch more in person, make sure you’re at our flagship event in Seattle come October.

But enough about us — here’s a list of the some of the food tech news stories that snagged our interest this week. Perfect for reading over a second cup of coffee while fighting jet lag.

Photo: Naturl’i Foods.

Sainsbury’s to add plant-based “bleeding” burger to meat section
British retail giant Sainsbury’s announced this week that it will debut a “bleeding” plant-based burger in June 27th in 400 locations. In the spirit of the Impossible Burger or Moving Mountains, the patty is meant to look, smell, cook, and taste like a beef burger in an attempt to woo flexitarians who are trying to reduce their meat consumption but don’t want to compromise on flavor. The plant-based burger, made by Danish brand Naturli’ Foods, will be sold alongside its beef counterparts in the meat section. This announcement comes not long after Tesco, another large U.K.-based grocery chain, released the news that it would carry Beyond Burgers in their shops beginning in July of this year.

 

Photographer: Arnaldur Halldorsson/Bloomberg

Iceland amps up drone deliveries in Reykjavik
Aha, Iceland’s largest online marketplace, is slated to expand the number of drones it flies through Reykjavik over the next two years. That’s right, expand — this would be in addition to the limited drone trial they launched last year with Israeli company Flytrex last year.

According to the BBC, the drones could be used to speedily shuttle everything from pizzas to organs destined for transplant — as long as they’re below the maximum weight of 3kg (6lb, 9oz). Their results will no doubt be critical for companies like UberEats, who are also piloting drone delivery programs.

Photo: Clearly Kombucha.

Molson Coors acquires California-based Kombucha brand
Kombucha, a non-alcoholic fermented tea beverage with purported gut health benefits, is not for everyone. But it appears that it’s certainly for Molson Coors, who recently acquired California-based brand Clearly Kombucha. The kombucha brand was founded in 2010 and will become part of Molson Coors’ craft and specialty import division, Tenth and Blake.

According to Grand View Research, the global kombucha market is expected to reach $4.46 billion by 2024. Lately, Molson Coors has been investing in non-alcoholic beverages; last year they purchased a minority stake in Bhakti, a Colorado-based chai tea company.

Photo: MiAlgae

Scottish agtech company raises £500k
MiAlgae, a Scottish company that turns algae into nutrient-dense animal and fish feed, raised £500k ($665k) from backers including Equity Gap, SIB and Edinburgh University’s Old College, reported the U.K. Business Angels Association. The algae is grown using co-products from the Scottish whiskey distillation process (yum), and is high in omega-3 and other nutrients. As it’s made from a byproduct, the feed also has a low environmental footprint, which is critical as our global demand for, and production of, meat and fish continues to grow.

 

Photo: ndb_photo via Flickr.

Google to sell groceries through Home and Assistant platforms in France
Earlier this week Google announced a joint venture with French grocery chain Carrefour. According to Bloomberg, the retailer said that this partnership marks the first time in France that fresh food will be marketed through Google’s platforms. French shoppers will be able to buy grocery products through Google Home and Google Assistant by 2019. This comes a little more than a month after Google unveiled Duplex, which allows Google Assistant to have surprisingly realistic phone conversations to do things like make restaurant reservations. We’ll see if this move can make Google Home/Assistant a competitor with Amazon’s Alexa, who already partnered with U.K. retailers for voice-controlled grocery shopping.

Is Google Maps Adding Food Identification and Ratings?

If you’re the type who takes pictures of your food and uploads them while still at the restaurant, you may soon be getting prompted by Google Maps to identify and rate your meal, according to a story from Ausdroid.

The publication’s description is vague: “When you add a new photo, Maps will ask you to name the dish and present you with a fairly comprehensive list of meals as autocomplete options.” Adding the new photo to what, it isn’t entirely clear.

However, their post does have screenshots showing a picture of a food item which, from the looks of it, has been identified by Google as taken near a restaurant. Google then goes on to ask what the menu item is, as well as your thumbs up or down on that particular dish.

We’ve sent a note to Google asking them to confirm Ausdroid’s report, and will update this story as we hear more.

As Ausdroid writes, on its face, this is another vector for Google to gather information on you to provide better recommendations. In addition to wait times at a particular restaurant, it could also tell you which specific dishes are popular.

This is in line with news out of the recent Google I/O conference, where it was announced that Google Maps will soon be providing users with a host of new features, including a personalized match score for restaurants. A Google rep told Travel + Leisure that the score indicates “how likely you are to enjoy a food or drink spot based on your unique preferences.”

But, as is often the case with Google, there is a larger data play here for the company. Having potentially millions of food images properly labeled is immensely useful for Google’s AI and machine learning algorithms. Additionally, having pictures taken under all sorts of conditions (different angles, poorly lit, etc.) and then labeled also helps Google better understand the real world better.

Restaurants seem to be of particular interest for Google. At that same I/O conference, the company trotted out a demo of its human-sounding virtual assistant, Duplex, which supposedly is able to make restaurant reservations on your behalf (though doubts about its full capabilities have arisen). Everyone eats, so there are ample opportunities for data collection, and restaurants can be at the center for a number of Google products: Maps, Calendars, Email, Chat, site hosting/web page creation.

Have you seen this ratings feature in the wild? If so, send us a screenshot!

Hey Restaurant Workers, What do you Think of Google Duplex?

Google stunned audiences last week as its remarkably human-sounding artificial intelligence software, Duplex, held actual conversations with presumably unknowing humans at a hair salon and a restaurant to make an appointment. Then, after the initial wave of amazement, the backlash began as people worried about the ethical implications of a human unwittingly being duped into interacting with Google’s virtual assistant.

I’m a firm believer that two things can be true at the same time. Yes, there are definite concerns surrounding the widespread deployment of such technology (the artificial “umms” to dupe the listener seem a little over the top), and I’m having a harder time thinking that the sky is falling.

Part of this stems from the fact that Google Duplex will be undoubtedly be used by restaurants as well. The software is still in development, and the only demo was around it being a (virtual) personal assistant for the customer. But Google is just a big data gobbler, and they aren’t going to only participate in half of this interaction, especially if there’s money to be made (and data to be had) in the other half.

As Jenn Marston wrote last week, almost 90 percent of consumers want to interact with businesses by text — sidestepping voice interaction altogether. And services like Guestfriend allow any restaurant to easily create their own chatbot to answer the same basic questions most customers ask, so automated assistance is going to become more mainstream for restaurant owners.

I assume that at some point, my virtual assistant will chat with the restaurant’s virtual host in some Matrix-like setting that may or may not end up in a gun-fueled kung-fu battle to get my 7:00 p.m. table.

On our internal Slack channel, The Spoon publisher Mike Wolf chimed in on this potential virtual assistant arms race, writing:

Today it’s pretty easy to get a seat at a restaurant, or even a decent restaurant. But what if people used AI-bots to do a high volume of calls/searches to make sure they have the best seat at the best time at the best restaurants? Or to get tickets for the local concert? Are we essentially arming people to remove any inefficiencies in a world? And does that leave those who don’t have access or the knowhow to employ a taskbot to pick up the scraps or eating at worse times or at worse places or getting whatever leftover appts there are at the styllist?

My point is: eventually the efficiency of automated taskbots will force others to use taskbots. The supplier of the service could create an impediment to this by requiring a “verified human” is making the actual engagement.

To get a more AI-focused perspective, I reached out to former colleague, Derrick Harris, AI expert and creator of the Architecht newsletter. He emailed the following, “My concerns probably mirror the ethical ones that other people have about forcing low-wage workers to try communicating with bots because rich people are too lazy/important to call.” He went on, “as we live more of our lives shopping online and not even making our own phone calls, do we look at people in the service industry as just things there to perform a service as part of digital transaction?”

But Derrick, Mike and I are all technology writers, so I talked with two people in the restaurant industry (a manager and a more corporate person) and asked them how they’d feel if they had to interact with a virtual AI-assistant. I know this is a very small sample size, and they didn’t want to go on the record, but their responses were illuminating.

The manager would be fine interacting with a virtual assistant who called in, as long as it didn’t take more time than talking to a human. If the AI couldn’t understand responses, or took to long, that would be frustrating and probably lead to the manager hanging up. They didn’t think that the assistant would be able to handle the more complex aspects of taking a reservation, like handling questions about seat preference and food allergies. (From a technology perspective, this doesn’t seem that hard to overcome.)

The more corporate person was from a different country. While their English was fluent, there is a noticeable accent, which, they said has caused audible frustration with people on the other end of the phone. The idea of having a virtual entity that could be “understood” more easily for basic tasks like booking a table was exciting.

The issue of AI in our daily lives isn’t going away, and it’s something we’ll have to grapple with as it improves. But it’s one thing for me, as a guy who only writes about food and technology to wax on about it, I’m more interested in hearing from the folks who will have to deal with it. With that in mind, if you’re in the restaurant industry, I’d love to know what you think about the idea of interacting with AI assistants rather than people. Do you hate the idea? Love it? Don’t care? Leave us a comment below!

Silent Hack Could Send Malicious Instructions to Virtual Assistants

As virtual assistants such as Alexa, Google Assistant and Siri become more popular, so too will they become a more popular target for hackers. The New York Times reports that researchers in China and the U.S. have discovered a way to surreptitiously activate and command those virtual assistants by broadcasting instructions that are inaudible to the human ear.

From that NYT story:

A group of students from University of California, Berkeley and Georgetown University showed in 2016 that they could hide commands in white noise played over loudspeakers and through YouTube videos to get smart devices to turn on airplane mode or open a website.

This month, some of those Berkeley researchers published a research paper that went further, saying they could embed commands directly into recordings of music or spoken text. So while a human listener hears someone talking or an orchestra playing, Amazon’s Echo speaker might hear an instruction to add something to your shopping list.

Hackers might not care about your shopping list, but considering 41.4 percent of smart speakers are in the kitchen, it’s important to consider whether they could be used to turn on an oven while you’re out, or secretly start up a video call.

You should read the full Times article for a nice dive into the world of AI, speech recognition and modern hacking techniques. For our purposes here at The Spoon, these security notices are good to be aware of as companies look to use food as a way to get further into — and control more parts of — our homes.

Consider that Amazon, which is behind Alexa, wants to use a series of connected locks and cameras to allow deliveries into your home or car while you’re not there. Or that Google just this week announced more native control over kitchen appliances and a strikingly human sounding AI that can make calls on your behalf to set up appointments. And shortly after launching last year, a WiFi exploit created a security hole in Amazon Key’s connected camera setup.

Now before you toss your Echo into the ocean, you should know that there’s no evidence that this type of attack has ever happened outside the lab. But that won’t stop hackers from trying and improving their techniques.

The point is that it will obviously be incumbent upon companies to be as proactive and transparent as possible in determining and thwarting these type of attacks. But if we as consumers want to turn over more control over our lives to these virtual assistants, then it’s up to us as well to stay vigilant and educated about the new threats their use brings.

Smart Kitchen Startup SideChef Plucks Execs From Anova & Vitamix

SideChef, a smart kitchen app and software platform startup, is growing its executive ranks.

I learned this week that the company is adding execs from both Anova and Vitamix in an expansion of its core team. The company, which is based in Shanghai, China, recently added Anova’s former head of retail, Michael Tankenoff, as head of a new west coast US office focused on business development and strategic partnerships. The company has also hired Ken Zhang, an engineer that had previously worked on Vitamix’s smart kitchen initiatives, to help lead its product management efforts.

I caught up with Tankenoff and SideChef CEO Kevin Yu this week. They explained that the executive expansion is part of a bigger push by SideChef to position the company as a smart kitchen platform and compete with the likes of Innit and Drop.

“I will be focused on continuing to build out [SideChef’s] portfolio of hardware, content, and other partners — as well as help build and scale the marketing arm of the organization,” said Tankenoff. “This comes in conjunction with opening up an office here in San Francisco, which will be the hub for all business development and marketing moving forward.”

These types of moves in a nascent market like the smart kitchen platform space — where everyone knows everyone — are hardly unusual. We saw it with Orange Chef, Whirlpool and Innit, and now we’re seeing with SideChef. While Tankenoff’s departure from Anova is not surprising given the company was acquired last year by Electrolux, the move by Zhang is a bit more intriguing. Zhang worked on the smart kitchen initiative that was led by Vitamix COO Tony Ciepiele, who I also recently learned has left Vitamix as of March to become CEO of toy company Step2 Discovery.

The executive shuffle also comes at a time of increased competition between software startups like SideChef and Innit to provide foundational software platforms for appliance makers to create connected kitchen products, as well as increased focus by big players like Google and Amazon to hook their natural language smart home platforms into the appliance market.

Should be an interesting rest of 2018 in the smart kitchen market.

Google Assistant Will Make Restaurant Reservations, Adds New Controls

Google is holding its big I/O Developer conference this week, where it’s been debuting forthcoming new features, bells and whistles around its products. Google Assistant, the company’s virtual assistant, was no exception, getting a big bell that drew lots of whistles about its conversational capabilities, as well as some new smart home controls.

The Internet has been abuzz after the Google Duplex demo yesterday. Duplex uses artificial intelligence to hold convincing--yet limited--phone conversations on your behalf. CEO Sundar Pichai showed the Google Assistant making a phone call to a restaurant and placing a reservation, and what’s amazing is how human Google Assistant sounds, even inserting “umms” and “uhhs.”

Google Assistant engages with the person at the restaurant to not only select a time, but answer questions, and it even tries to find out what the wait times are on a Wednesday night. The best way to understand it is to watch this video:

Keynote (Google I/O '18)

Pichai said that Duplex was still in development, and didn’t give a release date for its integration into Assistant. And even though this type of interaction only works in restricted situations and under narrow parameters, my first thought was “How quickly can I throw away my Alexa?” While Amazon’s virtual assistant has trouble understanding the most basic questions in our house, Google’s Assistant will soon take over basic tasks like booking a restaurant for me.

And it looks like Google Assistant will be able to take on even more work around my house: the company announced nine new devices types that have native integration with the virtual assistant, including coffee makers, fridges and ovens. This means outside developers can embed Google Assistant controls directly into their products. As CNet explains, by working natively, you can ask Google to just “preheat the oven to 350 degrees,” instead of saying “Hey Google, ask LG to preheat the oven to 350 degrees.”

It’s obvious that Google sees big opportunities in its assistant, tying together many of its products (maps, calendar, etc.) in a truly useful fashion. This plus all the data the company hoovers up every day could allow them to more easily leapfrog over Alexa (and leave Apple further in the dust).

Six Suggestions for Alphabet X’s Agriculture Ambitions

Alphabet X (part of Google family of companies), said that it is exploring ways to use AI to improve food production. During the MIT EmTech Digital event in San Francisco yesterday, Astro Teller, the head of X didn’t provide many details other than they may be looking at combining machine learning with drones and robotics for farming.

X is Alphabet’s ambitious “Moonshot Factory,” whose mission is to take on projects that they “hope could someday make the world a radically better place.” As the MIT Technology Review writes:

To be worthy of X’s attention, a project must fulfill three criteria: it has to potentially solve a problem that affects millions or billions of people; it has to involve an audacious, sci-fi-sounding technology; and there has to be at least a glimmer of hope it’s achievable within five to 10 years.

I understand that there are a lot of smart people working at Alphabet, and X’s mission is to invent and launch these world saving technologies. But we cover the intersection of AI and Ag here at The Spoon, and we’d love to put a few early stage startups on X’s radar.

First, let’s assume X is already looking in its own backyard. Sister Alphabet company, Google Ventures, has put money into companies like Abundant Robotics, Skycatch, and Farmers Business Network. And I’m sure Alphabet isn’t above synergizing across platforms to effectuate paradigm improvements throughout their stack.

But after that bit of corporate introspection, X can look outward, and we’re here to help. These are some startups we’ve covered that are intelligently using artificial intelligence or robots to improve food production and reduce food waste, and could make a radical impact at scale:

AgShift uses computer vision and machine learning to create objective food quality assessments throughout the supply chain. Using just a mobile phone app, inspectors can make nationwide food distribution more efficient and effective.

SomaDetect combines old light scattering detection tech with modern AI to determine the quality of milk and early spotting of Mastitis, a serious udder infection that is common–and deadly–among cows.

Farmstead is a grocery delivery startup, but it’s AI facilitates optimized inventory stocking in stores to decrease over-ordering (and subsequent spoilage and waste) as well as increased purchases from local sources.

ImpactVision applies its machine learning to hyperspectral images of food to assess quality and freshness to generate more transparency in the food chain.

Augean Robotics built the Burro, an autonomous worker robot that can carry food or tools and follow farm workers around with no beacon, remote or GPS required. Burro also maps routes to become a mobile conveyor belt for more efficient harvesting.

Huxley creates “plant vision” for farmers and growers. Through a combination of AI and augmented reality glasses, information about the types of plants and their various stages of growth can be quickly surfaced in a heads up display.

Of course, if X wants a deeper look into the future of food, we’re hosting our Smart Kitchen Summit Europe in June. Advances in AgTech are sure to be on the agenda.

The CES Foodtech & Smart Kitchen Trends Wrapup

Every year upon returning from my annual pilgrimage to Las Vegas, someone always asks me, “what was the big thing at CES this year?”

And this year, just like every year, I struggle to answer the question.

The reason? Because there’s never just one big thing. There are usually many big things.

This is in part because it’s such a massive show, one that’s gotten bigger both in scope and attendance over the years, and it’s hard to easily summarize the trends from nearly every corner of tech. Whether your thing is AI, IoT, VR/AR, cryptocurrencies, robotics, CES had something to make you happy.

Because of the overwhelming amount of news and stuff to see, it’s helpful to go to CES with a focus. For me, this year (and really, the last couple years) that focus was kitchen and food tech.  And because there’s no concentrated area at CES for food or kitchen tech (get with the program, CTA), that means I am usually scanning a bunch of different spaces (smart home, fitness, startups) to find interesting new companies or news.

This post is a wrapup of some of the important trends I saw. If I missed anything big (and I’m sure I did), email me, and I’ll update the post.

Smart Kitchen Platforms Emerge

This year was a coming out for connected kitchen platforms at CES. Whether it was Whirlpool’s big debut of Yummly 2.0 (which Brett Dibkey described to me as “the glue” tying together Whirlpool’s kitchen of the future), or offerings powered by Innit, SideChef or Drop, there’s no doubt we saw the intelligent, conscious kitchen undeniably emerge as a major focus for large appliance makers.

What do I mean? Basically, it’s moving beyond simple connectivity and apps to platforms that connect the cooking, storage, commerce, planning and every other aspect of the kitchen into a holistic system. A kitchen that is aware of the food inside the fridge, one where appliances coordinate to each other to help organize the evening or week’s meal, one in which a variety of intelligent sensors and interfaces make your life easier; it was all there. This is, obviously, a big focus for us here at the Spoon, so expect more on this topic later.

Voice Interfaces Everywhere

Speaking of interfaces, we’re on the third year of “Alexa sure seems like it’s everywhere” at CES, but the first year of “Google finally seems to be taking this seriously”. It was just over a year ago that Google finally introduced its development kits for actions for Google Assistant (its answer to Alexa Skills), and twelve months later we finally see the fruits of the company’s labor. We also saw massive investment in CES as Hey Google was plastered all over Vegas, and they had a particular focus on the kitchen with on-site demos of the kitchen with partners like Innit.

Digital Sensing

Part of the intelligent, conscious kitchen is one that understands the food that is in the fridge and on the plate. Some companies were showing off food image recognition tech, infrared spectrometry, digital noses and water sensors.  Companies like Aryballe showed off their high-end professional sensor but also indicated they were working with appliance makers to build the technology into appliances. After-market players like Smarter were demoing their products to companies like Whirlpool. Expect the concept of a sensing kitchen to become more prevalent this year.

Food Inventory Management

Food waste is a big issue everywhere, and there were companies at CES showing off solutions to help us all better track what food we buy.  Startup Ovie, which I would describe as “Tile for food” was showing off its food tracking/management system, while others like Whirlpool and Samsung were talking about how their fridges can help to manage food inventory.

Water Intelligence

Given that it’s one of the world’s most precious resources, it’s always been a bit of a mystery to me why there hasn’t been more attention paid to using IoT and smart technology to manage our water better.  Mystery solved because now it seems the tech world is paying attention. Belkin finally had a coming out party for its long-gestating Phyn water management system while others like Flo had their home water management system on display. Smaller efforts like that of Lishtot, which help us detect whether water is drinkable, were also on display.

Wireless Power

One of the coolest things about the Smart Kitchen Summit last year was the Wireless Power Consortium had its first public demo of its cordless kitchen technology, which features wireless power for small countertop appliances.  I got an early demo at the WPC booth this year as they showed off wireless power for small appliances from Philips and Haier.

I also saw a cool demo using infrared wireless power form Wi-Charge. The concept here is to put an infrared transmitter in the ceiling (they put it in a light installation in the demo) and then transmit power using infrared to various devices. The Wi-Charge folks said their patented tech is currently only targeted at small portable devices, but I’m intrigued with the possibilities for the kitchen as a potential future opportunity.

Specialized Living

I’ve been writing about the massive opportunity for smart home and kitchen innovation for the aging in place market for the past couple years, so I was happy to see a number of companies focusing on this important area.  Much of the focus was on safety, which obviously applies to kitchen/cooking scenarios, but I can also see how smart assistants, robotics and augmented reality could be applied in living scenarios to help folks with limitations due to age.

Robot Invasion

Robots and process automation were everywhere at CES, ranging from cute social robots like LG’s Cloi to delivery robots to the laundry folding robots. Some, like LG, saw the robot as a natural pairing with the kitchen, while others saw robots as more general purpose assistants for the home.  And while we didn’t see the robot chef at CES this year, I expect we’ll see that probably in the near future.

Humanless Retail

AI-driven point of sale devices and “humanless” markets were big at CES. AIPoly won best of show for the second year in a row, while a Bodega-on-wheels startup Robomart had a huge crowd at its booth for much of the show. More modest efforts like the Qvie were on offer to give Airbnb hosts a way to become even more like micro-hotel operators.

New Cooking Boxes Appliances

One of my predictions for the year was a new generation of cooking boxes. I use the term box because they’re not always ovens (though they can be), but often are like the NXP RF-powered smart defroster. We also saw Hestan on the other side of the country (at KBIS) talk about using precision cooking coupled with gas, a throwback to their Meld days.  There were also lots of folks I met with still operating in stealth that plan to debut their next-gen cooking appliances this year, so stay tuned.

Home As Food Factory

All of a sudden, everyone seems to be interested in home-grow systems, whether it’s the backyard IoT grow box from Grow to the Opcom’s grow walls, there was lots of interesting new home grow systems to see at CES. And while I didn’t see anything like food reactors or much in the way of 3D food printers, I expect CES 2019 to rectify that situation.

Smart Booze

Smart beer appliances, wine serving/preservation devices, and IoT connected wine shelves were plentiful this year. CES also gave many the first peek at the home distilling system from PicoBrew, the PicoStill.

We’ll be watching all these trends this year, so if you want to keep up, make sure to subscribe to our newsletter. Also, you can hear about many of these trends at Smart Kitchen Summit Europe, which is in Dublin on June 12th.

Google Assistant Is Becoming A Guided Cooking Platform. We Talk To The Person Leading The Charge

Back in April, Google’s Emma Persky wrote a post telling the world that Google has been working on a recipe guidance capability for Google Assistant.

At last month’s Smart Kitchen Summit, she gave a little more detail on what exactly the company has been up to.

Persky, who runs point for Google Assistant’s cooking guidance team, told the Spoon’s Allen Weiner how much of the focus has been on building in contextual understanding of recipes.

“We can talk you through step by step how to cook a recipe and answer contextual questions about how to do that,” said Persky. “To do this, we have to build a deep model of what a recipe actually looks like. Have to take the text of a recipe and understand that text so we know what pasta is, we know what type of pasta you’re talking about, we know what the units are, we know the cooking temperature.”

“On the other side, we know what the user is saying. We have a whole bunch of machinery at Google that is able to understand what a user is saying and turn that into a machine question. We have a whole bunch of data about how different people ask these questions, which we use to build a model and understand these types of questions.”

When asked about how Google is utilizing company competencies like search and YouTube, Persky said that while there’s been significant work done here, there’s an opportunity to get better.

“We do a pretty good job now when you ask on your phone or desktop ‘how do I sauté an onion?’ we show you a nice video of how to sauté an onion.”

But, she said, “there’s a lot of opportunity for guided cooking feature to more deeply integrate with this, so when your recipe says sauté the onion, and you don’t know how to sauté the onion, we are able to return these types of video answers on on Google Home platform to help you become a better chef over time.”

Persky also discussed how she thought web content schemas could evolve to create a foundation for richer content through platforms like Google Assistant.

“When it comes to companies that have this kind (recipe) data available to them, there are a lot of opportunity for finding ways to increase the fidelity of the data that we have access to. At the moment we have schema.org markup, which is a good first pass. We don’t have a lot of detail and use machine learning to extract a lot of the context from that. And I think where there’s an opportunity to where a lot of people working on this stuff is to find ways to access more high fidelity data that we could offer to the users as an improved experience.”

When asked by Weiner how schema.org and other web markup languages could improve, she had this to say: “There’s a lot of work we can do to improve the quality of that markup. For example, right now the markup just has one block of text for all the instructions in the recipe, but actually if we could break that down and have a step by step, it would be easier for us to parse that out. Right now we have to apply machine learning across that to do that.”

It’s a really good conversation to understand what Google has been up to as they look to combine recipe content with their voice AI platform. You can watch the full conversation between Allen Weiner and Emma Persky below:

Ed note: Answers in this interview have been edited slightly for brevity/clarity

Exclusive: Nest Working On “Smart Chime System” With Unique Sounds For Events, People, Zones

While it took a while for Nest to jump into the home security game, it looks like they may just be getting started.

What’s one potential trick up their sleeves?

Smart chimes.

That’s right, the folks within the Nest security team have filed for a patent that was published on November 2nd of this year that would transform a home security’s chime system into one that assigns unique chimes to specific people, locations, and events.

According to the patent filing, Google/Nest is working on a smart chime system that would play unique sounds depending on the location, type of event, whether a person is unidentified or identified, etc.

The description excerpted below describes the shortcomings of a conventional home chime system and how the one envisioned by Google/Nest would be different (bold emphasis added by me):

At least one problem common to conventional door chime systems is that door chimes offer only one type of sound effect for all doors and windows. As a result, the chime does not provide any information as to exactly which door or window was opened. Furthermore, the homeowner does not know if the chime sound indicates a person leaving the house, coming inside the house, just a window opening, or exactly who opened the door or window.

Another problematic feature common to many conventional door chime systems is disabling a chime from being sounded if one door or window is already open. In the scenario in which the front door is open and then a second door is opened, the opening of the second door does not cause the chime effect to be sounded, thus the residents are not informed.

The disclosed smart door chime system remedies these problems and provides many other improvements. The disclosed system can provide a customized chime or other sound based on current data obtained by sensors, historical data obtained by sensors, user input data, and additional factors as will be described below. The disclosed smart door chime system can process and store data that has been captured by sensors and analyze the data to extract information about the environment, such as activity of a person, identify of a person, activity of a pet, motion, etc. Based on the data, the disclosed smart door chime system selects an output profile that determines a specific sound to be played on a specific set of speakers. Accordingly, many different scenarios may be addressed and customized chimes or sounds can inconspicuously convey to users a wide variety of information about occurrences at a premises.

In short, Nest wants to be able to utilize motions sensors, cameras, facial recognition and other technologies to create a highly specific chime system to let the user create highly tailored audio signals to understand better what is going on around the home. Does Fido keep sneaking into the bedroom and eating your slippers? There’s a chime for that. Want to know when your youngest kid has walked out the backdoor? There’s a chime.  A stranger’s entered your home. Chime.

Not only that, it looks like the system could allow for some fun. For example, the patent application describes how you could tee up specific songs or audio files to play.  Spouse home from a long business trip and you want drop some clues for a romantic night at home? Just tee up a little Business Time from Flight of the Conchords when they walk through the front door.

You can see from the image below how the system would allow for tailored “zones” throughout the house:

Zones within the Google/Nest smart chime system

The image below shows how you can create specific rules and zones depending on the event. As you can see, you can play a chosen chimes/audio files in specific locations based on predetermined rules around events:

Rules for Google/Nest smart chime system allows user to tailor rules by event/zone/speaker

I’m not sure about you, but this sounds like a pretty useful feature. I can see how folks with young kids, pets or special needs family members could use a smart chime system to help them monitor the goings on in the home.

One such example would be to use a smart chime system to monitor autistic children. Parents with autistic kids often struggle with their children’s tendency to wander, and one could imagine using special chimes to alert when an autistic child gets near a door.

While this is only a patent application, let’s hope the team at Nest plan to turn the smart chime system into an actual product.

Tim Enwall Thinks A Robot Should Be Our Friend, Not A “Mechanized Piece of Metal”

If you have a robot in your home today, chances are it vacuums your floor or mows your lawn.  And while you may be thankful for the back pain the little guy spares you, I’m willing to bet you don’t consider this industrious single-tasker your friend.

Tim Enwall, the CEO of Misty Robotics, thinks that needs to change.

A robot “can’t be this mechanized piece of metal that runs around the house or office,” said Enwall. “It has to be able to develop a relationship with us.”

I recently had a conversation with Enwall about the future of personal robotics for the Smart Home Show. How we relate to robots is something Enwall thinks a lot about since his company has made it their mission to be the first to create a real world version of Rosie, the famed cartoon robot from the Jetsons.

Enwall outlined four features that Misty CTO and founder Ian Bernstein sees as crucial for a personal robot:

It’s got to be familiar. “It can’t be freaky, it can’t put us off,” said Enwall. He admitted that many of today’s robots have too much uncanny vally-esque creepiness.  “We will get there, but today they’re too off putting.”

It has to develop a relationship with us. This is where Enwall points out robots can’t just be metal cans running around our homes. “That’s not going to be valuable or interesting to us,” said Enwall.

It’s got to be multifunction.  “We can’t go buy 20 single purpose robots for our house or office,” said Enwall. Here Enwall essentially points out today’s world of primarily single purpose robots is not sustainable. In other words, unless we want to add a room to our home for a robot garage, we’re going to need robots that can perform multiple tasks.

It has to be useful. While this one seems related to the last one, it makes sense to break it out.  A multifunction robots that can do many chores is something I would put hard-earned money down for.

Lastly, Enwall added a fifth characteristic he believes important for personal robots, one which essentially makes them human-like:

It has to be able to manipulate things. Here Enwall is basically points out what makes man different from other animals, and that robots need to do the same. I’m assuming Misty’s robots will, at some point, have something resembling opposable thumbs.

I had gotten to know Enwall from his days running smart home startup Revolv before their acquisition by Google, and earlier this year he went over to run Sphero’s new personal robot spinout. He’d gotten to know the founders of Sphero through being a part of the Boulder startup scene, which Enwall has been active part of since his days as CEO of pioneering home energy management startup Tendril.

Overall it was a fun and interesting conversation. Take a listen below, download here, or head over to Apple podcasts and subscribe.

Rubin: Essential Home Will Be The “Bridge” Between Competing Smart Home Ecosystems

Last night Andy Rubin got on stage with Walt Mossberg at the Code Conference to discuss his new company.

The two spent a good chunk of the conversation talking about the Essential Phone, but when they finally got to the Essential Home, they didn’t disappoint.

I wrote yesterday about what we already knew about the Essential Home smart home product, but Andy’s discussion with Walt gave us a better understanding of the company’s strategy for the device.

Rubin and his team have (correctly) identified the main problem of the smart home as one of too many competing ecosystems. The main goal of the Essential Home is to solve for that.

Per Rubin: “One of the problems in the home is the UI problem. There are too many things you have to interact with in your home.”

While we often use that term UI to describe the various consumer interaction layers such as voice, touch or motion, Rubin is using the term more broadly here. He points to a fragmented smart home world with too many competing apps, smart home protocols, and technologies. And, as the guy behind Android, Rubin admitted that in many ways he helped create the problem.

“I feel somewhat responsible. One of things Android helped do make really easy to write a mobile app. the guy building your IoT doorbell, he’s going to write an app.”

According to Rubin, the problem with so many apps and technologies is each time a consumer walks through their smart home, they are walking through a series of competing apps and ecosystems.

“In certain ways,” said Rubin “the industry has recognized what the problem is, which is you don’t want to launch someone’s app when you walk up to your front door to unlock it, where they have their own UI, their own login credentials, and when you finally get through front door and its time to turn on your lights, do the same thing with the guy that built your light bulb.”

He’s right in saying the industry knows fragmentation is the main problem in consumer adoption. In our survey of over 100 smart home execs last December, the number one hurdle to adoption of smart home products identified by industry insiders is confusion over too many smart home platforms.

In other words, fragmentation. Or, as Rubin puts it, “a UI problem.”

Rubin said the solution to the problem of UI fragmentation is to bridge all of these competing ecosystems by working to integrate as many of them as possible together.

“You have to think of it as a UI problem,” said Rubin, “and you have to solve the UI for the home as an interoperability and integration issue. You can’t just support ten devices; you have to support one hundred thousand devices.”

That’s a lot of devices, but Rubin plans to get there by bridging the various ecosystems across the world of Apple, Amazon, Google and more. In other words, he doesn’t want to compete with the giants, but instead wants to connect them to one another.

“You can think of this as everyone is creating an island by creating their own ecosystem, so building bridges is the best way to describe what we’re doing. It has to talk to all these ecosystems, whether it’s Smart Things, HomeKit, or Google Home, or Thread or Weave.”

Rubin didn’t go into the specifics of how he plans to solve the fragmentation issue, other than to say they think they’ve found a way to do it. Whatever the approach is, it sounds like one built from the operating system on up with a focus on security.

“We had to build a new operating system so it can speak all those protocols and it can do it security and privately.”

That operating system is called Ambient OS. It will be part of the new Essential Home which is rumored to ship in late summer.

Make sure to subscribe to the Spoon newsletter to get it in your inbox. And don’t forget to check out Smart Kitchen Summit, the first and only event on the future of the connected kitchen and the future of cooking.