Hedge-mazeThe level of hype around the “Internet of Things” (or IoT) is getting a bit out of control. It may be the technology that crashes into Gartner’s trough of disillusionment faster than any other. But that doesn’t mean we can’t  figure things out. Quite the contrary, as the trade press collectively loses its mind over the IoT, I’m spurred on further to delve deeper. In my mind, the biggest barrier we have to making the IoT work comes from us. We are being naive as our overly simplistic understanding of  how we control the IoT is likely going to fail and generate a huge consumer backlash.

But let’s backup just a bit. The Internet of Things is a vast sprawling concept. Most people refer to just the consumer side of things: smart devices for your home and office. This is more precisely referred to as Home Automation but to most folks, that sounds just a bit boring. Nevertheless, when some writer trots out that tired old chestnut:  “My alarm clock turns on my coffee machine!”, that is home automation.

But of course, it’s much more than just coffee machines. Door locks are turning on music, moisture sensors are turning on yard sprinklers, and motion sensors are turning on lights. The entire house will flower into responsive activities, making our lives easier, more secure and even more fun.

However, I am deeply concerned these Home Automation scenarios are too simplistic. As a UX designer, I know how quixotic and down right goofy humans can be. The simple rule-based “if this then that” style scenarios trotted out are doomed to fail. Well, maybe fail is too strong of a word. They won’t fail as in a “face plant into burning lava” fail. In fact, I’ll admit that they might even work 90% of the time. To many people that may seem fine, but just try using a voice recognition system with a 10% failure rate. It’s the small mistakes that will drive you crazy.

I’m reminded of one of the key learnings of the artificial intelligence (or AI) community. It was called Moravec’s Paradox:

It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.

Moravec’s paradox created two types of AI problems: HardEasy and EasyHard.

HardEasy problems were assumed to be very hard to accomplish, such as playing chess. The assumption was that you’d have to replicate human cunning and experience in order to play chess well. It turns out this was completely wrong as a simple brute force approach was able to do quite well. This was a hard problem that turned out to be (relatively) easy.

The EasyHard problem is exactly the opposite: a problem that everyone expects to be simple but turns out to quite hard indeed. The classic example here is language translation. The engineers at the time expected the hardest problem was just finding a big enough dictionary. All you had to do was look up the words just plop them down in perfect order. Obviously, that problem is something we’re still working on today. An EasyHard problem is one which seems simple but never….quite….works…..the….way….you…..want.

I claim that Home automation is an EasyHard problem. The engineer in all of us assumes it is going to be simple: walk in a room, turn on the lights. What’s the big deal? Now, I’ll admit, this rule does indeed work most of the time but here are series of exceptions that break down:

Problem: I walk into the room and my wife is sleeping, turning on the lights wakes her up.
Solution: More sensors: detect someone on the bed.

Problem: I walk into the room and my dog is sleeping on the bed, my room lights don’t turn on
Solution: Better sensors: detect human vs pets

Problem: I walk into the room, my wife is watching TV on the bed. She wants me to hand her a book but as the the room is dark I can’t see it.
Solution:  read my mind

Don’t misunderstand my intentions here. I’m not luddite! I do strongly feel that we are going to eventually get to home automation. My point is that as an EasyHard problem, we don’t treat home automation with the respect it deserves. Just because we can automate our home doesn’t mean we’ll automate it correctly. The real work with home automation isn’t with the IoT connectivity, it’s the control system that will make it do the right thing at the right time.

Let’s take a look at my three scenarios above and discuss how they will impact our eventual solutions to home automation.

1. MORE SENSORS
Almost every scenario today is built on a very fault intolerant structure. A single sensor controls the lights. A single door knob alerts the house I’m coming in. This has the obvious error condition that if that sensor fails, the entire action breaks down. But the second, more likely, case is that it just infers the wrong conclusion. A single motion sensor in my room assumes that I am the only thing that matters, my sleeping wife is a comfort casualty. I can guarantee that as smart homes roll out, saying ‘sorry dear, that shouldn’t have happened’  is going to wear very thin.

The solution of course is to have more sensors that can reason and know how many people are in a room. This isn’t exactly that hard but it will take a lot more work as you need to build up a model of the house, populate it with proxies, creating, in effect a simulation of your home . This will surely come, but it will just take a little time for it to become robust and tolerate of our oh so human capability to act in unexpected ways.

2. BETTER SENSORS
This too should be soon in coming. There are already sensors that can tell the difference from humans and pets, they just aren’t widely used. This will feed into the software simulation of my house, knowing where people, pets and things are throughout the space. This is starting to sound a bit like an AI system, modeling my life and making decisions based on what it thinks is needed at the time. Again, not exactly impossible, but tricky stuff that will, over time, get better and better.

3. READ MY MIND
But at some point we reach a limit. When do you turn on the lights so I can find the book and when do I just muddle through because I don’t want to bother my wife? This is where the software has to have the ‘humility’ to stop and just ask. I discussed this a bit in my UX grid of IoT post: background swarms of smart devices will do as much of the ‘easy stuff’ as they can but will eventually need me to signal intent so they can cascade a complex set of actions that fulfill my goal.

Take the book example again. I walk into the room, the AI detects my wife on the bed. It could even detect the TV is on but still know she is not sleeping. But because it’s not clearly reasonable to turn on the lights to full brightness, it just turns on the low baseboard lighting so I can navigate. So far so good, the automatic system is being helpful but conservative. When I walk up to my wife and she asks for the book, I just have to say “lights” and the system turns the ‘lights on” which could be a complex set of commands turning on 5 different lights at different intensities.

Of it may not be voice commands, they too have issues. A classic button or even a gesture will also work. These ‘intent cliffs” are needed because human interaction is too subtle to be fully encapsulated by an AI. Humans can’t always do it, what makes us think computers can?

SUMMARY
My point here is to emphatically support the idea of home automation. However, the UX designer in me is far too painfully aware that humans are messy, illogical beasts and simplistic if/then rules are going to create a backlash against this technology. It isn’t until we take the coordinated control of these IoT devices seriously that we’ll start building more nuanced and error tolerate systems. They will certainly be simplistic at first but at least we’ll be on the right path. We must create systems that expect us to be human, not punish us for when we are.

20 thoughts on “Home Automation is an EasyHard problem

    • Not exactly. I think we need to break up automated tasks into ‘conservative’ and ‘risky’. The conservative tasks, like lowering the heat or turning baseboard lighting on low should just happen automatically.

      The more risky actions, like turning on a ceiling light full brightness, needs to be done with some type of ‘intent trigger’ like a voice/gesture/switch. Of course, as we get better sensors (or if, for example, it’s very clear you are alone in the house) the need for these intent triggers can be relaxed.

      I’m not trying to paint hard and fast rules, more layout insights to enable better discussion around the issue.

  1. […] Home Automation is an EasyHard Problem by Scott Jenson. I’m a big fan of the Internet of Things and look forward to a more connected future. However, maybe our ideas about what is possible are misguided. In this short piece Scott explains that it’s possible we’re not properly classifying the actual problem at hand,  “[…] humans are messy, illogical beasts and simplistic if/then rules are going to create a backlash against this technology.” […]


  2. culture and philosophy
    the easy hard problem
    By admin, 2014/02/28
    http://jenson.org/easyhard/
    what made the iphone initially so successful? it was its ease of use – its beauty on the outside – the feeling you had some high quakity reliable software-hardware combination.
    tthats exactly what is needed to be successful in any area especially something that messes with your privacy and security.
    tthis is my first blog entry typed on an android device…. wordpress wont even let me create a link… so much for mobilesupport.

    What do you think?

    Click here to cancel reply.

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked * Name *
    Email *
    Website
    Comment You may use these HTML tags and attributes:

  3. In doing product design for nearly 30 years, I’ve  witnessed  several waves of innovation. Some were obviously successful like personal computers and the internet. But others were much less so, such as MultiMedia CDROMs, and open document formats.
    What people don’t appreciate is that great ideas, before they are great, tend to look remarkably similar to stupid ones. Few are willing to take a chance on a new, seemingly crazy product idea. This is why the technology industry tends to be so incremental. Most companies, truth be told, wait for others to make the mistakes.
    Innovation vs. Risk
    Truly disruptive products make most companies nervous because they imply new use cases and pricing models. You can’t calculate ROI when so little is understood. I saw this at frog design where customers would come in, desperate for innovation, wanting to emulate companies like Apple.
    I learned an important lesson at frog: every company wants innovation, but very few want risk. They assumed that innovation somehow comes with certainly. The hard truth is that you can’t have one without the other. No one can look at the history of Apple and say they took safe bets. It was my job to help companies navigate this particularly bipolar tightrope, helping them come to terms with this relationship between risk and innovation. We often said we weren’t in the design business but the design therapy business.
    As such, our job was to get our customers out of their normal, conservative mindset. Incremental thinking and cost reduction are all important aspects of maturing any product but it’s not what’s needed to create breakthrough product concepts.
    Transformative Coevolution
    You don’t however, just flip a switch and get people to start ‘thinking big’. Part of the problem is understanding the product process. Truly transformative products are not “one hit wonders” but a complex journey: over time an interesting coevolution takes place between a product and its consumers. The product enables new consumer activities which, in turn, further inspires new changes to the product. A truly transformative product works, fractal like, splintering into multiple, unseen use cases, creating new markets. These new opportunities inspire variations on the original products that could not be foreseen.
    The iPod has been used to death as a design example, and not without good reason: it represents many lessons in strategic and patient product development. Apple didn’t just stop with the first incarnation of the iPod, it was constantly evolving.  Nearly 4 years after the first generation, it became clear people were using their iPods in very mobile situations, so much so that they were willing to strap nearly half a pound of tech to their arms to go running. Apple responded with the 22 gram iPod Shuffle. It’s very unlikely that Apple envisioned the need for the Shuffle when the first iPod was released. This new need coevolved with the market, creating a feedback loop: a company influencing consumers who in turn influenced the company. Transformative products not only change the consumer but they also change the companies nimble enough to listen.
    Thinking about the Internet of Things
    Unfortunately, people don’t tend to look at the internet of things (IoT) this way. While they are certainly enthusiastic about its prospects, their energy is a bit simplistic, extrapolating fairly naive scenarios about coffee pots turning on automatically or smart alarm clocks that talk to your calendar. But these intellectual fireworks burn bright and fade quickly. There is a tendency in the tech press to jump on the uselessly sensational and completely miss the transformational potential of the mundane.
    When thinking about where we are going, it’s too easy to just extend existing models; solving yesterday’s tasks with tomorrow’s technology. The IoT will make the iPod look like child’s play. It is the ultimate in fractal coevolution in that as it gets used, it will create not only new use cases, but motivate entirely new products.
    To make this point, let’s take home lighting as a tiny example of how this could work. I’m going to make a few predictions but the purpose isn’t to confidently invent the future but to dent the complacency of existing models.
    Let’s start off with the classic use case: adding a motion sensor to a home that turns on the lights as someone enters. My previous post discusses why even this simple scenario is harder than you’d expect. However, the problem is that it barely scratches the surface of what is possible. Once it’s easy to have dozens of sensors in a home and any light can be controllable, the effects are likely to cascade out, transforming not only what ‘a light’ is, but even how lighting can be used. Here are three examples:
    1. Virtual Swarm LampsOnce control becomes virtual we’ll abandon the idea that a “room light” is only a single wall switch hardwired to a ceiling light fixture. We’ll see instead a room filled with many small lights acting as a virtual swarm lamp. One switch (or sensor event) will turn them all on. Additional smart lights can be easily added to this swarm to fill in dark corners with no additional wiring. This will radically change what we even think of as a lamp. One likely consequence will be a shift from large lights to a range of smaller lights that can be scattered about, placed for example, over a bed or a desk. In addition to the number and size of lights changing, the we’ll likely get levels of lighting so strip lighting along the floor would be a secondary swarm that would only be turned on in the middle of the night if you were to get up.
    2. Painting with light
    Once the virtual swarm lamp fragments the idea of a single light source into many, this will encourage other devices to join in. TVs, furniture, picture frames and even luminescent paint can respond to lighting commands. Entering a room at night will not only turn on the strip lights on the floor, but the TV could ambiently glow, doubling as a night light. But as picture frame, appliances, and even furniture become light capable, It will be possible to start ‘painting with light’ and use light as decoration instead of just illumination.
    3. Lighting as material
    But lighting can go even further, it could also become interactive and impart emotion as well as information into a space. The dozens of lights in a virtual swarm lamp could ‘ripple across a room’ when turned on for dramatic effect.  This might even drive the size of lights down to pinpoints of light, scattering a room with hundreds of them. If music is playing, a small subset could animate and pulse to the beat. There could even be simple notifications: animating  when my phone rings in the other room or if someone rings the doorbell.
    Conclusion
    These are of course, just exploratory guesses; a bit of design therapy to get you to think outside the box. But few people discuss the IoT in this manner. They are so consumed with basic automation, that they don’t see how much this ability to communicate will coevolve products into something very different.
    Let’s get past the shallow fireworks that passes for discussion about the IoT. It’s only natural that simple automation of desk lamps and coffee makers is the first task that people assign to the IoT, but it’s solving yesterday’s problems. More importantly, it’s vastly underselling what the IoT is capable of doing. The fractal coevolution of smart, connected devices will change the very products themselves and profoundly change how we will use them.  It is this crazy new world that excites me. It is in understanding this deeper impact that will help us build the real internet of things.
    Share this:TwitterFacebookGoogleGoogle+Scott Jenson

  4. I was recommended this web site by my cousin. I’m not sure whether this post is written by him as no
    one else know such detailed about my trouble. You are incredible!
    Thanks!

  5. […] classic use case: adding a motion sensor to a home that turns on the lights as someone enters. Myprevious post discusses why even this simple scenario is harder than you’d expect. However, the problem is […]

Comments are closed.