Smart Toaster

When I read things like this, I just shake my head. You can’t evaluate tomorrow’s concepts by yesterday’s tasks! Of course a smart toaster doesn’t need apps. A horse doesn’t need wheels either. We just need to look at what smart means in a more nuanced way. The goal of this post is to convince you that the SmartToaster™ actually is a bold new vision of the future. If I can unpack what it means to be smart, so that even a lowly toaster starts to make  sense, we’ll be thinking about the IoT in a much broader context.

The Classic Vision
The classic vision the internet of things is a fairly complex cascade of activity triggered by an event. For example, when I pull into my driveway, my garage door opens, the lights turn on, a status update is sent to my family (“Dad’s home!”), the house has been preemptively warmed up and the music in my car is seamlessly transferred to my home stereo. This is a great vision but it also implies some very deep cooperation and standardization. We’ll get there over time but I’m still living in a world where my bluetooth phone has trouble talking to my bluetooth car stereo.

My point is that we shouldn’t jump straight into the most complex vision of the future. It’s confusing, excessively difficult, and can lead to a backlash when we can’t do it all at once. Small can be beautiful. Let’s look a bit deeper at my smart toaster and figure out how to peel back this grand vision. Are there layers of functionality that are easier to obtain but still provide value? I’m going to talk about three: discovery, control, and coordination. Let’s take a look at each and how they have value.

Layer 1: Discovery
This first layer is simple, very simple. At this level, a smart device just broadcasts a URL, wirelessly, to any nearby smart display. By smart display I mean a phone, a tablet, a smart TV or even Google Glass. This would allow me to walk into my kitchen, open my phone and ‘see’ that my toaster was offering up a URL. When I click on this device name, I just go to that URL in the browser. It’s that simple.

Now, to be honest, just broadcasting a top level website is pretty boring, more marketing than real consumer value. But if that URL were to contain my model number, I could be directed to a very useful support site. Even further, if the URL contained my serial number, then things could get really interesting as everything about my device, and my interaction with it, could be recorded. It could completely change customer support (and registration, and online instruction, and recycling, and….)

Companies would kill for this. They already put URLs and even QR Codes all over their product packaging. To have this extend into the life of a product is extremely useful. The problem is that they can’t. Well, some companies do, it’s just very expensive: it’s called writing a native app. But this isn’t just expensive, it doesn’t scale: no customer is going to have an app for every product in their house. If we could make the broadcast of a URL cheap to add to a product, and make it so any device could discover it, it would create a new ‘web race’ where every device could take you to their web site.

This would be the first, simplest form of the internet of things: things on the internet. It would succeed because it is universal, like the internet itself it. It also would start off a a bit assuming, a bunch of Black & Decker products pointing to instructional videos. But once the ecosystem takes off, it will start to create a linkage between anything in the real world that wants to take you to something on the web. Museums, Airport kiosks, bus stops, kids toys, and even dog collars could all sprout very useful web sites that could then in turn be a platform more much higher level services.

Layer 2: Control
Once you’ve taken that step, it’s not that much harder to add a few dollars to your product cost to wirelessly control the device. This is where people tend to misunderstand as they expect the functionality of smart devices to be on par with iPhones.

If my smart toaster had only a single, trivial function: to change my ‘toast done’ sound to a range of different sounds, from soothing zen tones to goofy kid friendly slapstick, that would be huge! Yes, in the world of toasters, adding a few additional dollars to its production cost is indeed a big deal. There is no question this is a risk, but I also know that there are many parents that would spend a lot more for a crazy, kid pleasing toaster. The Nest home heating system has proven high end devices that break outside of the norm can indeed attract larger price points.

If this extra ‘toast done’ sound feature doubled the cost of the product, it would be a terribly bad idea. However, as the cost of system on a chip devices fall, the cost of this addition will fall as well, becoming trivial over time. The trick is going to be finding the right value/price point for these early control layer products. It will get much easier as the price falls. One daring company needs to take the first scary jump and validate the approach. But whomever succeeds will be immediately followed by the rest.

Layer 3: Coordination
At this level, things start to get much harder. The previous two layers were pretty much solo endeavors. The only shared mechanism was a universal form of discovery. In fact, every smart device today, from Nest, to Sonos, to Withings smart scale ‘go solo’ by writing their own apps. The value of discovery/control is already a well known value for the trail blazer companies. The problem is that the only vialble way forward is to write an app. If we could remove that barrier, many more products would be able to sprout interactivity.

However, once we enter the realm of cooperating devices, the level of complexity rises quickly. My toaster is now part of a broader home system that can:

• log sensor information to a common data store
• store that information in a common format
• respond to a range of shared commands that all devices support

These abilities are what most people associate with swarms of IoT devices: everything works together to share data and functionality. The difficulty with this level is not so much in making it happen technically but socially. In order for these scenarios to play out, companies need to cooperate and agree on certain APIs and communication protocols to get things to play nice.

We have a terrible track record of device cooperation today.  I can’t even get my TV and BlueRay player to communicate over an HDMI video cable, and this is after a decade of standardization effort. Right now, the big companies are trying to create self protecting ecosystems, attempting to crowd out competitors. But things are changing rapidly. A slew of new, scrappy Kickstarter/Indigogo projects are going to rally around a self forming standard. The tiny mammals will out race the powerful dinosaurs. Once something gains traction, it will build quickly.

This coordination effort will also likely to have deconstructed layers of its own, pushing data up and control down but that’s a separate blog post. I very much want this third coordination layer to arrive but I’m realistic we have lots of experimentation and agreement to get through first. Fortunately, this is already happening with cooperating companies like Spark Devices, SmartThings, pinocc.io, and many more.

Conclusion
The internet of things is a big complex beast and we’re in love with the most complex version of it, the coordination layer. While this layer is the most impressive of the three, we shouldn’t ignore the fairly low hanging fruit that comes from the first two: discovery and control. The discovery layer creates an entirely new way to link the real world to the web, lowering the cost so any company can easily connect their device to the internet without building an app. This is the first and easiest step. The control layer just takes it a bit further and allows you to connect and control the device. This is harder only because it is more expensive per device but that will fall quickly. Both of these layers are still fairly easy because they’re not beholden to anyone: you can control your device in any way you see fit, using any commands you want.

The only thing we have to build is the discovery system that needs to look for these URLs. This will open up an entire ecosystem of interactivity, one which, like the early internet, start off modestly but would grow into a platform to allow all sorts of products to flourish.

I’m on sabbatical now and I’m working with others on how to write this open source app. If you have any comments or suggestions, please feel free to contact me.

At the end of 2012, Gartner puts the IoT just before the Peak of Inflated Expectations and I agree: right now it’s all about potential and crazy projections. People are extrapolating both amazing possibilities as well as equally scary “spying on your kids” scenarios. This is classic early thinking for any new technology, exaggerating both the values and the perils as we slowly, collectively winnow wheat from chaff.

Smartphone symbiosis
What I find surprising is that we’ve been talking about the IoT, in some form, for nearly 20 years. Even after all this time, it is still only on the early part of this curve; a bit like Sisyphus, constantly trudging up the hill, never making it over the top. Smartphones, however, are a companion technology that appears to be pulling the IoT out of this rut. The most obvious impact smartphones have had is the profound lowering of component costs. Chris Anderson, of Wired, calls this “the peace dividend of the smartphone wars” by driving these costs down so strongly over the last few years, these smart devices of IoT can leverage not only the parts, but also companion technologies like bluetooth, wifi and even software stacks.

But an even bigger impact has been in the symbiotic relationship between smart devices and smartphones. Nearly every smart device today comes with a smartphone app to control and configure it. This has allowed these smart devices to lower their costs even further, removing the need for expensive UX driven components like bitmap displays and most physical controls.

Mobile apps encapsulate my data
But this symbiosis comes at a cost. Smartphone apps create a certain rut of their own and will likely hinder rather than improve the trajectory of IoT. I’ve written about the many issues with mobile apps here so I won’t belabor the point. But mobile apps have also introduced a new data storage model so my Withings bathroom scale sends my data this way but my Nest heating controller sends it over that way. Even if all of these companies would let me export my data (many of them don’t) the point is that merging all of this data together is what could really be powerful.

Here’s one example: I was talking to a ‘smart pill bottle’ maker that used the activation of the device to remotely assure relatives that their parent is not only taking their pills, but actually in the house and active. Great stuff but it was a rather primitive measure and only happened once or twice a day. If his service could get access, for example, to all of the appliance data in the house, it could have a much richer picture with a much higher resolution of how safely active the patient actually was.

But let’s not throw stones at these companies, it’s clear that we are in a Catch-22 of sorts. While most people would agree we need an open system for smart devices, the unfortunate reality is that any product today has no choice: they must go it alone and build their own proprietary silo. Companies like Withings, Nest, Thingworx, SmartThings, and Evrythng are all doing the best they can, but this results in each creating their own systems and storage silos.

I’ve written previously about how I worry that we’ve forgotten the very reasons why the internet was so successful: its very openness encouraged an ecosystem much bigger than anything a single company could have done. Not many remember the ‘bad old days’ of Compuserve fighting with AOL and how each had not only their own, proprietary email system but also their own local content. It’s hard to believe that people even thought that was a good idea yet here we are, doing exactly the same thing with the IoT.

Stormy sky of cranky clouds
These are all good companies and many even have parts of their product open sourced. But simple math shows that any centralized system is doomed: billions of devices can’t go through a single service. While each of these companies is just trying to get a basic system working, in the long run, we all need to get our heads wrapped around what it means to have all these systems work towards a common data store.

The single biggest fallacy I want to blow up is this utopian idea that there is this SINGLE thing called ‘The Cloud’. Each company today reinvents their own cloud. The Cloud as a concept is dead and has been for years: we are living within a stormy sky of cranky clouds, all trying to pretend the others don’t exist.

One Ring can’t rule them all
For me the core issue is about scale and openness. The entire world could never run on a single email client, why would we expect the same of any IoT service? It is mathematically naive to think one company will unify the entire world of IoT, yet that is exactly what so many companies are attempting.  You could argue that Facebook is actually just pulling off this type of single service for the world but that misses 2 critical issues: Scale and speed. Facebook maybe able to handle 1-2 billion users but we’re talking about trillions of devices that need low latency communication that would prefer not having a single point of failure. The IoT is a type of communication orders of magnitude more complex.

I’m talking about internet grade, redundant, and even federated openness that lets me choose which service I want to use. You are free to choose a different service but all of our devices work with our selected cloud service. We’ve done something similar before, with email, federating an API that lets many different companies offer an email service, yet still interact with everyone else.

Of course, whenever I talk about this, I almost immediately get a pragmatic eye roll and a slow head shake with comments like:

“Scott, Scott, Scott…. You clearly don’t understand how business works. How could any company open up their users’ data? A company has created a great app and it’s capturing that data, it’s theirs! To give it away would be naive, inviting a race to the bottom.”

To be fair, there is a certain amount of truth to this. It’s a tough world out there and companies look for every advantage they can but there are two reasons why this is approach is short sighted.

1. It’s not their data, it’s mine.
If I use your product and it creates data that represents my life, I want to control it. For you to think this data belongs to your company is an ethical mirage cast by technical constraints. Just because your product is implemented to upload my data to your servers doesn’t make it yours.

The main reason people put up with this is that there is no other choice. The biggest .5% of companies, the Samsungs, the Apples, the Ciscos, all have the motivation and means to attempt a closed ecosystem approach to the IoT.

However, the majority of companies just don’t care to be involved at this level. They just want to jump on the bandwagon. It’s a bit like writing operating systems versus applications. Most companies just want to write an app, almost no one is brazen enough to write an OS. Once we get an open standard to let any company play by these open data rules, the other 99.5%  will create a stampede. The “It’s my data” mantra will become a critical PR issue for companies to grapple with and make their own.

2. It’s always a race to the bottom.
I smile a bit when people think you can avoid a race to the bottom. No industry is immune to this in the long run. Even Apple is running into trouble as smart phones start to surpass the iPhone. It’s innovate or die.  A company that locks up their customers data for competitive advantage is using their customers as human shields.

Progressive companies will get ahead of this curve and realize that the best way forward is to give up ownership of what was never theirs in the first place but offer innovative services on top of that. By letting all of my data, from a variety of devices, be stored together, the overall quality of services will be so much greater.

Two bits of good news
I have no illusions here. What I’m asking for is ambitious: an open data and communication stack for the IoT. We actually don’t have too many positive role models for this type of openness. But keep in mind we were using the web for over a decade before we got fancy AJAX web apps. Let’s cut ourselves some slack and initially start with a thin shim of functionality. There are deep, meaty technical issues to discuss on latency, local vs server processing of messages, and data formats. All thorny issues that won’t be easy to solve. However, let’s make a start with this thin shim, design it with these initial open, federated principles in mind and just get some experience walking down that road.

Recently, two very good bits of news have arrived. The first is that Jawbone, makers of the UP fitness band have opened up a means to sync with Nike+. It’s not a perfect solution but it’s a start, with the VP of Product management calling out for the need for these products to cooperate. The corporate wheels seem to be creaking in this direction.

The second bit of good news is that the folks at Spark are leading the way in this more open systems approach. Their devices have 1) an open RESTful api to control their device 2) an open API to communicate and store data and 3) an open server API. This means that any server can be used. It also means that any device can play. This is a deeply open and redundant system. Are there things missing from Spark? Quite a bit actually! It is just a start, but one founded on the correct, initial approach. I think the world will be watching Spark quite closely as they are experimenting with a fundamentally different business model, one built on top of an open system, not protected by a closed one.

I’m hopeful and encouraged that companies like Jawbone and Spark are starting to move in a more open direction. Each has issues but they understand that we need new models and a fundamentally different model for the IoT to work. My guess is that many companies are beginning to work this way and let’s hope that they start finding each other and building off each others expertise.

Yes, this radically open approach is ambitious. Some might even call it naive. But to think that any single company can have a viable, scalable way forward for the entire IoT seems equally naive.  A stormy sky of cranky clouds is indeed our current, default path forward. But it is not a viable future. Let’s discuss and appreciate that we need an open, federated system and encourage companies and universities that work in this direction.

Over the last year, my writing and speaking has focused on a fairly straight forward thesis:

1. Cheap computation/networking will make nearly any device ‘smart’
2. There will be lots of these things
3. Using ‘an app’ to control each one (what we currently do) just won’t scale
4. Smart phones will be joined by smart TVs, smart glasses, and smart tabletops
5. All the ‘smart devices’ will want to play with all the ‘smart displays’
6. We need to make this happen through open source solutions

I’ve given this talk to thousands of people and it’s so gratifying to get such enthusiastic endorsement. But whenever it comes down to concrete next steps someone almost always says “you really need to sketch out how a company can actually make money doing this”. The intent is good; they are clearly trying to be supportive, but this is exactly the wrong approach.

The world is littered with ‘service discovery’ systems that have come and gone over the years. For the most part, they were all proprietary systems. We’re talking about building a network of nearly everything on the frigg’n planet. Who in their right mind wants to hand off that kind of power to a single company? It is antithetical to the entire history of the internet.

And that, at its core, is what is so frustrating: people have forgotten how the internet was made. It wasn’t built by a company trying to get rich quick. It fact, it wasn’t built with any business goal in mind at all. It was built to create a highway of connectedness that morphed and grew over time into something that allows all sorts of companies to make money. Your city’s roads and streets don’t make money but they sure make it easier for FedEx to do so.

Our brains have collectively gone startup-crazy, seeing the world through stock option colored glasses, assuming that if there is no money, there is clearly no value. This is madness. I’m so desperately worried that the internet will turn out to be a happy accident. This is going to sound a bit trite but the “internet of things” has the word ‘internet’ in it! When are we going to start building it with that word in mind? There are several excellent internet bill of rights already floating around, focusing on the rights we, as citizens of the world, have in relation to this technology. However, we also need need one from the devices’ point of view. Any smart device has the right:

1. To have access to the internet
2. To be discoverable by anyone or anything nearby (without necessarily being on their subnet)
3. To be able to broadcast information on what it does
4. To offer up a web page to do whatever the hell it wants to do
5. To offer up a RESTful interface of actions that it is capable of doing
6. To optionally require a secure connection/login

Is this a company? Not at all. It’s not even a product. I’m sure others will point out what it’s obviously missing. But let’s start simple and work our way up: build an open and discoverable system that looks, and actually is, nearly identical to the internet that brought us here.

Much like TCP, this bill of rights doesn’t have to be tied down to a particular transport. My guess is that wifi direct is going to very helpful but it should just as equally work over ethernet and Bluetooth. We just need to agree as a group that any standard must have these core open principles for the internet of things to take off. I’m doing what I can to push this meme along but just like the internet, this isn’t something a single person can do. We all have to appreciate how we need a deep, open solution to solve this problem. If we don’t understand, demand even, that hardware devices need to be just as discoverable an open as web servers are today, we’ll never see the internet of things come to pass.

However, it’s clear that making that first step is the hard part. How do we get started? Rome wasn’t built in a day. I’m exploring how to build this over Wifi Direct using Arduino. I’d love to hear what else others are working on. Please post in the comments other folks that are working on this. It takes a village…

The Internet of Things (or IoT) is finally going mainstream. Not only do I read about it frequently online, but I’m now talking about it with clients at frog. Unfortunately, as it has become popular, it has also grown to the point where it can span everything from home Wi-Fi networks to smart cities. Much like the story of the three blind men describing an elephant, the essence of IoT depends on your point of view.

What we need is a simple breakdown, both physically and functionally, so we can discuss the many facets and challenges ahead. The simplest is to start by device category as there is a huge range of devices that can be thought of as being part of the Internet of Things. Fortunately, there are really only three broad physical categories: Bears, Bats, and Bees.

Bears
Bears are the big boys, the classic computers with big screens, user-visible operating systems, and many applications to install, launch, organize, and delete. These devices are the classic beneficiary of Moore’s Law: getting faster and more capable each year. Bears aren’t limited to just laptops but include mobile phones, tablets, and even smart TVs. Any device that is multipurpose and primarily driven by a user’s direct interaction is a bear.

To be fair, bears historically have not been a part of the IoT conversation as they are so big and expensive. However, bears are still the dominant computation device: they are our gateway into interactive devices. It’s not uncommon for some homes to be filled with a dozen bear devices, and they are starting to communicate and chatter among themselves. Of course, they’ve done that for years with simple home networks and web surfing; but both of these are, for the most part, about sharing files. Whether you are pushing a file to a printer, pulling a page from a website, or copying a photo to a server, it’s a fairly file-driven process.

What is new with bears is that they are forming into ecosystems or clans, if I can stretch the metaphor. (Technically a group of bears is a sleuth, but I can’t use that with a straight face…) For example, some consumers are now going “all Mac” by purchasing a MacBook, an iPad, an iPhone, and an AppleTV, because they just work better together. It’s not only about files any more but awareness (AirDrop), shared web services (iTunes, and iCloud), and video/audio streaming (AirPlay). Apple has built a series of proprietary protocols that go beyond the standards of the Internet.

But there are other bears out there: Microsoft, Amazon, Google, and Samsung. They are all building their own suites of services, creating their own clans and trying very hard to crowd out the other bears.  Of course, innovation almost always occurs along proprietary lines. It is sometimes necessary for companies to experiment and not be hamstrung by standards and push the boundaries of what is possible.

But will it ever converge? Remember that email was once a series of incompatible islands until it standardized around the SMTP protocol, uniting everyone and creating a significantly larger opportunity. Unfortunately, bears are unlikely to play nice. They want to try a winner take all game and, at least in the short run, will shun any attempts to create a greater shared standard. I chose the term “bear” carefully. Bears are not only territorial but also very big and very grumpy.

Bats
Bats, on the other hand, are small and nimble. They eschew the grand multi-purpose functionality of bears and try to do just one thing well. They are still a bit experimental; but devices like the Nestcontroller, the Withings bathroom scale, or the Glowcaps medicine bottle are all good examples. What defines a bat is its need to be found (usually wirelessly) and interacted with (usually through mobile apps). They don’t care about fairly high-end bear-ish needs of file transfers or video streaming.

Right now these bats are, contrary to the metaphor, a bit solitary, each acting as a standalone device, usually storing information in their own siloed cloud. They are not using “the cloud” but rather “their cloud,” creating a cluttered sky of incompatible data. There will clearly be a need for bats to cluster around a shared cloud storage model. That, unfortunately feels a bit utopian as there are no clear standards to store this type of data in a uniform, sharable way. It’s also a bit out of scope for this article, but it seems inevitable that if a shared data platform existed, it would attract a wide range of bat devices.

Bats are what prompted me to write my “Mobile Apps Must Die ” blog post as their app-for-every-bat approach just isn’t sustainable. As we get more and more bats in the world, the user overhead of app maintenance is going to break down. This is the primary difference between bears and bats: bears carry their interaction within them as they can afford big screens and pointing devices. Bats, being small and inexpensive, need to have it added on by another device.

But we’re not quite done with bats as there is a new form of bat that sheds their solitary outlook, creating a proper swarm of bat-ish devices. This is the “smart home” concept: filled with smart lamps, heating systems, door locks, stereo systems, and video cameras that all work in concert. For example, when I open the door, the lights go on. This raises an entirely different need for bats: the ability to coordinate.

This is a nascent field. Very few products exist now, and those that do are copying bears’ proprietary behavior: they are creating proprietary protocols and custom applications to accomplish this coordination.

Again, innovation and experimentation must take place. I don’t begrudge any company for trying to create their own space. However, proprietary lock-in is so self-defeating in ecosystems like this. There is no way any single company can build every possible smart device for a home. The lessons from the Internet are so easily forgotten: you need to have an open “plumbing system” that allows everyone to play. Proprietary products can be built on top of that, but the more plumbing that is shared, the bigger the pie becomes.

This point is driven home when I discuss my Just-In-Time interaction concept: some feel that my “discover and interaction” model is too simple; it can’t handle the coordination that a swarm of bats needs. But this concern teases out a key point: when talking about the IoT, people immediately jump to coordinated swarms of devices. That is a mistake. The IoT demands a layered solution, exactly like the Internet. We must first have an open means of discovering every device. Second, we need to be able to interact with each one, openly and across any platform. Third, through RESTful API calls, devices can discover and use each other’s functionality. Finally, and only then, can we build services that tie them all together. It’s not rocket science; it’s just the tried and true lessons we’ve learned from the Internet, but this time applied to physical devices, not servers.

Bees
Bees are completely different. They are tiny things that only really have power in swarms. There will likely be many types of bees, but I’ll discuss three here. The first is what I’d call a “worker bee” which is a wireless sensor, usually put in factories to monitor production. This is often referred to as M2M or machine-to-machine, and while this is a huge industrial movement, it’s not frequently discussed in the design community. This type of bee works in swarms as entire systems of sensors are hooked up to monitor large factory processes and can have huge improvements in production monitoring and quality. Even though this is very big business, this crowd has realized the value of open industry standards. Fortunately, not everyone behaves like a bear.

Another type of bee, a “production bee” is any box or package that can be tracked. These objects don’t have any computation within them; they just use tagging technology such as NFC, RFID or even barcodes. These devices, by being both unique and trackable, gain a form of virtual computation. Each time a bee device is scanned, cloud storage is updated so it can have some data added to the cloud. Production bees are really a merging of a real life object and a virtual cloud computer.

Production bees are nothing more than trackable objects that collect data such as where they were made, how they were transported, what temperature they were held at, and when they were used. This was first proposed by Bruce Sterling, in his concept of “ Spimes.” Spimes are objects in space and time. It’s not as much about the object, but its journey: Where has it been and what has it encountered? It collects data like a bee collects pollen, hence this category name. The purpose of spimes is to revolutionize the production process, providing a more efficient and sustainable world economy.

The production bee doesn’t really want you to notice it, it’s just a bag of frozen garlic bread that has a secret history: it is part of an invisible system that works in the background. But there is a final bee type, an “interactive bee,” that is its big brother.

An interactive bee is capable of everything the production bee does but adds interactivity. When I plop that frozen garlic bread on an interactive kitchen table in the near future, a digital window flies out on that surface next to the bread. It’s a webpage on demand within my environment. As interactive surfaces spread throughout my life interactive bees can leap onto them. In the case of garlic bread, it would show not only nutrition information, but also offer a video demonstration of how easy it is to use its “easy open packaging.” Interactive bees clearly could be over the top and must be user driven, but proper use provides a huge extension of the overall customer experience into any physical device. Now using an interactive table is clearly a bit futuristic, but I could have also used Google’s Project Glass goggles instead or if you insist, my mobile phone.

But interactive bees can go beyond even tagged objects. Anything physically stationary, like a bus stop, a tree, or even a statue in a museum can be geo-tagged and gain the same benefits. But instead of walking up to the bus stop and discovering where it was made, I instead find out when the next bus is coming. Similarly, everything in a museum becomes interactive, and parks have a clear history of everything done to their trees. Bees allow any physical object to become a hyperlink into an interactive world.

Conclusion
The Internet of Things is a growing, changing meme. Originally it was meant to invoke a giant swarm of cheap computation across the globe but recently has been morphing and blending, even insinuating, into established product concepts.

Bears are the old guard of computation but are assimilating much of the communication attributes of IoT. Bats are an entirely new category of devices, starting off as solo beasts but slowly, haltingly, turning into an interoperable swarm. Bees on the other hand, are a fascinating flip on the entire problem, virtualizing even the computation within each device.

What is clear from this exploration is that the old school capitalism of monopoly economics is not going to see us through. If every company wants to act like a bear, they win in the short run, but we all lose in the long run. We need to remember that the web is not the internet. The web tends to think in terms of winner take all systems like Facebook. The internet, on the other hand, was a fairly humble and simple means of discovery and access: the plumbing of the digital world that allowed the web, and eventually Facebook, to be built. We have to start thinking in layers. It’s perfectly fine if the very top layers are proprietary; that is not the problem. It’s when companies try to own every layer that things go wrong. We have to break up the concept of the internet of things from a proprietary play into a shared play: one where everyone can enter the playground. If we don’t get our head around this, we’ll be spending the next decade spinning from one tiny playground to the next.

Smart devices require a significant shift in thinking
This blog explores how to design smart devices. But these new devices are just so new and require such new insights, that our quaint, old school notions of UX design are completely blinding us. We are stuck between the classic paradigm of desktop computers, and the futuristic fantasy of smart dust. The world is either fastidious or fantastic. The path ahead is hard to see. Alan Kay said the best way to predict the future is to invent it… but what if we don’t know what we want?

Coffee Maker Syndrome
I’ve long proposed just-in-time interaction as a core approach to smart devices but while presenting on this topic over the past year, it has astounded me that people have such a hard time just thinking about the overall landscape of smart devices. Take for example this tweet:

    Overheard  at  #CES:  “I’m  pretty  sure  my  coffee  maker  doesn’t NEED  apps.

On the face of it, this makes perfect sense. It seems unlikely you’ll be reading your email on your coffee maker. But this dismissive approach is an example of what Jake Dunagan has called “the crackpot realism of the present”. We are so entrenched in our current reality that we dismiss any exploration of new ideas. By stating that apps on a coffee maker would be silly (which is true), we easily dismiss any discussion of other potential visions of functionality.

When television was first introduced, the earliest programs were literally radio scripts read aloud in front of the camera. Radio had been dominant for decades so broadcasters just coasted into TV without thinking creatively about how to approach the medium differently. As Marshall McLuhan said, “We look at the present through a rearview mirror; we walk backwards into the future.”

Smart devices require three big shifts
Assuming that smart devices require apps is like walking backwards into the future. We don’t need our smart devices to run Microsoft office, we just need to them to, say, log their electrical usage (internal, invisible functionality) or give us quick how-to videos (simple user facing functionality).

If we want to properly discuss how to design smart devices, we must appreciate how they shift away from standard computers in three significant ways: micro functionality, liberated interaction, and a clustered ecosystem.

Shift 1: Micro Functionality
In my last post I discussed a fundamental UX axiom that Value must be greater than Pain. This handy little axiom implies many useful theorems. The most radical is that if pain gets very low, the actual value can also be low. While expensive tablets demand significant functional apps, cheap processing allows for more humble micro functionality. It’s one of the biggest hurdles that people have in discussing smart devices. They are so entrenched in the PC paradigm that they assume every device with a CPU must be bristling with functionality.

However, simple doesn’t equate to useless. For example, whenever I offer up the possibility of a ‘smart toaster’ people often chuckle; it’s the coffee maker syndrome all over. But there are lots of small and even fun things a toaster could do: log it’s electrical usage, offer up an instructional video on how to clean the crumb tray, report any diagnostic errors, call the support line for you, or even tweak it’s ‘toast is done’ sound. All of these are fairly trivial but are still useful if a) the value is genuine and b) the cost of adding the functionality is small. $600 tablets must to do quite a bit but this isn’t true for a $40 toaster.

The biggest impact of micro functionality is in how little real interactive power is required. So often when I talk of using HTML5 as the lingua franca of smart devices, people trot out the ‘it can’t do X’ argument, extolling the superiority of native apps. But micro functionality is so basic and simple that HTML5 is more than adequate: you’ll normally only need to view or change a simple value. Micro functionality only requires micro expressivity.

Shift 2: Liberated Interaction
Remember that Value must be > Pain. Micro functionality requires micro pain to be viable. No one is going to argue with their toaster; this type of functionality has to be quick, fast, and easy. Unfortunately, the trend today is that any device with functionality will usually have a tiny display, tinier buttons, a complex user manual, and a tech support line.

Smart devices need to be liberated from being solely responsible for all interaction. I’ve written previously about just-in-time interaction which allows any smart display (a phone, tablet, TV, interactive goggles, and yes, a laptop) to interact with a smart device. Using a significantly more capable device is so much better than cobbling together a cheap LCD display with fiddly little buttons on the device itself. A generation raised on rich phone interaction will expect, even demand better.

Moving interaction to smart displays also has a huge benefit for manufacturers. The cost of computation will likely be the least of a manufacturer’s concerns. Small displays, buttons, complex instruction manuals, and tech support lines are all very expensive. What if manufacturers could assume that any smart device they built would have free access to a big interactive color screen? Not only that but it would have excellent computing power, smooth animated graphics, a robust programming environment and to top it off a universally accepted consumer interaction model that didn’t require any training? Using these displays would allow enormous cost reductions, not only in parts costs, but in simpler development costs as well.

Shift 3: A Clustered Ecosystem
Once we liberate the interaction from the device, we’ve unlocked its functionality across many locations. Not only can I use my phone in front of my thermostat but also with my TV across the room and with my laptop at work across the city. By liberating functionality from devices, we liberate our ability to use these devices from anywhere. My previous post was called “Mobile Apps must die” not because apps will actually die (they will always have a role) but the shortsighted desire to funnel functionality exclusively through them must stop. If these very simple apps are written in HTML5, they can be used across any device which is very powerful indeed.

Conclusion
It is inevitable that devices will ship with interactivity built in. But as more devices become functional, it’s going to become overwhelming to have each device be it’s own island. The three shifts discussed here: micro functionality, liberated interaction, and a clustered ecosystem all point to a new pattern of thinking: small devices with small functionality that all work together in profound ways. This is a triumph of the mundane; a challenge to our PC soaked way of thinking.

But this new approach requires an open standard that all devices would use to announce their presence and offer their functionality in a universal language like HTML. In many ways we are at the cusp of a new hardware era of discoverability much like the web first had in the 80s.

What’s holding smart devices back is our oh-so-human ability to misunderstand their potential. These three shifts are a big step in understanding what we need to do. Let’s be clear, this is not technically challenging! We just need to understand what we want.  Alan Kay is right: we have to invent our future. frog, where I work, is just starting to build simple prototypes to validate these concepts. As they mature, I’ll be sharing more information about them. It’s clear that technology is not the limiting factor, it’s just our desire to imagine a different future.

I’ve written previously that the history of mobile has been a long, painful process of copying desktop computers and then sheepishly realizing that it just doesn’t quite work right. This is actually the way of all progress, not just in technology. Art and music follow a similar pattern of copy, extend, and finally, discovery of a new form. It takes a while to shed old paradigms. 

Mobile apps are clearly successful and in some cases, very profitable. For me to say they MUST die appears to fly in the face of overwhelming evidence. But all things come and go, especially so in fast paced world of technology. When a paradigm shift occurs it’s rarely because the old model is hated or even useless, it just can’t take advantage of new opportunities. The old guard clings to their ways, angrily shouting that everything is perfectly fine, you’re exaggerating!

Too much trouble
The problem with apps, and by this I mean native apps that must be downloaded to your phone, is that they are just becoming too much trouble to organize and maintain. It’s just not realistic to have an app for every store you go to, every product you own and every website you visit. This creates an ever increasing set that must be curated, organized and culled. It’s a common task we all perform, removing old and unused apps every few months, effectively garbage collecting our phones. Very organized folks relish the opportunity to tidy their burgeoning app menagerie but most can’t bother and their home pages scroll into a receding haze of choices.

By itself, this issue clearly doesn’t doom apps, but it does show an ever increasing problem. What would happen if we wanted to have twice as many apps, or 50 times as many? Can we keep this up?

This is reminiscent of the early days of the web when Yahoo had a fixed hierarchy of websites that became more and more difficult to keep up as the web exploded. Google’s approach completely avoided this problem by removing any organization and using only search. This allowed you to have access to literally millions of pages easily and quickly. Google broke the established paradigm.

The UX golden rule: Value > Pain
There is a subtle force at work here. When discussing any product, technology, or idea, it’s easy to focus only on its value, what problem it is trying to solve for the user. This is a good start, and has historically been the only consideration. Recently however, people have started to realize that it also has to be well designed; it can’t be painful to use.

These two primary aspects of any product, it’s value and it’s pain are usually treated as independent variables. Of course you want a high value product and just as importantly the pain of use must be low as well. What people don’t appreciate is that these two are intricately linked. What’s most important is their relationship: as long as value is greater than pain, you’ll be ok.

For example, early SMS systems were crazy hard to use but the value, avoiding expensive minute charges, was so high the value transcended that pain. Of course, improvements in the SMS experience vastly increased usage and pulled in even more users. Just because value > pain doesn’t mean that you’re done, it just means that it’s good enough to ship.

But this same equation explains another, even more important, user behavior. As pain goes down, people will use a product more often for less valuable tasks. Value is still > pain but now it takes much less value to trigger usage. The classic example here would be doing a google search. Google has publicly said that if they just reduce the load speed of the Google.com home page by TENTHS of a second, usage noticeably improves. The Google.com home page doesn’t change in any user perceivable way, it’s just a tiny bit faster, yet people use it more. This is important as only reducing pain, not improving the value of a product in any way, can significantly affect usage.

So let’s bring this back to using native apps. If you were to walk into a store and they proudly proclaimed on the door that they had an app, would you immediately install it? What value/pain calculation would you perform in your head before going through the trouble? If you were a big fan of that brand, then the perceived value is likely high and you’d likely go through the pain of the installation; value > pain. However, if you’d never heard of the store, you likely wouldn’t be bothered, value < pain so you’re not willing to take the risk. My point is that if somehow the app magically appeared on your phone as you walked in the door would you be more likely to try it? Of course you would because there was no pain, it went to zero. You’ll try the app because you’ve got nothing to lose. We’ve got to figure out a way to make trying apps painless.

For native apps, pain is >> 0…
Making the user responsible for app management is effectively inflicting a steadily increasing amount of pain upon them. This puts a increasing pressure on apps and they are going to be used less and less often. It’s not an absolute issue, but rather a negative example of Google’s home page usage. By making things slowly more complex and cumbersome, usage will start to fall bit by bit, hardly measurable at first but likely to increase over time.

I can still understand if you are unmoved. You may be thinking, “Really, how many apps do you need? The problem can’t get THAT much worse can it?” Anyone in the technology space has to always keep in mind that nothing is stable. What seems completely reasonable today becomes a prison in a very short period of time. I’m sure most of you remember that the original PC DOS team expected that no one could possibly use more than 640K of RAM…

And it can get worse…
What is likely to drastically change our currently tiny world of apps is the plummeting cost of computing and connectivity. This will significantly increase the type and number of devices that we will encounter every day. In fact, it’s likely that we’ll pass hundreds if not thousands of devices that will each be capable of offering me some type of interaction. There is more detail in my Zombie Apocalypse post but the cost of a smart device is dropping and in many cases going to zero. Here are a few examples:

• Movie posters with radio tags such as RFID or NFC will allow me to get an interactive version of the poster on my phone to show me more information
• Any consumer item, such as ketchup or milk bottles, also with radio tags, will allow me to not only get more information on these items just like the poster, but also track usage and even offer to purchase replacement items
• My local bus stop will be geo-located so all I need is my current GPS fix and I can get just the information for that specific bus stop, knowing when the next bus will arrive. While this is possible today with some fancy urban systems, deploying a geo location system allows any city to do this, across all bus lines much more cheaply.
• Any store will have an app that I can interact with as I walk through their door
• Shopping malls will offer maps and hours whenever I’m there
• A local food cart vendor will offer not only their menu but where they are going next and when they’ll return
• An on demand rental car company, such as Zipcar, will allow me to register and drive away with one of their cars, just using a bluetooth connection on my phone.

Just-in-time interaction
All of these concepts are of course just speculation but they represent a trend that is thundering down upon us. Each of these devices will likely need some form of interaction but only as I approach them, a “Just in time” interaction model that gives me interactivity but only when I need it. More importantly, the vast majority of these interactions will be a one shot experience. I’ll interact with the device, like a poster, for literally seconds and then move on. It’s a ‘use it and lose it’ situation. What ever I’ve pulled down to my phone to interact with that device is no longer needed.

This is why this interaction will most likely be some sort of web page. It’s a simple way to pull down an interactive experience, on nearly any device, in a way that does not involve installation. The web is uniquely suited to a ‘use it and lose it’ approach. In fact, it’s been doing just that for over 20 years. For those that claim the web isn’t as capable to native apps, that is indeed true. But keep in mind that 1) the standard is improving very rapidly and 2) interaction with these small, cheap devices is likely to be quite modest, you won’t need the capabilities of World of Warcraft to interact with a bus stop.

There is no way that I’m going to be able to or even desire to try this type of just-in-time interaction with our current application model today. The energy involved in finding, downloading, using, and most importantly, throwing away an app is just far too great. The reason mobile apps must die is that it is a paradigm that is holding us back. The whole concept of just-in-time interaction is structurally impossible with installed apps.

Discovery service
Quickly opening and interacting with smart devices isn’t possible unless you can find that device in front of you quickly. To effortlessly interact with the cluster of devices I’ll pass by throughout my day, I’ll need a service that is constantly searching, using the bluetooth, NFC, GPS, and Wifi capabilities of my phone to not only find, but also rank, the devices nearby. I’m not looking for needle in a haystack, I’m looking for a needle in a needle stack. This will likely need some help from cloud servers that will know a bit more about me to enable a reasonable ranking of these devices.

I feel very strongly that this type of discovery service will be the next Google in a few years time. It’s something that Google, Apple, and Microsoft are all equipped to tackle right now. It could even be attempted by a clever startup. The idea of a system that finds and ranks the physical devices around me now is a nearly inevitable service.

What happens when I click on each nearby device would be to simply open a web page, served either on that device or more likely on a central server. What actually happens is entirely up to that device, I really don’t care.  Just like there are very few restrictions on native apps today, there should be very few for these smart devices. The purpose of this system is to just identify and offer just-in-time functionality.

Going forward
Native apps are a remnant of the Jurassic period of computer history, a local maximum that is holding us back. The combination of a discovery service and just-in-time interaction is a powerful interaction model that native apps can’t begin to offer.

No one is close to offering anything like this today. In fact, making this happen is likely going to be a long, slow process. But if we continue to worship native islands of siloed functionality, we aren’t even going to know this is possible. You have to know what you want before you can build it.

Some innovations transcend short term competitive advantage
Since capitalism and design are, for the most part, governed by market forces, there’s symmetry between them. Capitalism tends to be most effective at its lowest levels, meeting demand through efficient supply. Companies that do that well succeed and those that don’t fail. The same is true for design: At its “lowest levels,” a clean look, a simple flow, and an elegant layout are fairly well understood and valued. Products that reflect these low levels succeed and those that don’t fail.

This isn’t foolproof of course. Just as capitalism doesn’t guarantee all companies will be profitable, there is no guarantee good design will always happen either. What I’m saying is that there are clear market forces operating for basic UX design, but only at these “lower levels” of design.

However, it is at the opposite end, when we get into the “higher levels,” that things get sticky. For capitalism, monopolies warp our comfy view of “the market,” which prompts contentious laws to pass. The parallel with design happens when proprietary technologies gain a broad audience. When companies get control of a market and have a lock on users, it is in their financial and—dare I say it—capitalist best interests to use that market power to make a profit.  This cannot only lock out choice, but can prevent innovation. It’s at moments like this that the quiet capitalist in me transforms into a design communist.

Why this matters to mobile
Like many of my neologisms, don’t take “design communism” too literally: I’m certainly not asking for state control of technology. We need to stop assuming that pure market forces will blissfully take us towards an innovative future. We need a deeper understanding of where we are going in order to encourage more innovation. Raw capitalism is fantastic at improving quality and driving down costs but it also rewards a monopolistic lock-in that tends to discourage companies from playing nice with others.

The entire purpose of this “Beyond Mobile” blog is to discuss and explore unlocking the future of smart devices. I feel strongly that this is being held back by a number of forces, but a critical one is being able to liberate web applications on handsets. I’ve discussed this in greater detail in my App Myopia post where I make the case that providing a discovery service to all smart devices around me, through HTML, is a fundamental shift that can unlock huge potential.

We’ve seen big shifts like this in the past. The standardization of railroad lines and universal container ships were broad efforts that reduced proprietary lock-in, but revolutionized their respective industries. Mobile web technology is in a very similar situation. It is not an alternative to native apps, but a basic, core technology that works on many levels. It’s a lingua franca of functional expression that can work not only on multiple devices but even as a service between multiple applications.

The mobile web matters for the simple reason that it doesn’t require an install process. This also means that users don’t need to manage the storage of that app: once used, it can quickly fade away with no uninstall necessary.

Companies like Apple and Google appear to be locked into a race of application dominance. They are like the grand railroad barons of the 1800s, arguing over whose gauge rail is best. Yes, they both also have excellent browsers, but we are in need of deeper innovation in mobile web technologies. They don’t appear to see this as in their best interest to push it forward.

The patient always stops bleeding
So what’s the alternative? The evolving HTML5 standard is clearly a good start. Unfortunately, it is a slow, plodding beast. It may be attempting to create a set of non-proprietary solutions, but damn, it feels like it’ll never get there at times.

Medics in World War II had a rather grim saying about field injuries: “the patient always stops bleeding” (whether they survived or not was a secondary point). In many ways, capitalism is the same way; companies always stop using proprietary technology. Design communism is about pushing hard on companies to be more enlightened, realizing that there are issues that, on the surface may not give you lock in, but raises the playing field so everyone wins.

The Browser Ghetto
I will understand if you think I’m a bit crazy. Apple, after all, embraced the CSS3 standard aggressively on the iPhone. In fact, the first iPhone didn’t initially have native applications, it was all web based. How can I possibly claim they don’t care about mobile web? To this all I can say is of course they like the mobile web, but they like it in a tightly little fenced in ghetto, a browser ghetto.

The Browser Ghetto is effectively a subpar OS with it’s own incompatible user conventions. What is needed is a deep integration of mobile technologies into the OS of a phone, done in a way that works across multiple platforms. This is where my design communism comes into play. Both Apple and Google seem unlikely to consider this. Their fear is that doing so would cede much of their app market away, turning their handsets into commodities. A naive form of capitalism dictates that they keep control and not embrace any approach that gives away their lock in.

But this is shortsighted and ultimately counterproductive. The mobile web, by its very consensus-based approach, will always lag what a native app can do. Besides, why does it have to be a winner-take-all argument in the first place? Native and web apps actually complement each other. All I’m asking for is that the mobile web doesn’t have both hands tied behind its back. Many others have discussed what’s needed to improve HTML5 and JavaScript. These are critical ‘within the window’ technologies that are well in hand. But let me give you three examples that are more about integration with the mobile handset:

1. Basic Window parity
Mobile web pages live within a completely parallel universe of windows. If I have a web game open and want to switch to my contacts app, that is fairly trivial on any mobile phone. But if the reverse is true, it’s a completely different experience. First, there is no identity of my app anywhere; I have to switch to my ‘browser’ first. Why should I care what technology I’m using? Imagine if switching apps on a Mac made you choose “Cocoa” first (Cocoa is a basic mac technology). Once I’m in the browser, and if I’m *lucky*, the web app I want will be front most. Otherwise I have to open the browser’s menu to find the subset of browser windows that will get me to the web app I want.

What do I want instead? I want mobile apps to be first class citizens to native apps. If I leave a mobile web version of Angry Birds, when I switch back, I should see Angry Birds in my list of choices. I’m a bit shocked that I even have to argue for such a basic feature, or that others don’t see this as a profoundly ‘ghetto-izing’ aspect of living within the browser.

By the way, I understand that classic web browsing creates a slew of windows and I’m not asking that ALL of these windows ‘pollute’ my application switcher. But as soon as a web page wishes to be considered an app (most likely by using some combination of HTML5 technologies) it should graduate to a full-fledged application window.

2. Background processing
I worked on the GoogleTalk mobile web app while I was at Google. It was doomed from the start as it only worked when the app was front most. Again, there is a ray of hope here, in that web workersprovides a technical means to peel off background processing but it is hobbled by the fact that it only works when the browser is open. It is locked within the browser ghetto. Web Workers need to run even when the browser is closed, on par with any OS background process.

There are many good reasons why this hasn’t been done yet. There are some who even claim it is a security risk. While it might be true today, is giving a web app access to background processing *so* different from a native one? If so, what can we do to make sure they are the same? It’s hard for me to believe we can’t solve this so web apps can have the same background processing that native applications do.

3. Access to all sensors
This is a much trickier issue and one that, by the nature of technology, will always be changing as new sensors and capabilities will likely be added to mobile devices. Fortunately, there is a strong community in the mobile space working on this and this will eventually get addressed, albeit in stages.

The reason I include this point is political. This *will* be solved in the near future by the HTML5 community, but how quickly will the mobile handset companies support it? I’m actually taking aim at Google Android here as it has a fairly decent mobile browser but it’s horrifically slow at even catching up with where Apple was 2 years ago (e.g. hardware accelerated CSS3 transforms). Having a spec is one thing, but fully embracing it is entirely different. Slow adoption by handset manufacturers just prolongs the browser ghetto.

But will these changes ever happen?
The mobile web community is a crazy, amazing, determined bunch and they are fervently working on making the mobile web reach it’s potential. In many ways this community feels a bit like barbarians at the gate. What will hold this crowd back isn’t their energy but the foot dragging that we’ve already seen in this industry. The Browser Ghetto isn’t an accident, its more ‘abuse by neglect’ on the part of Apple and Google. They’ll get around to it, but only eventually.

Design communism is about pushing this conversation, making it clear that there can be alternatives. It’s not always about competition or lock in. A rising tide can indeed raise all boats. Can we push this discussion forward? Can we get web technologies built deeply into mobile handsets to create this common platform? This is where I feel capitalism can actually save the day, as some handset manufacturer will eventually turn this into a competitive advantage.

Moving beyond the desktop towards just-in-time interaction

Default Thinking comes up frequently when discussing technology, but a particularly virulent form of it has taken hold in mobile: App Myopia. This is a paradigm that sees every possible mobile opportunity only as an exercise in creating an app. This is a rather useful myopia, to be sure, as some people are making lots of money selling apps, but it is beginning to feel like a local maximum and a paradigm that can only get us so far. As Thomas Kuhn might say, we are in need of a revolution.

This approach blinds us to other ways of looking at the deeper potential of mobile. As the number of apps we use grows, there clearly will come a tipping point where we’ll just drown within the task of finding, downloading, and managing applications. It is just not practical to have an app for every store we visit, every company we deal with, every entertainment outlet we patronize, and every product we have. The sheer weight of user responsibly will create a negative feedback loop and people will just refuse to be bothered. In some ways, it is similar to the shift from Yahoo’s original hierarchical list of web sites to Google’s search model: as the number gets high enough, a list of anything becomes overwhelming.

So far, I’ve just described the pain that App Myopia will eventually, inexorably, bring. In my previous post, I talked about an “opportunistic cluster” of hundreds of smart devices. The pain of App Myopia makes these types of devices nearly impossible to interact with easily as each new device would require it’s own unique app to discover and interact with them. This new world of smart devices represents a huge opportunity but they require new interaction patterns around information, functionality and even distributed intelligence that can’t be addressed only using the old-school application model.

Example 1: 10,000 bus stop apps
Right now, if I want to see when a bus will arrive nearby, I need to get the city bus app (native or web based, it doesn’t matter). Then, I have a fairly complex task to find not only the bus line I want to take (if I even know the correct one), but also which particular stop I’m at. Even the best-designed apps will require significant effort and understanding to do this.

In my opportunistic cluster model, the bus stop I am standing in front of *is* the app. I open my phone and I’m looking at what *this* bus stop has to offer. It’s the purest form of progressive disclosure: it shows me the immediate, obvious information needed, but with some small bit of functionality near the bottom for the full ‘city bus app’ experience. This is the complete opposite of the current app experience today.

Example 2: My personal bottle of ketchup
Today, through bar code scanning, photo recognition, and soon, RFID tags, my phone can recognize the brand of ketchup I’m holding, allowing me to do a web search on it. This is cool, but severely limited as every single bottle registers exactly the same way. When I look up this bottle, I’m seeing the platonic ideal, not *my* bottle. A clever mix of history stored on my phone and cheap sensors in the bottle will create an experience for just this particular bottle: how many times it’s been used, when it’s been used, and even by whom it’s been used, which will create a data cloud that is unique to this particular bottle of ketchup. You may recognize this as a spime created by Bruce Sterling.

Example 3: Trouble beeper
My last example is a simple device, a tiny disc that I attach to a wall where I put my car keys. I’ve configured it with my morning commute. It only has one job: to know when I’m passing by. When I leave the house for work, it notices my passing and agreeably glows green, indicating that my commute appears unencumbered. This will quickly become a routine part of my day and I’ll hardly even register this green glow. The disc will, however, glow red and beep if there are any potential problems, such as a traffic jam or a delayed train. It only grabs my attention when there is a problem.

Most of us are creatures of habit with very repetitive parts of our day, and we really don’t want surprises. Instead of building an app that I have to consult to determine if my commute is ok, I have a device that only talks to me when it is not. I’ll need to pull out my phone and figure something out when this occurs, but that will rarely happen. This completely turns the interaction model of an app on its head.

Just-in-time interaction
Tying interaction to a specific device becomes, in effect, my personal gateway into precise, targeted, functionality. There is something very powerful about the relationship between a specific object and myself: it contextually unlocks information and functionality that is normally too complex to get through a classic application.  Apps can still exist, of course, but they are the old school, heavy lifters I go to when I really want to roll up my sleeves and work. In a sea of smart devices, I will pass by hundreds of devices a day, ignoring the vast majority. But when I do choose to interact with them, I want just-in-time interaction, exactly what I need from that object, at that time.

This conversation is not utopian. Primitive versions of these examples are being built today. What is holding us back right now is the mobile phone’s inability to properly act as a navigator through this opportunistic cluster of devices. We’re forced to use barcode scanners, cameras, or soon NFC chips to cobble together clunky approximations of these visions. Here are the types of advances in mobile phones that would unlock this potential:

Advance 1: A Discovery service
Phones today have lots of sensors built into them but no service to tie them all together. There needs to be a service between the phone and the cloud that offers a ranked list of devices and information sources nearby. This would include geo-tagged objects (such as every bus stop in a city, but there’s no reason this can’t extend to every tree in a park), nearby RFID tags, low energy Bluetooth devices, and narrow function Wi-Fi devices such as bathroom scales. This will be an ever-changing list of technologies, which is why it’s so critical that a service exists to act as buffer. As new technologies appear, our phones just see more stuff.

This vision isn’t as nearly as difficult to achieve as you might think. First, there doesn’t need to be a monolithic phone or web service by single company, several can exist at once and compete. Second, any company that wanted to put their geo-tagged data into a service would just need to publish their feed on the web once in a standardized format. Any cloud service that wanted to play could then crawl the data and offer it up. The RFID, Bluetooth, and Wi-Fi ranking would most likely reside on the phone itself and could be cleverly augmented by cloud services. Signal strength should also play into the ranking so the device near me would be ranked higher than the one 40 feet away (a common problem in Bluetooth pairing today)

Keep in mind that only devices that want to be found, such as bus stops and posters, would be ranked. This service would not be used for personal devices such as mobile phones.

Advance 2: A Common Interaction language
Each of these devices would need to have their local information and functionality displayed in a universal form. HTML5 is the obvious candidate here. It can’t be a native app because they would have to run on every device on the planet. As these information services are, at least initially, a bit simpler than classic applications today, it’s possible to imagine their requirements being less demanding. A tremendous amount of functionality could be unlocked by using just what’s available today in HTML5, let alone what will be coming over the next few years.

A huge side benefit to using HTML5 is how easy it would be to embed this functionality in existing desktop web services. Any web mapping service today could show these geo tagged services (e.g. both fixed stops bus stops and moving buses) so you could look at the approaching bus from your office computer just as easily as your phone.

Advance 3: OS level access to these nearby devices
If I need to launch an app to browse and interact with what’s nearby, I’m not much better off. There needs to be a type of radar built into the phone’s home screen experience. Speed and simplicity are key to exposing devices that are nearby and allowing me to see and choose them with as little trouble as possible.

Of course, there will likely be a range of newer, more focused devices that could offer a similar discovery service (watches and digital projection eye glasses come to mind). Mobile phones are just the tools we’re comfortable with today and the best place to experiment.

Conclusion
The history of mobile phones has been a long slow process of copying what works on the desktop and then sheepishly realizing that it just doesn’t quite work right. App Myopia is one of the final bad habits we need to abandon. Just-in-time interaction is a new opportunity and represents a style of interaction that reinvents basic interaction styles between devices. The ideas in this paper are a tiny start in this direction. My purpose isn’t to predict a full list of future products as much as to tear down old models so we can start to build these breakthrough products. We can’t see the future if we can’t let go of the past.

Image from Ignacio Leonardi

I assume most readers are familiar with the concept of cloud computing, but it’s a very broad concept encompassing a wide range of technologies. This article will focus on a core aspect, the storage of a user’s data outside of their personal devices. This is a very disruptive shift that enables user experiences that would be impossible with only local storage, and creates a new facet of design: theUX of data.

Breaking Down “User Experience”

At first glance, it seems rather silly that data would have a user experience. Part of the problem is in how we understand the term “user experience.” In my book The Simplicity Shift, I wrote about how the term “user experience” is too imprecise. It can be used to describe everything from deep technology trends to graphical design. When people ask for a “simple UX,” there are just too many ways this can be interpreted.

We can understand user experience better by breaking it up into three broad levels: the presentation,task, and infrastructure layers. This is a broad taxonomy to be sure but it allows us to have a more nuanced conversation.

The presentation layer

The presentation layer is the visual/graphic layer—how the application looks, the choice of color schemes, use of whitespace, and the visual design language. This is a critical aspect of the product and usually has the strongest emotional appeal.

The task layer

The task layer is about actions and user models, how the application operates and flows. What are the core concepts the user must understand in order to use the product? A classic example is the difference between the DOS C: prompt and the Macintosh Finder. Both manipulate files but they have vastly different task layers: remember and type, versus click and drag.

The infrastructure layer

The infrastructure layer is the base technology the product it built upon. Is it battery powered or plugged into the wall? Does it use Bluetooth or Wi-Fi? Does the low-level network code return detailed error conditions or just a single, cryptic failure? While these may seem like “just technology issues,” the point is that these choices enable or, more likely, hinder what is possible in the UX. For example, if a portable product has an extremely short battery life, no amount of task layer interaction or presentation layer graphics are going to fix this poor user experience. In this way, infrastructure is the great wet blanket of design, it’s what keeps products from being great and rarely seen as a means to create a revolutionary UX.

Data Is Infrastructure

There’s lot to discuss about these three layers and their roles in the communication of design. However, the key insight from this model is that the most profound impacts on the UX usually come from innovations in the infrastructure layer.

This has been true throughout history as a new disruptive change in technology goes through a predictable set of stages: innovation, execution, and acceptance. Carlotta Perez writes about this in her book, Technological Revolutions and Financial Capital. She shows that it takes time for any new technology to be understood, accepted, and then actively experimented with to find out what it’s ultimately capable of achieving. She documents how this is a sociological phenomenon as much as a technical one. Just because something is possible doesn’t mean it is widespread, and it certainly doesn’t mean that people are willing to use it.

We are in the middle of this social change. Cloud data has been around for quite a bit but we are still tinkering around the edge, not really understanding and accepting what it has to offer. It order to talk about a UX of data, we need first to understand that data storage, cloud-based or otherwise, is a fundamental part of the infrastructure layer, and we need a more robust vocabulary to describe how it impacts UX design.

From File to Cloud

Historically, data was typically stored locally using proprietary formats. This was a profoundly isolating, inefficient approach that required users to store, organize, name, back up, and even manage version control all on their own. Data was so siloed, in fact, that discussing broader issues such as collaborative editing was a bit like the two-dimensional characters in Flatland talking about the three-dimensional spheres. For example, if in the 1990′s if you wanted to create that collaborative word processing application it would involve writing custom server software, using proprietary formats and protocols, and almost certainly only working on a local network. It was tried by a few crazy companies, of course, but for the most part it was a fool’s errand. The infrastructure challenges were so great that acceptable UX wasn’t even practical.

One of the first major applications to break out of local storage and move to the cloud was the Web-based email service Hotmail. This was not just another flavor of the client/server approach. By moving to web servers, email could be read by any computer anywhere, an entirely new usage model.

This move from locally based data to cloud-based data enables many deep new advantages:

Advantage 1: Device independence

This is the fairly obvious and initial win of cloud based storage. You could now use nearly any computing device to access your data. This initially appealed to people who travelled frequently, but the value became even clearer once device prices fell and many people routinely started using multiple computers and smart mobile devices. Unfortunately, this is where most people stop, thinking this is the only significant advantage of cloud data.

Advantage 2: State as data

But what we think of as “data” needs a bit more exploration. For example, with email it’s tempting to think of the data as just headers and a body. But if I have my email inbox open in two browser windows, when I read new messages in the first window, they also become marked as read in the other. Data pertaining to the read/unread state of my inbox is synchronized as well.

This is a fairly subtle point, but a deep one. This meta information about the email is just as much data as the headers and body, and the way this meta information is displayed, changed, and propagated between screens is a critical part of the user experience. We need to move from thinking of data as old-school bits in a file to dynamic display states that need to be shown, manipulated, and updated.

Another example comes from two somewhat related music products,Pandora and Sonos. Pandora is an application that streams music to devices: TVs, phones, laptops. Sonos plays music much the same way as Pandora, but to a distributed set of speakers throughout the home.

These two products have vastly different concepts of state. Pandora treats state as a client-side attribute so if I have three Pandora devices in my home they can only be controlled independently. They are effectively islands from each other. Sonos, on the other had, puts the music state fully into the cloud, so each client controls a single play state for my house. If I pause my music on client 1, and can resume it on client 2.

Neither one is better. They are just different UX approaches. I’d argue that the Sonos model is more sophisticated and shows how state, not just data, is becoming a core aspect of UX design.

Advantage 3: Sharing with people

When data is in the cloud, sharing becomes easier. This is, of course, trivially obvious when it comes to such things as blogs or a YouTube video. But that collaborative word processing application I discussed earlier, which was nearly impossible in the 1990s, is now being done by many companies. This new product type has been enabled by a new form a data.

In fact, the ability to share data so easily has left people drowning in an excess of shared media such as email, wiki pages, Google docs, online todo lists, and other cloud-based data. Companies likeAsana are even being built around the UX of data, finding new, creative ways to share and link data to make it easier to associate, organize, and browse.

Advantage 4: Sharing with computers

Once you have the ability to share your data with other people, it’s just a short jump to share that data with computers. For example, Evernote is a product that allows you to capture information such as text, photos, and web pages and store them all up in the cloud. But Evernote adds a clever new twist: data is also shared and processed by their computers. Evernote takes your photos and automatically scans them for any recognizable words, attaching them as keywords. This means searching for “STOP” will find all uploaded photos of stop signs.

Sharing data with computers lets them modify and augment your data, effectively making it improve over time. Evernote just adds tags, but you can imagine all sorts of other services, from face recognition, to sorting, to related searches; all processing and enhancing your data with no effort on your part.

Advantage 5: Devices as data generators

The next logical step is to move past sharing and have computers, or more likely small devices, actually create the data for you. New products like the Spot, plots your GPS location onto an ever-updating map, is a simple example of this. You are responsible for creating a shared map, such as a four-day hike in Yosemite. This wraps a purpose around the map, putting in key, high-value locations such as start and stop. A Spot device is then tasked with taking care of the fairly mundane task of plotting the user’s location in great detail whenever turned on.

This active data model allows users to delegate, still owning the big picture aspects of the data, but the allowing an array of devices to handle lower-level raw data that that fills out the picture and adds precision. We’ve moved from a model where users create all of the data to one where the data is co-produced with a collection of devices.

Note that what enables this is that the data model, in addition to being shared, it also allows write access by other devices. This data model is encouraging new product concepts that were practically impossible a decade ago.

Conclusion

Too often “the cloud” is discussed only in terms of the technologies used to engineer it. When discussing its impact on UX, most people can only name device independence. This article has discussed how just data storage in the cloud is a deeply enabling shift and listed four additional advantages, and there are certainly more to come. But we need to stop talking only in terms of technology; there is a UX of data, we are only now beginning to discover what it can do. Our first step is to appreciate how important it is and then to break free of our file-based thinking of what data is, and how it should behave.

Small, cheap devices will disrupt our old-school UX assumptions.

The commoditization of smartphone hardware is just the beginning. Plunging prices of integrated “system on a chip” devices, paired with free Linux clones like Android, have enabled not just cheap devices, but cheap cloud-based devices. This has applied to phone products like the Sony Ericsson LiveView, and also to home appliances like the Sonos home music system.

These examples are just the initial, telltale signs of a huge new wave of cheap devices about to invade our lives—a zombie apocalypse of electronics, if you will.

Imagining the potential impact of cheap computing devices isn’t a new idea; Don Norman discussed it in his book, The Invisible Computer, over ten years ago. And the proliferation of cheap devices isn’t a new concept, either; the ubiquitous computing (UbiComp) community has been discussing this even longer.

Taken together, these two trends, represent an interesting challenge, especially for the UX community. There is an apparent gulf between the commercial world of desktop computers and the “smart dust”science fiction visions of UbiComp. Now that we’re actually seeing real, viable, low cost products emerge, it’s as if we can’t really believe it—we’re still stuck in our old school ways of thinking about computing. The desktop paradigm seems to be binding us, preventing us from seeing new ways of working.

This is especially evident with mobile phones, which are definitely not desktop computers. Yet time and again, classic desktop apps are “ported” to mobile phones with little rethinking of how the mobile context would change, sometimes fundamentally, how that application will be used. I call this “Default Thinking,” and have written about it previously in The Inside Text: Social, Cultural and Design Perspectives on SMS. This problem is simple, but pernicious: designers think of new technologies in terms of yesterday’s tasks, failing to clearly see the real potential of the new technologies.

The Dynamo and the Computer

This slow painful change of technology paradigms isn’t new; it was described in 1989 by Paul David in his paper, The Dynamo and the Computer. Dynamo is an old-fashioned term for a very large, powerful electric motor (does anyone remember Iron Man’s nemesis, the Crimson Dynamo?). This large power source had several profound impacts on the industrial revolution.

The first was freedom from geography. Factories no longer needed to depend on rivers to generate their power. But this benefit was limited by the fact that factories had only a single, large dynamo, which required very clever pulley systems to distribute the power throughout the plant, making for a very convoluted workflow. So the next major advance arose from the falling prices and sizes of dynamos, which allowed them to be used in multiple locations throughout the factory without complex distribution mechanisms. This had a liberating effect, enabling the assembly line to finally escape the constraining pulley system. The power could flow around the process, not the other way around. This enabled significant increases in productivity.

In his paper, Paul David used dynamos as an analogy for what was happening in the office: instead of a single mainframe everyone had to work around, desktop computers were flowing into the office, conforming to and aiding the workflows of the modern office.

David’s crazy prediction, put forward In 1989, was that personal computers would allow everyone in an office to have the power of a mainframe at their disposal.

The Dynamo Today

The trend seen by David continues today, with devices getting even smaller and more numerous. In modern homes, there are small computing devices such as laptops, MP3 players, and mobile phones, but also more unexpected types of computerized devices such as smart TVs, stereos, cars systems, home heating controllers and, yes, even refrigerators. But that’s just the beginning. Key chains, credit cards, headsets, head-mounted displays, smart jewelry… hell, even smart blenders and knife sharpeners are not far behind.

These new smart devices are turning our original desktop-PC-as-hub model on its head. People used to choose between a Mac and a PC and then buy hundreds, if not thousands, of dollars of software, becoming locked into a single platform on a single device. But now there’s a different model forming in which people work with multiple devices on the same data, usually through the cloud.

An obvious example of this is email that can be read from both laptops and mobile phones, but it’s also cropping up in “mini-versions” of desktop apps on mobile phone and iPads—e.g., the iPad versions ofKeynote and OmniGraffle.

Pandora pushes this idea even further, allowing users to listen to music on a wide variety of devices. This was brought home to me when my wife started using Pandora on her laptop, and when she got an iPhone, I installed the Pandora app for her. Her first comment about it made me smile: “How do I sync my music?” I just told her to sign in and then, as if by magic, all of her music appeared. She was amazed. It was kind of like buying a refrigerator and having it delivered full of food.

This Pandora story shows how powerful it can be to have an application’s data be in the cloud. But a shift in my wife’s way of thinking happened soon thereafter: while I was researching a new television, she told me to be sure to get one with Pandora. This is when I knew a potential shift was in place; instead of sticking to the of model of choosing the device first and then loading it with software, here was a case where software was not only driving device selection, but also encouraging the use of multiple devices as an integrated family. The world was inverted: our software is now dictating which hardware to buy.

As smart devices continue to increase in power and reduce in cost, size, and power requirements, this dynamo effect will continue, driving more technology around our processes than the other way around.

Devices Are Greater Than Apps

I’d like to be very clear: I’m not saying the current hot mobile apps like Facebook, Instagram, or Angry Birds will all be transformed by these new cheap devices. The zombie apocalypse is showing how new patterns are emerging, ones that will not likely be served by a classic “app on my phone” model.

The fundamental paradigm of the mobile phone app is basically exactly the same as has existed from the dawn of computing time: a single piece of software to install, open and interact with directly, and then put away. But the coming zombie apocalypse is revealing new UX patterns that will start to form from the connections and interactions between these devices. Three new patterns come to mind, but I expect more to form as this unfolds:

1. Fixed cluster

A fixed cluster of devices will come into the home and encourage the use and transference of functionality between them. Pandora is a primitive version of this, and the Sonos sound system is another. Both of these systems stream music to multiple devices within the cluster. Pandora takes a one-at-time model where Sonos is working with more with a “swarm” of devices.

However, both of these examples are just the beginning. Here is an imagined scenario that would push this concept further: while I’m working out, I’m listening to some exotic, crowdsourced playlist on my wireless headphones that is being streamed via my phone. When I get into my car, the music automatically transfers to my car stereo. When I get home, the music pauses while I walk into my house but once I dock my phone, it transfers into the house stereo system, which will play on the speakers nearest to me, and will follow me as I move around the house.

Of course, a fixed cluster can do more than just play music: it could regulate energy usage in the home, synchronize personal data (such as photos between cameras, phones and family), personalize settings for devices based on who is using it, to name just a few examples.

2. Personal cluster

 A second type of pattern will be swarms of devices on people’s bodies that will be able to collect data and collaborate. There is likely to be a two-tier caste system between a relatively small set of smart devices such as a phone, headset, watch, and shoes, alongside a much larger ragtag gaggle of “dumb”RFID devices that represent nearly everything a person is wearing and carrying.

A classic example of this pattern is a “distributed phone,” where the basic communication brick containing the radio, main processor, and storage hide in the user’s pocket while an earpiece, watch, and jewelry work in concert to interact with the user and the central device. A smart watch can show message alerts and a photo caller ID, and subtler “Info-jewelry” could display information ambiently through color changes (e.g., different colors based on number of pending messages). There is even a burgeoning field of person bio monitors that could monitor your condition and report back to your doctor.

But even the “dumb” RFID tags in people’s clothes could be read by one of their smart devices to become part of a larger, personal profile: who people are becomes an amalgam of what they’re wearing. This could be shared by physical proximity or even projected onto Web-based profiles. This applies to more that just clothing; it could involve any number of objects, each having an owner, so “who” you are wearing could be more interesting than what: objects associated with causes, movies, books, or pop artists, could each evoke a special message or buying opportunity. What will the RFID equivalent of a ‘kick me’ sign look like?

3. Opportunistic cluster

The previous two patterns involve fairly stable, known clusters under the user’s control. Another likely pattern will involve the entire world of smart devices that people will pass throughout the day. Bus stops, rental cars, store kiosks, movie posters, and even entire buildings will offer value by allowing people to interact with them.

In this world, the idea of “an app” is ridiculously quaint, a little like Scotty speaking into the Macintosh mouse in Star Trek IV. In a world of millions of smart devices, a UX lingua franca will be a necessity so any user can approach any device and be able to interact with it directly, without downloading a specific app.

In a sea of these cheap devices, we’ll likely need a display on our phones/tablets that lists (and most likely ranks) nearby devices that might be of interest. Selecting any one would allow that device to interact in any way it chooses. In this manner, the classic concept of “an app” will just be available on demand for any device or object a person happens to be in front of. The idea of a user downloading, managing, and launching apps will feel just plain silly.

Some of the cluster devices won’t even need computation to be “smart.” Just geotagging every bus stop in a city would allow people to walk up to any one and interact with it by looking up its exact location through a cloud service, creating, in effect, “websites on demand”. The great benefit of these “dumb points” would be that since they’re nearly free, experiences can be deployed literally anywhere and at great scale.

But it’s not just about quick access; these dumb points explode the classic concept of an app by focusing on the precise node I’m current in front of. I don’t need a ‘city bus app’, I just need the app for this particular bus stop, which shows me, without any interaction, the next 3 buses. The same is true for classic store apps. Instead of firing up the GPS and figuring out which Starbucks I’m in, the on demand page shows, without effort, the one I’m currently standing in, complete with today’s special right there at the top. This approach works not only for my current location but now adds significant depth to any mapping application: I can now ‘peek into’ stores when I zoom into a street, seeing mini pages for each location right in context of the map itself.

Conclusion

Some of these ideas are aggressively forward-looking, but most really aren’t technologically that complex. The very idea of what a device is and how it interacts with other devices has already started to change. Like most exponential trends, it is a subtle one that isn’t really obvious until it is nearly undeniable. The UX community needs to embrace this coming zombie apocalypse not because we need to invent the future, but that our past is holding us back. We’ll only really discover this future if we shed our default thinking of desktop computers.