Software Everywhere — David Beers on Mobile Computing

Wayne Parrot thinks it's going to be HTML5. His work on WebKit for SWT aims to prove it:

We have implemented a bidirectional bridge that allows you to implement JS objects in Java and to interact with JS objects from Java. This feature has turned out to be very useful as we are leveraging the power of HTML5 technologies in a much more dramatic and efficient manner from Java now. This is a dev strategy that I have wanted to leverage for sometime. I'm so impressed by what is possible that I'll make a controversial claim that if you are a Java UI developer and are not paying attention to the advances in web UI technology then prepare yourself to be blown away in a very short order by much more capable and productive UI developers that understand how to use new killer web UI features on the desktop.

WebKit's growing dominance in the mobile space (iPhone, Android, S60 and now Palm webOS) as well as the desktop make this a pretty intriguing UI option for Java developers—more so than JavaFX, perhaps. I like JavaFX script, but it wasn't the first dynamic scripting language specifically designed to complement Java for the purpose of developing RIAs. That was JavaScript, which already has great mindshare and energy that shows no sign of flagging. One can't help but wonder about JavaFX while the very future of Sun itself seems so uncertain.

Speaking of RIA's, a call-out to Jo Ritter for his recent comment on RMAs—Rich MobileNet Applications. This application model, which spans mobile and the desktop by introducing the stackless Java stack, is finally hitting some mainstream handsets thanks to ProSyst and Sprint. But let's get serious: that can't possibly come to pass until there is an entry for Rich MobileNet Application in Wikipedia, guys! Get on it!

A while back I posted some thoughts about how the mobile user experience could be greatly improved if we paid less attention to applications (I called them silos) and more attention to tasks the users want to perform and the data associated with them (regardless of where that data lives on the device or what app created it). A part of this vision is a move away from traditional GUIs that focus on launching applications to start a task and toward a natural language interface that gives users a faster, more direct way to get things done when they are mobile. Unfortunately, it's all a little abstract until you see an example of what I'm talking about so I really need to get some animated screenshots up here soon. And I may have compounded the confusion by referring to this as the "return of the command line interface" which suggests a blank DOS or Bash console that requires the user to learn a special language and do a lot of typing. Some readers (mostly developers and Mac Quicksilver users) got the vision and saw the opportunity like I do, while others thought I was talking about something that was designed to support power-users and geeks at the expense of newbies. My goal is actually to make the mobile UI a lot more obvious and intuitive, and to cut down on all the clicking and tapping you have to do today to get where you're going on your device. Maybe a better term would have been "text-driven interface"? "Natural language interface?" Bah, sounds pretentious. I don't know.

Anyway, for those who are following this nascent "CLI renaissance" meme (which Don Norman started) here are a few recent links and products that explore the idea:

The CLI is cool again! by Malcolm Lithgow at Smart Dreaming follows up on my original post with a really thoughtful breakdown of the advantages and disadvantages of the CLI vs the GUI. While he is less optimistic than I am about the CLI creating a better mobile experience, I think our difference of opinion mainly hinges on terminology—he's really thinking of the traditional CLI, even to the point of talking about scripting. He does point out that part of what makes a CLI powerful and fast is auto-completion. In the Serenity prototype I've been working on, autocompletion and autofiltering of commands and data records are pervasive, so you are fully prompted at all times. You never need to wonder what you might be able to enter next and usually don't need to enter more than one or two letters before the software completes the word for you (or at least prompts you for the next required information). So, for example, from a visible list of task "verbs" like Call, Read, Play, Message, New, Find etc. the user can type "c" to select Call, see the completed "Call" keyword, and be prompted with their contact list. Entering "jh" would usually be enough to filter that list to, say, "Jeff Hawkins" and initiate the call. Meanwhile the user might be prompted with Hang up, Add caller, Record call, Write note, etc., all accessible with a single keypress if you only have one hand free and don't want to tap these options on the touchscreen. No using the D-pad to traverse the controls on a form or the items in a menu and get the focus on the one you want. Malcolm's post points to some alternative GUIs that aim in some measure to achieve this kind of directness and it's great stuff to ponder for folks who are thinking about what "simplicity" and "ease of use" really mean on a mobile device.

Last week I ran across a good example of the return of the CLI outside the mobile context: Google Calendar's Quick Add feature. Go try it. I wish I didn't have to click the "Quick Add" link to have the command line display, but once it does I can type something like "Movero conference call 2p Mon," hit the Enter key and it will add the appointment with a default duration at 2pm next Monday. I seem to remember a Palm application that worked a little like this (parsing and interpreting which PIM database and which fields the data needed to go into). Drop me a comment if you know it.

Another nice example that shows with some screenshot mockups how CLI can be combined with prompting and autocomplete (again, on the PC): Conservative Design: Command Line Interface at a blog amusingly named "Web 3.0, 6 Bladed razors, 7 Minute Abs."

Finally, Russell Beattie's new Mowser project has a hint of the CLI in his "one box" interface (I like that term better than "command line," by the way). You can type "am Nokia n95" to search for that handset on Amazon—somewhat better than having to select Amazon from a drop-down list as you do in most desktop browsers. But the result is not what you expect. Instead of getting a link to the most relevant listings you get a transcoded-for-mobile version of Amazon's home page, which is too big for your phone, I guess, so it's broken by Mowser into 5 pages. Only when you get to the third page do you find the Amazon search box with "Nokia n95" pre-entered. Click that button and you get your search results.

All right, all right, Mowser's been up for what? 3 days? I'm guessing that in a future iteration Russ will make these commands use the web services that these big sites provide for search. The real point I wanted to make is that there's the potential for a powerful but simple command language here. Mowser's eBay command ("eb") could have some handy ways to extend the search that are eBay-specific. Consider "eb Palm TX PDA notify $100" which would send your phone a text message with a link when an auction that matches the "Palm TX PDA" search string is at $100 or less, say, 2 hrs before closing. Without good prompts to guide you in writing the command this would not be for the casual eBay user, but there are a lot of non-casual eBay users out there that would learn this quickly and use it often. Still, this kind of interaction only gets really good and simple when you get out of the browser and use web widgets or web clipping apps (see yesterday's post) that can prompt you with the commands and parameters to build the query and auto-complete them as you enter a few characters. On the web side of things that's the kind of interaction I'm trying to enable with Serenity.

Ok, now I have just used my vaporware non-product to diss a cool, free new web product from a great guy who we're all glad to see back in the blogosphere after a sad (for us) hiatus. He already has some great ideas packed into Mowser and some good suggestions about how to make your blog work with it to be more accessible to mobile browsers. So hats off to Russell Beattie!

[Response to Carnival of the Mobilists #66]

I like those guys Steve and Rafe over at AllAboutSymbian.com. The quality of the writing on their site is very high and, not surprisingly, Rafe does a great job with this week's Carnival of the Mobilists. There were a couple posts that I'd particularly like to comment on.

M-trends on personalization

I enjoyed reading Andrew Bergland's post about the mobile phone being the new "it" fashion accessory—one that itself can be accessorized in so many ways. The only way I like to personalize my phones is with software, and since I also write that software for a living I'm naturally inclined toward the view that software is the ultimate form of personalization. Andrew convinced me of what I already really knew, but didn't want to admit to myself: except for geeks and businesspeople, it's usually going to be the outside of the phone that gets most people excited, not what's on the inside. I do think the inside matters when it comes to them falling in love over the long term, though. In Don Norman's terms, it's the visceral ("cool!") and reflective ("what it says about me") emotional responses to a personal device that most often make you buy it. But after you begin to use it, it's often the behavioral ("I like using this") response that makes you loyal. It's software more than hardware that makes for brand loyalty in mobile devices today. Look how the Treo has persisted and grown with (distressingly) little change to the form thanks to good software, while the RAZR (notorious for it's lousy UI) flamed out spectacularly once the reflective and visceral attraction wore off.

Funny that the phone software that has elicited the most visceral response in recent memory (the iPhone interface) is also among the least personalizable. Apple seems to want to imprint their image on you, not the other way around, which is fine as long as that image continues to be one people find attractive. Apple should keep the price high to retain a sense of exclusivity about the iPhone, because I don't think its usability is going to be all that good on an everyday basis.

Smart Dreaming on "contextuality" in the mobile UI

Speaking of iPhone usability, the other post I found really interesting is from a new blog that I'll definitely be adding to my regular reading: Smart Dreaming. Malcolm Lithgow has some very thoughtful analysis of "contextuality" in UI design: the importance of reminding the user how the thing they are doing in the software relates to other relevant data. The example he uses of poor contextuality is the common practice (also found in the iPhone) of forcing you to edit an appointment in a modal dialog that blocks your view of the rest of your calendar. Most mobile phones today fail miserably in this respect and I agree with Malcolm that it's a big problem. Drilling down through nested menus that all look pretty much the same is another disorienting experience that covers up both where you've been and where you might be going (if you can figure out how to get there).

There's another kind of context mistake that most smartphone UIs make, as a vestige of their PC heritage. That's the assumption that the "application" must be the arbitor of context for the mobile user. For example, if you want to send a picture to someone by email you are expected to launch an email client and if you want to send it to your PC you should go find another application to help you do that. As I've talked about before, mobile users aren't as inclined to have extended application sessions on their devices as they are at their PC. They are usually more focused on performing one of a relatively small number of tasks and doing it quickly so they can get back to whatever else they were doing. This sometimes means that the best mobile usability is had with the interface that makes the task the context rather than the application. If you're looking at an image on the phone that you might want to send somewhere you should be able to directly act upon that impulse from that screen: "send by email," "send to PC." The context that determines what you should be able to do from that screen shouldn't be the application you are in, it should be the picture itself and the things people typically want to do with pictures.

Ok, the picture example isn't the best one, because a lot of phones don't do too bad a job there, but messaging is a good example of something that's messed up badly on the mobile. The relevant context for messaging is the person you want to send the message to or the content you want to send. Any time you're looking at a contact—not just when you're in the Contacts application, but in the "from" field of an email client or in a chat window, or the call log—you should be able to click on that person's name and pop up options of all the ways to contact them right there: phone, SMS, email, carrier pigeon, etc. Likewise, if you highlight some text in a document or browser window or select a filename in a file browser you should be able to click the physical email button that most smartphones have now and have it open a new email with the selected text in the body or the file already attached. The content creates the context that makes that button click mean "new email message with this content" instead of just "forget what I was just doing and launch my email client."

This kind of contextuality requires a lot of integration between the application and the system software, much more than you have on a PC. Applications need to be able to register at the system level objects that are usually considered to be application-specific, and do this in a way that other applications that use these objects know what actions can be performed against them. Palm OS made some small but important steps away from the traditional application model that have given it an advantage in usability over the years. In my mind, though, none of the mobile OS vendors has taken the idea of contextuality seriously enough to produce really functional mobile computing interfaces.

Can't seem to get to sleep tonight. Surfing around aimlessly I stumbled on another great example of the Command Line Renaissance, as one trackback referred to the concept of my earlier post. A creation of Phil Torrone, MAKEbot is an automated AIM/iChat buddy that you can add to your list and "chat" with to get information. For example you can open a chat with him and type "latest" to get a list of links to the latest HOW TOs in MAKE Magazine. Or type "bb" and get the latest five entries on BoingBoing in your chat window. Use the "ping" command (like "ping digg") and the MAKEbot will IM you each time a new item is published to the feed you entered. Type "help" and, as with all good CLIs, you'll get a listing of all the new commands they've been teaching MAKEbot to understand.

Is it a replacement for graphical user interfaces for reading feeds or searching the web? Of course not.

Is it useful, simple, a little bit addictive the way chat can be? Yes. But could be even a bit better.

I believe the mobile user interface should be harnessing the reflex we've cultivated to interact using text. With all the texting you do on your phone today, what could be more natural than to open up a chat with your phone to find out your next appointment, your wife's location, or your flight status? When responding in text your phone could have a touch of personality that GUI companions like Clippy and Rover never pulled off convincingly: "Your flight boards in 43 minutes. Get the hell out the door... but don't forget my charger!"

If you come across other examples of the Command Line Renaissance that's under way, please drop me a message.

Don Norman has a suggestive and thought-provoking essay about the latest breakthrough in the user interface for computers. It's not virtual reality or heads-up display eye-glasses with retinal I/O. It's the command line interface. He uses the evolution of Internet search engines to make his point:

We navigate the internet by typing phrases into our browsers and invoking our favorite search engine. But more and more, we type in commands, not search items. All the major search engines now allow commands to be typed that directly yield answers without the need to go to an intermediate webpage. Consider these three examples, each for a different search engine.

• Google: the phrase 'define:embodiment' directly provides the definition.
• Yahoo!: the phrase 'time in Nagoya' directly provides the time (3:13 AM Friday, when I tried it last).
• Live.com: the phrase 'cars in China' returns with '15 per 1000 people'.

Even though these three services are called "search engines," in fact they are becoming "answer services" controlled through their command line interfaces.

I've been wanting to write an essay myself about command line UIs because it's an idea I've been working with in the mobile software context for a few months now and I'm quite excited about it. Frankly, I think Don's focus on "answer services" doesn't scratch the surface of the potential. That's not his fault; I think that's mainly because he's responding to what he's seeing happening with the PC user interface right now, where the bigger (and as yet uncaptured) opportunity for the command line is on smart mobile devices.

Let me try to explain why.

There are three UI patterns in the mobile interface that are seriously in need of reconsideration: heirarchical menus, the "application icon checkerboard", and the form. All were borrowed from the PC GUI and transferred without very much thought onto mobile devices. "What's wrong with menus, icons and forms?" you ask. "They're familiar! They're intuitive!" Perhaps, but they're intuitive in the context of the PC desktop, at which people have long, immersive sessions, not the mobile context where people have a task that comes to mind (perhaps prompted by the device itself) and need to get it done with minimal time, effort and attention. Mobile users are out and about doing things, interacting in the real world, and only briefly dipping into the digital world. This suggests a different approach to the mobile UI to support the different mode of interaction.

The PC GUI is designed around the concept of data silos and applications. A data silo (like a file format, a directory, or a database engine) is meant to separate different types of data, while applications are meant to unlock one of the silos so you can get in and do some work on what's inside. Neither data silos nor applications (in the PC sense of the word) are a good thing on a mobile device and it's high time we figure out how to get rid of them. They create indirection and fragmentation of the user experience, which are tolerable while you're seated at your desk but become major stumbling blocks on the mobile. The mobile UI needs to be as direct as possible. Even the best that we have today (Palm OS comes to mind) doesn't come close to being as direct as it should.

You know the drill. You want to send a quick email. You click a button to flip between your "phone" screen to "applications," traverse a field of application icons to get to one you want to drill into (Email) then drill into a menu and traverse its items to "New Message," or traverse a form to a button that does the same thing, then click a couple of things to bring up a pick list that you sift through to find the contact you want, click OK, navigate down through the email form (past the "CC" and "BCC" fields—click, click click) ... we've clicked 15 or 20 times and we haven't even started entering the message yet.

Some operating systems (like Palm OS and Windows Mobile) have support for touchscreens so you can go more directly to the applications or screen elements that you want to interact with, but it's a weak and partial solution and it comes at a price. Now the phone is a two-handed device like your PC, maybe even one that requires you to pull out a fidgety little stylus and put it away again once you go back to using the keypad. Devices like the Treo and BlackBerry have honed and touted their one-handed navigation abilities, which hints that this is the preferred way to use a handset. Either way, we are fighting bad UI and losing—losing the mass market customers if we are selling smartphones or software for them.

My contention is that with the right user interface 70% of what you want to do with a smartphone can be done in three clicks or less, not counting entry of text content if any is required. Almost everything you want to do on a mobile device can be initiated with a verb and a noun: "call Jane," "check mail," "make appointment," "text Bob," "play REM," "read Pikesoft," "price stock," "time Tokyo." A well designed mobile device knows all the verbs it can perform and knows most if not all of the nouns it can perform them on. Furthermore, it knows them the same way we do: by name. Best of all, it knows how to auto-complete these verbs and nouns:

You: "cj"

Device: "Call Jane Doe or Call Jim Beam?"

You: "a" (second letter of Jane's name) or "d" (last name initial)

Device: "Calling Jane Doe... 719-555-1234"

This wouldn't be a secret language. The screen from which you launch every task expects you will enter a verb first and should display a list of available verbs. One or occasionally two letters is enough for the device to know the verb you want from the list. Then it expects a noun. Nouns on this ideal device aren't trapped in silos like on a PC. When they signed up for duty on the phone they broadcasted their availability to all the verbs that they know how to perform, possibly adding a new verb or two that they didn't find already in the system's dictionary. "Play" might prompt the user with nouns like "Doom" or "Bejeweled" and if you had an MP3 player in there you'd also see "Favorites," "Clapton & Cale/It's so Easy," and a list of other recent songs you are likely to want to play again, then "Title search" and "Performer search." You don't care that Doom and Music Player are different applications, and a good mobile UI should be just as indifferent when offering you options of what to "play."

"Adjectives," "adverbs" and "prepositions" would be added to the command for more complex operations. In these cases where an action involves more information the UI would show in natural language the status of the task you are setting up and prompt you for required information or ways you could modify it:

To make an appointment:

You: "ma"

Date and time are required to make an appointment so you're prompted immediately:

Device: (status: "Making appointment") "Tomorrow, This [day of week], Next [day of week], Enter Date"

You: "n" (you're going for "next Wednesday" here)

Device: (status: "Making appointment for next...") "Sun, Mon, Tue, Wed, Thu, Fri, Sat"

You: "w"

Device: (status: "Making appointment for next Weds at...") "Time?"

You: "11a"

etc.

Seeing it all spread out on the page like that it may not look obviously faster and it doesn't give a good impression of the visualizations that are possible, either. (Sorry I don't have animated screenshots that I am ready to show yet.) But the full extent of the input for an 11am appointment with Ed Colligan next Wednesday in conference room 1234 with a notification 15 mins before the appointment might look like this: "manw11awecir1234n15m" See if you can guess what natural language prompts and options were presented to guide you in entering this complex appointment to your calendar. It's not hard to guess when you take it one character at a time and think in natural language.

What happens when you enter a word that the system doesn't understand? It searches, based on the context of the parts of the command it did understand. But it doesn't just return a list of search results—it uses the results to form actual courses of action that it can directly perform. "btm" might be understood as "buy ticket for movie" so the unrecognized movie name that followed would become part of the query to a web service that sells movie tickets for a theater in your area. "tofu junction" would be completely unrecognized, so the software might search first for these characters in local storage (like the wonderful Palm OS "find" feature, the original Google Desktop). Maybe it turns up the result of a Google Maps query you made a few months ago when you last thought about going to Tofu Junction. If not it prompts you to try online services like Google, Amazon, eBay or Flickr. As Don Norman points out, command line interfaces can degrade gracefully when they don't understand something completely. It's actually not so much a "degrading" as an extending of the command language so it can perform queries against a local database or services on the web. The impulse to consider the command line was prompted by my frustration with the tedious GUI of today's smart devices, but it's become apparent to me that the big win is the way a text-based interface naturally extends the capabilities of the device to work with remote data and services. Ordinary graphical user interfaces tend to be far less flexible and powerful because of their explicitness, their emphasis on drill-down, and their insistence that everything must be done by first launching an application. "You want to buy a ticket online? Ok, so first you find your browser application and launch it, then drill down into a menu to find the link to Fandango in your bookmarks, then find the place on the page where you enter the movie you want to see (getting past all the ads and suggestions about other tasks or services you don't care about)..."

There's an obvious objection that probably formed in your mind a few paragraphs ago. It takes form in words like "command lines are ugly," "why have a big color screen if your whole user interface is a textbox and some words on the screen?" or "people want iPhone, not DOS prompt."

Well, yeah.

But who says this has to look like a DOS console? The command line I have in mind sits atop gorgeous dynamic visualizations of tasks being assembled to take the place of tired forms with grids, droplists and buttons. With a simple text box at the top of the screen, the rest of the screen is a wide open canvas for artistic renderings of the state of your interaction with the device, the presentation of requested information, or ambient information about things that are going on in the background that you care about. Significantly, the screen need no longer be owned by an application in this system and limited in the information it presents by the imagination of a single software developer. (Things like browsers and video players are obvious exceptions.) This means the screen background can be open to creative visualizations of anything that the user is interested in even as they go about performing a task: a glow in one part of the screen the intensity and color of which indicates the number and importance of messages in their inbox, the ghosted icons of two favorite blogs softly pulsing in another area to let you know that there are new posts to read. (And yes, there could be some familiar GUI elements to facilitate certain kinds of interaction.) I'm not a great designer or artist, so these examples aren't necessarily wonderful. But I'm convinced that there's a huge opportunity for deep, unified, visually stunning personalization of the user experience that is difficult to capture in the chopped up, application-centric UI of present-day smart devices.

Initially, the environment I'm developing is just for my own use and doesn't offer much in the way of beautiful visual effects. It runs cross-platform in Palm OS, Windows Mobile, and S60. In my first pass I'm implementing PIM applications (tuned for use with David Allen's "Getting Things Done" method) with integrated calling and sending of email. I'm excited about what I've got so far but sometimes wonder if my judgment isn't impaired by my heavy usage of command interfaces as a developer. When it's a little more complete and polished I will share it with users and start collecting feedback. If the response is as good as I hope I will open up an API for developers to create their own plug-in extensions to the command language.

But if I'm right about this, where it belongs is not in an application that runs on a smart device, but at the ground floor of the mobile user interface. It's a replacement for the whole idea of menus, applications and forms, and to be the most beneficial it should be the way you access every feature of your device. Here's hoping that Don Norman's command line idea makes the impression I think it should on the folks who are producing the next great mobile UI.

(Thanks to mobiface for the link to the Don Norman piece, and for the opposing (but also retro) perspective: that the mobile user interface will be heading back in the direction of Microsoft Bob.)

If you are a mobile developer and you don't have the Little Springs Design blog in your RSS reader it's time to remedy that situation. Barbara Ballard does the developer community a great service in getting us to think like users when designing user interfaces. Spend some time browsing the site when you can. This week she writes "If the user can't use it, it's broken."

A user attempts to use a product, and fails.

A developer points out how to use the product, and walks away believing there is nothing wrong with the product.

The developer is wrong. If the user can not use the product, the product is broken.

This key concept to usability is not well understood amongst many developer communities, including (perhaps especially) Java ME and mobile web communities. You can see this in some of the comments I get... "I now know the special tricks for using Gmail and Opera Mini." This does not mean that people can use the products.

The problem is worse—much worse—in the mobile community than it is for desktop applications and sites.

Barbara goes on to talk about how broken user interfaces are to a significant extent the fault of the carriers. Their "organizational silos" have the effect that each phone's user interface is almost an accidental product of various outsourced decisions that have little or no cognizance of each other. I agree. But even though as mobile developers it's written in our union contract that we must denigrate the wireless carriers at every turn, I don't want us to be removed from the hot-seat quite so quickly.

It's a rare mobile application that I install where I don't get pulled up short by thoughtless user interface elements in the first few minutes of use. Java ME and S60 apps that rely on soft buttons and menus for navigation suffer the most, but apps written for touchscreen devices running Palm OS and Windows Mobile still seem harder to use than they should be, often because they don't cope well with the situation where someone has only one hand free and can't use the stylus. I think most developers feel that designing a user interface is the easy part of writing software, and I admit that most of the time I think of UI design as fun and relaxing, and start feeling proud and attached to my designs much sooner than I should.

More recently, though, nothing seems to satisfy me and in some cases I've started to feel that guidelines I've followed for years don't produce interfaces that are simple enough for the first time user to discover without mucking up the experience for the veteran power user.

I haven't always been able to find an answer to the problem. And that reminds me just how many of the fundamental principles that will turn mobile computing into a mainstream phenomenon have yet to be discovered. As developers we are in a position to engage in a discovery process with our users that can help define that future. We need to think outside the box, because the simplicity that is needed is not inside any box we have before us.

Strategy Analytics says touchscreen phones are about to take off. They're predicting that 40% of mobile phones will have touch-sensitive screens within 6 years, which seems pretty optimistic given that we are somewhere around 2% today. They also say the timing of the inflection point (predicted around the end of next year) will depend on three conditions:

1. touchscreen prices must drop (as they are expected to)
2. revenue-generating applications must be developed to differentiate these phones
3. an iconic touchscreen phone needs to appear in a blockbuster movie

Is it just me or do the rest of you feel your heart sink when you hear that the future course of our industry depends on whether Tom Cruise taps a hardware button or a touchscreen during a three-second clip from Mission Impossible 4?

But Strategy Analytics may be right. Only I think they are missing an important condition for touchscreen adoption. And don't think of Mission Impossible, here. Think Minority Report. You know the scene I'm talking about:

The visual of Tom Cruise peeling through seeming terabytes of visual data was mind-blowing not because of the curving wall-sized glass tableau that was his computer screen (although I think that did speak to our lust for gigantic monitors). It was more the way he was able to navigate and manipulate the data so fluidly and rapidly with nothing but hand gestures.

Gestures--simple movements of which small variations can convey rich meaning--are not something you can do well with a mouse and keyboard, still less with a mobile phone keypad. We have "point," "click," and the occasional "drag" as available gestures on the PC. On most phones we have "click" and "click." And "click" and "click" some more. The coarseness of the input we can give to our mobile devices accounts for the biggest complaint we hear about phone user interfaces: the menus.

Navigating the handset with the "popup-> scroll-> select-> view screen-> repeat" motif is bad UI not just because it takes many clicks to do simple, everyday tasks. It's bad because you can't even see what your options are until you start opening menus. And because looking at the menu affords little or no view of the road ahead. Obscure menu options like "Settings" or "My Stuff" can seem almost completely opaque from the standpoint of a user who wants to perform a certain task. We often wonder why so few users bother to learn what their phone is capable of, still less to install third party software. Well, menus are a major reason.

The sad thing is that while a touchscreen can offer much better modes of navigation, no mobile OS vendor has developed a stylus-based user interface that's appreciably better. Palm OS and Windows Mobile replaced some of the menus with on-screen buttons, which helps a little. You can at least see some of your options on the screen without popping up a mystery menu. And yes, they offer stylus gesturing systems as a means of entering handwritten text--something you don't want to do a lot of on a mobile device. But for navigation the motif is still that of the PC: point and tap, view the next screen, point and tap some more. The possibility of performing a task like "check my Gmail" or "make an appointment with Judy" with a single gesture of the stylus seems never to have been considered, though it's thoroughly practical with a touchscreen and stylus (or finger).

Here's just one example of how this could be done. Imagine that instead of the usual smartphone graphical menu--a grid of icons to tap--we had the icons arranged in, say, a ring with a tiny "+" to mark the center. That mark is where you begin the gesture to perform a new task. To check your Gmail account you move the stylus from the center of the screen toward the Email icon, which in turn enlarges and moves to greet your stylus point. Other icons shrink and move out of the way--you're not interested in them now. As the pen approaches the Email icon the most common email tasks emerge as icons and text around it: perhaps Fetch, New, and Read. Moving the pen smoothly toward Fetch it expands and account option icons blossom from it: Work, Gmail, and All. Change the direction of the pen movement to meet the Gmail icon and lift the stylus point. The email client launches and checks your Gmail account after a single stroke of the stylus. The visual effect could be stunning--variously rendered as flying into the interface or watching a vine sprout in stop motion video. It would be easy to make this the stuff of a Hollywood sci-fi thriller.

Hollywood can help sell such a smartphone, but consider what this might mean for increasing peoples' use of its smart features, which is the brick wall we're hitting now with popular smartphone platforms. No memorization of gestures is required, so unlike "Grafitti" the learning curve is near zero. In fact, exploring the capabilities of the phone for the first time is vastly easier and more fun. As you start a gesture, you're given a view "down the road" of the tasks you can choose even before the applications that perform these tasks have launched. With a few minutes of eye-popping doodling a user could gain a very good idea of most of the phone's capabilities and how to use them. Over time, common tasks will be committed to muscle memory as quick gestures that make access to these tasks dramatically faster than is possible on current smartphones or PDAs.

This kind of gestural navigation requires a new software platform. Applications must register with the system their menu hierarchies and certain data queries for the "view ahead" interface to work. (A service-oriented architecture like I've discussed here and here would help.) This platform would provide rich new opportunities for personalization as it could be skinnable to produce all kinds of beautiful visualizations.

If touchscreen devices do take off as Strategy Analytics predict, I contend it will happen because they offer the first really discoverable and fun user interfaces for mobile phones. Stylus input needn't be a watered down version of mousing on a PC: it can and should be an experience that is a great improvement over the PC experience. One of my company's current projects is to explore this possibility, but I hope that others will try.