Tag Archives: future

Microsoft’s Envisioning team in the Office division puts out a video every few years (20082009, 2011) highlighting ideas and concepts they feel are coming in the next 10 years or so. Besides being extremely high quality and intriguing, these videos also give us a glimpse into what different individuals inside Microsoft are working on, researching, and conceiving. I’ve taken an especially keen interest in their most recent 2011 Productivity Future Vision short because of its ties with communication and email. Watch it before continuing (and perhaps revisit it as you read along so as to keep up):

Pretty cool, huh? There are some fantastic concepts in there (and very cool technology as well). I’ve watched the video many times, and I was even able to get some commentary on it directly from one of the Microsoft employees who helped create the video. Let’s take a closer look at some of the more pertinent and interesting scenes.

Short-form Messages and Integrated Calendaring

Near the beginning of the video, we see businesswoman Ayla reading an email while traveling in the cab to her hotel. You can see the email and the integrated response in the image below (click for full size).

A short message and integrated calendaring with the response.

First thing to note is the length of the original message. It’s short! Far too few messages are that short in real life. The sender, Qin, manages to get in a salutation, a comment about the flight, and a request in five lines, leaving enough space for a big, beautiful picture of him. We then see Ayla dictate a response that indicates she will get to the review “first thing tomorrow.” The device recognizes that a temporal event has just been mentioned and determines what the temporal event was (“Review Proposal”), checks her calendar, and inserts it, asking Ayla for confirmation.

What is wonderful about this proposed interaction is that it allows Ayla to use a “fire and forget” strategy because the task has been moved from her inbox straight to her calendar. Ayla does not need to flag the message, mark it as unread (as many do), or hope that she sees it tomorrow morning to remember to actually perform the review. Probably one of the most common strategies of coping with numerous emails and tasks is to leave the message in the inbox (Whittaker, 2005), often marking it as unread “so it will be visually distinctive and continue to catch [the user's] attention” (Wattenberg, Rohall, Gruen, & Kerr, 2005). However, Whittaker notes that “visual reminding is compromised when there are too many inbox messages” because “older outstanding items are overlooked when processing new incoming messages” (2005). And for busy executives receiving tens or hundres of messages a day, this can happen quickly.

Furthermore, when Ayla has the device actually create the calendar appointment automatically, it saves her from having to transpose the information from her email to her calendar, which is a step that prevents many people from actually completing a task (Blandford & Green, 2001). Later in the video we see Ayla attend to the task the next morning, and the review file is automatically associated with the appointment so Ayla doesn’t have to dig around in her emails or on a server to retrieve the file. This again is a huge step in reducing overload because it does not require Ayla to transpose any information or documents to the task.

Some email clients do currently attempt to pick out metadata like this from emails (Mail.app will look for dates and times; Outlook will too), but no client does it as effectively as is shown in the video, nor does any client do it from your reply, only from the original message, to my knowledge. This is certainly a technique worth implementing for those creating email clients.

5 Minute Focus

5 Minute Focus screen on Qin's device

One of the concepts reiterated several times throughout the video is that of a device automatically displaying tasks for a user to complete based on the situation. David Jones, the Microsoft program manager who was kind enough to give me a few thoughts on the video, said the following:

“The bigger idea we tried to get across is that we can surface the right emails for you to look at based on your current context – Ayla in the car, Qin with 5 minutes to spare in the subway, and Ayla in the Hotel room based on the fact that Shannon was worried.”

Qin replying via an integrated app to a quick questionQin might not have been able to write an employee review while waiting for the subway, but he certainly could respond to a quick one-line question. Somehow the device is aware of two things: the current context (the fact that Qin is waiting for a specific subway train), and the estimated time required to complete a request in his voicemail inbox (automatically transcribed, something that I think the latest version of Exchange Server will do for you, as will Google Voice). As a side note, I also like the nice touch of giving the option of grabbing something at the café as well, a little geolocation magic!

This also brings up the concept of an integrated mini-app that is displayed instead of a typical reply field. No written response or call is required by Qin because he approves the order on the spot from within the original message. Who remembers Google Wave? This type of ability was built in, where you could conduct polls, attendance, etc., in-message. That wasn’t the first time this idea had been proposed, however. Duchenaut and Watts found that the first time this was suggested was in 1976 by Anderson and Gillogly with “computational mail,” where when “the message is opened, the program is executed” (2005). Now obviously we would have to be careful to avoid the email viruses of yesteryear, and this would mostly likely be a very custom development job inside of each company, but it would certainly also be a leap forward in usability.

Conversations from Anywhere

Another powerful idea developed in the Productivity Vision film is that content, no matter what or where, is all linked, and people can have a conversation about any piece of content, augmented by context. At the end of the video we are introduced to Shannon, Ayla’s daughter. Shannon is trying to find a recipe to make for school, and writes and then video calls her mother.

Shannon initiating a video call from within her message

I was wondering just how that video call was started. Here is how David elucidates this scene:

“Shannon initiated the call. She wrote a handwritten note, and then initiated a call from “within” it. The idea here is that you can have a conversation from within any piece of content. This is similar to how Ayla leaves Jeff a comment in the presentation she is working on in the Office scene. And how Jeff leaves Qin a comment in the document that we see in 5 minute mode in the Subway scene. Unfortunately we don’t show Ayla actually answering the call—we were re-using footage and didn’t have a shot of her pressing anything on the phone. It goes by so fast that  no one notices though.”

So Shannon started by writing a note, but then used that note to start a video call. How many times have you been writing a reply to a colleague, or been in the middle of composing a report you are both working on, and realized that just wanted to talk to them? Or how often have you been reviewing some work done by a co-worker and thought of leaving a comment, but didn’t want to compose a whole new email or add the comment in-file and then resend it?

Similarly, we see other examples of a related concept elsewhere in the video. For example, when Jeff is writing the report, two sets of words are underlined for him, with the computer offering to insert supplemental information about those objects into the report for him. Arrows have been added to show what the computer offered in the video.

The computer automatically offering to insert supplemental content.

I’m not so sure that I would find this type of feature handy very often, but the idea that related information should be surfaced is extremely powerful, and also very pertinent to information overload. It is time intensive to open attachments or find previously saved data when conducting business over email, and according to the most current research, the biggest contributor to email overload are messages/tasks that require lots of interdependent, qualitatively complex messages (Bellotti, Ducheneaut, & Howard, 2005). Stopping mid-sentence during composition to go and find an Excel document that you know has a figure in it related to what you are writing incurs a large task-switching cost. If the computer can automatically determine and retrieve relevant information, documents, and links, then this task-switching cost can be avoided and the complexity of those messages greatly reduced. Likewise, pulling people’s schedules and locations when discussing them in an email would also be very helpful, avoiding an extra trip to the often-separate calendar. Both Bellotti et al. and researchers at IBM (Kerr & Wilcox, 2004) have experimented with making related information in email clients “first-class citizens,” with very positive results. Implementing such a feature would obviously require very complex natural language processing and metadata handling capabilities, but it would be a huge boon to reducing complex and overloading emails.

Note the pink bar indicating Shannon's emotionAs a bonus, David also pointed out something that he said most people miss: the fact that Shannon was worried about finding a recipe, and therefore gets a pink bar above her as an indication to her mother of her emotion. “Tried to work in some emotional reco here,” said David. This is quite advanced, but makes a lot of sense. How would it apply in the business world? We are often unaware of the current workload of the people we are sending messages to, and are often unaware of how much work we have delegated them in the past. Sending emails is “cheap,” monetarily of course but also cognitively, and we get in the trap of sending too many to be useful. Could some sort of indicator be included marking how “overloaded” the individual is (this is measurable, in fact; see Hogan & Fisher, 2006; or Jonathan Spira of Basex), or how much we have delegated to them in the past? Indicators such as these may help senders to temper or change the workload they are sending out via email.

Conclusion

Whew, there’s a ton here and I haven’t even covered the half of it! Let’s review the major ideas and concepts that spring from this short that would be useful in the fight against email and information overload.

  • Shorter, more complete messages: Sounds simple, but it would be a big help for many people to retrain ourselves to write more clearly and succinctly (especially me; as you can tell, brevity is not my forté). Could we write email client plugins that evaluate writing style and give us helpful hints?
  • Automatic information transposition and task creation: Huge because it removes several steps that many people simply don’t take the time for (transposing information) and removes the burden of task management from your inbox, organizing it more effectively elsewhere.
  • Context-aware task surfacing: Do you have five minutes to spare? Your email client should give you tasks that it knows you can complete in that time frame so that you don’t have to hunt and peck for them.
  • Integrated mini-apps: Similar to Google Wave widgets, business-related application components that allow non-text actions to be taken from emails.
  • Conversations from and in any content: Any and all content is up for use as a conversation starter, and doing so allows the conversation to always be context aware.
  • Context-aware data surfacing: By pulling up relevant documents or information as you are working, the computer can greatly reduce task-switching costs associated with searching for needed data. This would be an enormous boon to productivity.
  • Emotional/historical indicators: Is someone overloaded because you’ve unconsciously sent them too many tasks recently? Indicators could help us track how someone else is doing, and we can adjust our behavior as senders based on that.

A big thanks to David Jones for taking the time to respond to my questions about the video and for the information he provided. I’m hoping that we can take the concepts presented here and turn them into reality and see the benefits we can reap from it.

Sources

Bellotti, V., Ducheneaut, N., & Howard, M. (2005). Quality versus quantity: E-mail-centric task management and its relation with overload. Human-Computer Interaction, 20, 89-138. Retrieved from http://portal.acm.org/citation.cfm?id=1466574

Blandford, A., & Green, T. (2001). Group and individual time management tools: What you get is not what you need. Personal and Ubiquitous Computing, 5, 213–230.

Ducheneaut, N., & Watts, L. A. (2005). In search of coherence: a review of e-mail research. Human-Computer Interaction, 20, 11-48. Retrieved from http://portal.acm.org/citation.cfm?id=1466572

Hogan, B., & Fisher, D. (2006). A scale for measuring email overload. Microsoft Research, 7-9. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:A+Scale+for+Measuring+Email+Overload#0

Kerr, B., & Wilcox, E. (2004). Designing remail. Extended abstracts of the 2004 conference on Human factors and computing systems – CHI  ’04 (Vol. 23127, p. 837). New York, New York, USA: ACM Press. doi:10.1145/985921.985944

Wattenberg, M., Rohall, S. L., Gruen, D., & Kerr, B. (2005). E-mail research: targeting the enterprise. Human-Computer Interaction, 20(1), 139–162. L. Erlbaum Associates Inc. Retrieved from http://portal.acm.org/citation.cfm?id=1466575

Whittaker, S., Bellotti, V., & Moody, P. (2005). Introduction to This Special Issue on Revisiting and Reinventing E-Mail. Human-Computer Interaction, 20(1), 1-9. doi:10.1207/s15327051hci2001&2_1

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook

Your alarm goes off at 5:30 AM and you reach for your smartphone, which cheerily notifies you that overnight you received 54 emails. And you only went to bed five hours ago. After a quick glance, you reply to a couple and get up to start the day. After arriving at the office you’ve suddenly receive another fifty, and so you take the first hour of the workday to sort through them and respond to those that you can. A 9 AM meeting cuts you short, but you continue to tap out a couple of replies during the boring bits. All throughout the day your laptop is dinging and popping up notifications about new and urgent request from colleagues to get them those numbers for the report, or to figure out where the best place is to get sushi. You receive several hundred emails throughout the day, and despite your best efforts, your unread count consistently hovers around 1,200 or so. Finally, quitting time rolls around and you head home, only to later take an hour or two away from your family and rest in order to try and tackle a few more in the futile attempt to get down to inbox zero.

Sound familiar? This is the doom loop experienced by many people, especially managers and executives who are suffering from information overload. No one is immune to a complete glut of messages, information, reminders, and group communications. Yet the email system we employ remains generally the same as it was over 30 years ago, when email was invented. Well, believe it or not, technology has come a long way in the last 30 years, and I believe we are finally at a point when we can begin tackling this issue head-on and improve the work lives of many people, subsequently giving them more time to spend with their family and friends.

Why doesn’t our email client tell us what needs to be done in the next five minutes? Why do we still treat inboxes like massive lists of equally-weighted unique messages? Why are we not using the resources of the cloud to apply more machine learning to the conversations we have to allow us to be smarter about them? Which behaviors are bad when it comes to communicating, and which are beneficial? Though not exactly declarative of what the future will really be like, perhaps we can glean some good ideas on communication from this video:

Motivation

It is the final semester of my masters program of Information Systems at BYU, and we are enrolled in a capstone course where we have free reign to choose a project that melds all of the material we have learned over the past several years into a culminating show of knowledge. I followed a Ph.D. prep track during my degree and therefore wanted to incorporate and hone the research skills I gained in those classes. I’ve been searching for a project that would be more than just building a web app or creating a marketing plan; I truly want to start changing the world. When I saw businessmen and women who are also husbands and wives, fathers and mothers, taking unreasonable amounts of time out of their day just to try and keep the beast that is email in its bursting cage, I found a problem that finally fit the bill.

As I am still formulating the exact streams of research I want to pursue and the deliverables I want to create over the next three and a half months, I am operating at a general overview level. As such, let me just share with you some notes that I have been jotting down as I’ve explored different ideas. There is no real organization to this, and it may just be my own thoughts hastily copied down. However, I will be honing my concentration over the next few weeks and will include more detailed, knowledge-rich posts as I go. In the spirit of open knowledge, I will include all my research here for public consumption. For now, here’s what I’m thinking:

Braindump:

  • Email is a huge timesink for executives, and they never ever catch up
  • It represents huge inefficiencies introduced by…???
  • What do we need to do to fix it?
  • Some are beginning to claim that “email is dead” and that social will overtake it. Email will not die, but it will evolve. I want to be a part of that evolution.

Fixing it:

  • Introduce natural language processing to automatically categorize and sort messages based on their content (what project is an email referring too, what actions are required in it, is it just an announcement, etc.)
  • Apply contextual clues to the email, like the org chart and where the person and other people lie in it to assign weight
  • Remove emails out of the “line item” view, and into more a natural, project group view
  • I should be able to look at my email client and have it tell me what I can get to next, what I need to do.
  • People don’t want to spend time building filters, categorizing, or sorting. Facebook taught us this. (Can we use a social graph to add context like they do?)
  • Labels are way better than folders. What is better than labels?

Questions

  • Does Gmail’s priority inbox work for some people?
  • How do you solve the issue of a new message always popping up and distracting you from an existing task? (Well, don’t have a popup or a sound notification)
  • What do plugins like Xobni bring to the table?
  • What exactly are the inefficiencies? (I’m guessing research has covered this before)

What has been covered before?

Topics to research

  • Natural Language Processing (NLP)
  • What efforts have worked before?
  • Affordance

Call for Help

Please, if you or anyone you know has even the smallest bit of domain expertise in email, communications, affordance, NLP, or machine learning, I would love to talk with you! You can’t solve big problems in a vacuum or alone, and I will need all the help I can get. Contact me here.

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook