DISQUS

Have had a dishwasher like this for about 6 years...never knew it didn't have to go on the door. Can you say "lack of affordance" or "non-intuitive?" I thought you could. :-)

Congrats on the new position. Sounds like a great and fun challenge ahead.

Question: How will you determine what the next gen/level of users need? What if the "masses" need something different from the (presumably tech-savvy) early adopters?

I think having fewer distractions during testing can be a Good Thing. Granted, it may falsely set the application up to succeed or to do better than it might in the real world. But even given that kind of benefit, most applications will have many problems uncovered during testing.

In your hypothetical example, you have to question whether the reason the app earned undeservedly high marks was due to a lack of interruptions or due to some other issues with the usability testing methods applied. For example: Were the tasks too simple and not representative of real world tasks? Were the evaluators truly representative of the user base? Did the facilitator ask any leading questions? etc...

Of course, your theory about distractions might be right for the hypothetical situation...and you could use other UCD methods to validate or reinforce that theory. For example, you might use a little ethnographic field study with tech support and managers and observe them using this application and others to determine if distractions were the cause of user errors that didn't turn up when there were no distractions.

P.S. Glad to see you blogging!