Saturday, June 5, 2021

Handling Ambiguity

The following is an excerpt and case from one of the classes I teach in Crisis Informatics.


Ambiguity is a topic that most students find unsettling, especially in the university setting. And this is worth talking about and how it differs from the workplace. First, let's talk about a definition. If you Google ambiguity, you get this rather good definition of "being open to more than one interpretation; inexactness".  Inexactness is good one-word definition of ambiguity. Merriam Webster provides another one-word synonym, uncertainty[1].  When it comes to assignments and grades, students want to know exactly what's expected.  And as a teacher, I am obligated to provide that.  But as a senior manager, or as a customer, I don't expect this.  I expect you to make sense out of the uncertainty, out of the possibilities. 

Allow me to illustrate by way of a story:


This is a story that relates to this and is one of my favorite experiences. I went to a cooking class with my wife in Varenna, Italy. We were staying in Bellagio and took the ferry over across Lake Como and then took a taxi cab up to this restaurant on the top of the hill, over Varenna where we met our host.  I called him Professor Marino, because he was the chef who was teaching this class and this day he happened to be teaching in English. He taught it in certainly in Italian, and may have taught it in French also. But I called him Professor Marino because he was a bit of a philosopher, as you'll see. So, the group that was there for the class, we were told we were going to make gnocchi. So, the first question, we asked him after he took and he poured a whole bag of flour onto the table. And he made like a volcano shape out of it. And he put four eggs in the center. And then he starts mixing it with his fingers. And he's mixing the dough and mixing the dough. So, the first question was, how can you tell if the dough is mixed enough? And he said, close your eyes and touch the dough. Think of something beautiful and touch the dough. And you'll know when it's ready. Of course, if you've done it many times before you know when it feels right, because you've been there. And the next thing he does is he starts rolling out the dough with his rolling pin. And then we asked the next question, Well, how do you know when the dough is thin enough? And he says, you can see the grain of the table, the grain of the wood will show through the dough and that's how you know that it's thin enough. Now, he said, in Italian cooking, for most dishes, you need to taste and adjust, taste and adjust, taste and adjust. It's an inexact method. And he said that's true for most every Italian dish you make, with one exception.  Italian pastry. For Italian pastry, he said, you must follow the recipe, you need to have the ingredients exact, you need to have the temperature in the oven exact, you need to have the time that it's in the oven exact, otherwise you ruin the pastry. 

So, an interesting question to consider, and applying this as a metaphor: is most of business, including technology, about making pasta, or pastry? [dramatic pause]. It's about making pasta.  In other words, it's handling ambiguity. So, if you prefer to work with a pastry approach and have an exact recipe, you're going to get very frustrated in your business career. Learning how and developing strategies for dealing with ambiguity will help you navigate that transition. Things at the university tend to be fairly exact. Rubrics are fairly exact. Exams are exact. But work that we do in corporations, in nonprofit organizations, and governments tends to be rather inexact. And there's a fair amount of discovery, experimentation and exploring, that you just plain have to figure out. That's a challenge. 

So, why is it important to be able to handle ambiguity? Well, everybody in leadership expects you to be able to handle the ambiguous situation. And ambiguous projects, especially when they're projects from clients.  These have a greater probability of being ambiguous.  

Your initial encounter with client ambiguity may come during your interview for the job.  I'll illustrate this with the Amazon Interview case, below, where the candidate was deliberately asked ambiguous questions to see how they would handle it, to see what their thought process was, how they broke it down, and made sense out of it.  Because when you're doing client projects, that's the thought process you're going to need to follow.  It's also an opportunity to prove your value-add at times. How do you make sense out of the chaos? Can you demonstrate that?


The Amazon Interview Case 

A question was posted on Quora.com, presumably in Dec. 2016 [3], by a developer who had applied to Amazon and did not pass the on-site interview. They asked Quora readers for advice about what to do next. One of the responders was Matt Kellner, a software engineer from the Seattle area [4]. 1.5 million people have read his answer to-date.

Matt indicated in the follow-up comments that he previously worked for Amazon, and that he has been on both sides of the interview table. He notes that the line of questioning where an interviewee “locked up” is typical at tech companies. It’s deliberately ambiguous. It creates a mini-crisis to see how the applicant handles it. Read his response in full, below.

Also read the other replies, as there are interesting perspectives on tough interview questions. The bottom line is that companies are looking for candidates who can handle ambiguity. It’s not only for solving open-ended problems internal to the organization; often customers do not know exactly what they want and are not particularly adept at providing focused requirements. In fact, one could argue that in the realm of customers, whether internal or external, perfect requirements don’t exist. So handling ambiguity is fundamental to tech-related work.

Here are the discussion questions:

  1. Have you encountered ambiguous questions during interviews? How have you handled them?
  2. What do you think about Matt’s advice on how to handle ambiguous questions?  What resonates and what doesn’t?
  3. If the goal is to have a successful job interview (i.e., generate an offer of employment), how will you handle the inevitable ambiguous or open-ended question?
  4. What do you find frustrating about this, and how do you propose to deal with that?  
  5. If a goal of crisis preparedness is to build more resiliency, or the ability to “bounce back”, how will you prepare for the interview crisis? 

Here is the Quora.com excerpt:

What should I do after failing at Amazon's on-site interview?

This question previously had details. They are now in a comment.

11 Answers

Matt Kellner, Software Engineer, Video Game Historian and Part-Time Farmer

“I wouldn’t call you a “loser”[5], but it does sound like you did one thing that interviewers don’t like to see in technical interviews. When you get a technical question in an interview, that question is almost always going to start out vague - it’ll have some important pieces missing such that you actually CAN’T adequately solve the problem without more info.

In a case like that, the interviewer is being very deliberate about the ambiguity. They may be simulating a request from a customer where the customer knows what they want but didn’t do a good job of communicating it. The point here is to see how you handle ambiguity: Do you make assumptions? Do you lock up? Do you go down a path toward a solution without understanding the question? Or do you try to identify what you don’t know and start asking questions to clarify?

I’ve interviewed a lot of people who did what you described: They just stood there staring at my question, not talking at all, not letting in on their thought process, etc. The problem is, you might be thinking through the problem, but the interviewer can’t tell what you’re doing, so to them, it looks like you just lock up.

We generally understand that people get stage fright in some situations. The interview is a stressful, grueling process for most people, so it’s not surprising that you’d have a case where you just lock up - some people don’t even realize they’re doing it. That’s why a good interviewer will try to prompt you, will ask you to describe what you’re thinking, will give you small bits of guidance, etc.. We WANT you to succeed, and it’s in our best interest to do whatever we can to help you succeed.

But there’s only so much we can do. If it becomes obvious that you’re completely lost on a problem, or you have to be hand-held through the whole thing, or you’re just not going to “perform” no matter how much help we give, that’s not a good sign of how you’ll do in a real work environment. The real work environment, especially at Amazon, is often more stressful and faster-moving, and your success or failure on a real problem could make the difference between success and failure of the product you’re working on.

So, here’s what I always recommend to interview candidates:

·         Brush up on your problem-solving skills. Remember that most interviews are about how well you can understand and solve problems, not just how good your code looks.

·         Always, in some way, repeat back the question you’re asked. It’s up to you to decide how best to do this - you can paraphrase the question, you can draw a diagram on the board, you can describe it in terms of behaviors, etc. But make it clear that you do (or do not) understand what the interviewer is asking you to do.

·         If you find a point where you CANNOT do that, ask the interviewer to clarify. For example, if I ask you to divide two numbers, and you realize that I haven’t specified what should happen when the second number is zero, just ask me - something like “What should this method do when I try to divide by zero? Throw an exception? What kind of exception should we throw?” etc.

·         When the interviewer clarifies a question like that, you might then realize you have even more questions. Explore those. Take notes on the board as you get these clarifications, as they will inform your solution.

·         Keep things moving. Keep talking, think out loud, draw, write, whatever you need to do to keep your interviewer informed as to your thought process.

·         Finally, don’t worry about finishing your solution. Most of the time, it’s not about actually writing and fully testing a finished piece of code - if you get there, that’s great, and a good interviewer will probably add a bit of complexity, quiz you on testing or on the operational complexity of the solution, etc.. But most of the time, they’re more interested in seeing how you approach problems, how you think and how willing you are to reach out for help. (This includes cases where you know what you want to do but don’t necessarily know the correct syntax to do it in whatever language you’re working in. If you can describe what you’re doing, the interviewer can usually play technical reference for you.)


If it helps any, I’ve done a number of interviews (on both sides - being interviewed and giving the interview) where we never got around to writing a single line of code. Instead, we spent the whole time just discussing the problem. :) I personally think those kinds of interviews are the most fun.

Hope this helps. :) Last piece of advice: Don’t be afraid to apply at Amazon again in the future. You’re unlikely to have been “blacklisted” - let some time pass (say, 6 months or a year), brush up on your skills, and if you’re still interested in trying again, apply to new positions. I can’t guarantee that they WILL interview you again, but they certainly might.

 I remember that someone in the comments took exception to where I said “Instead, we spent the whole time just discussing the problem”, as it seemed (to that person) that I was saying interviews really are only about conversations and not technical. I want to clarify this point:

What I meant there was that we sometimes ask (or are asked) questions that are really huge in nature. For example, in one of my own job interviews, I was asked how I would design a banking system that could handle deposits, withdrawals and account transfers. When I started exploring edge cases, concurrent users, data storage mechanisms, account locking and fraud detection, etc., the scope of the problem blew up to enormous proportions. What started as a simple exercise in defining some common-sense interfaces for these individual functions turned into a wide-ranging discussion that touched on both general banking concepts and on distributed systems design. Given that where I work is concerned with such things, this was on-point.

Once I was hired, the person who did that part of my interview mentioned that he fully intended for the discussion to grow like that, as the “test” he was giving me was on whether I would just try to solve the direct problem, or if I would start looking at the “big picture”. I’ve found in many cases that this is one primary difference between “junior” and “senior” engineers - since I was applying for a senior position, this level of discussion was appropriate.

My point here is to illustrate what is meant by some interviews not involving coding, even though you may be interviewing for a coding position. Software engineering, particularly the type that Amazon does, entails a lot more than just basic problem solving and algorithms. As you move up the chain, you can’t avoid taking on more in the way of large-scale design, testing, project management and other things that you may not be used to doing. And these interviews, in part, are done to see how ready you are to step outside your comfort zone.

(Update 2, April 8, 2018) I was reviewing this question and found another answer I hadn’t seen before, stating that you should be happy that you didn’t get hired at Amazon, and linking to a Google site documenting “management abuse” and such. I’d just like to say that, as a former employee of Amazon, I strongly disagree with this person’s answer (they disabled comments, so I can’t respond directly). While I cannot speak for all departments at Amazon, I worked in a total of four of them while I was there, most of that in AWS, and my experience there was nothing like the majority of reports on that site.

There are a couple of groups at Amazon that most people agree are less than ideal to work for, but my experience was largely positive, and the difficulties I encountered had more to do with the “frugality” aspect of Amazon’s culture - most notably in running “lean” teams that do more work per person than at most other companies. But I was proud to be a part of the groups I worked for - I did some of the best work of my career (so far) in those groups - and I left the company on good terms and would consider going back in the future.

I won’t tell you not to check out that “FACE of Amazon” site, but keep in mind, it appears they have posted only universally negative stories there, and thus the site seems to suffer from confirmation bias. Their language and their use of an anonymous email system suggests that they are not interested in positive stories for balance. If you’re concerned about what it would be like to work at Amazon, I would encourage you to look also at sites like Glassdoor to view ratings from a more representative sample of current and former employees, as well as by job type and title.

In my experience, Amazon has very high standards for its employees, and it CAN and WILL burn you out if you let it. However, they do also respect you when you tell them what your limits are - if you can still do your job, you should not fear letting your manager know that you can only work 40 hours a week.”

 

About the Author

Matt Kellner

Software Engineer, Video Game Historian and Part-Time Farmer

Software Engineer1997-present

Studied at California Polytechnic State University, San Luis Obispo

Lives in Puget Sound



[1] Merriam-Webster Dictionary, https://www.merriam-webster.com/dictionary/ambiguity   

[2] From my "Letters to a Young Manager," series, http://www.hpmd.com/hpmd/personal/LTYMstories.nsf/links/488 

[3] The earliest comment is from Dec. 8, 2016, after expanding all the comments, here: https://www.quora.com/What-should-I-do-after-failing-at-Amazons-on-site-interview . The original question posting does not appear to be currently available on quora.com.

[5] See the comment by Heck Evergreen, Senior Software Dev Engineer in 3/10 Top Companies (2005-present), answered Jun 4, 2017


"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."