Saturday, June 5, 2021

Handling Ambiguity

The following is an excerpt and case from one of the classes I teach in Crisis Informatics.

Ambiguity is a topic that most students find unsettling, especially in the university setting. And this is worth talking about and how it differs from the workplace. First, let's talk about a definition. If you Google ambiguity, you get this rather good definition of "being open to more than one interpretation; inexactness".  Inexactness is good one-word definition of ambiguity. Merriam Webster provides another one-word synonym, uncertainty[1].  When it comes to assignments and grades, students want to know exactly what's expected.  And as a teachers, I am obligated to provide that.  But as a senior manager, or as a customer, I don't expect this.  I expect you to make sense out of the uncertainty, out of the possibilities. 

Allow me to illustrate by way of a story:

This is a story that relates to this and is one of my favorite experiences. I went to a cooking class with my wife in Varenna, Italy. We were staying in Bellagio and took the ferry over across Lake Como and then took a taxi cab up to this restaurant on the top of the hill, over Varenna where we met our host.  I called him Professor Marino, because he was the chef who was teaching this class and this day he happened to be teaching in English. He taught it in certainly in Italian, and may have taught it in French also. But I called him Professor Marino because he was a bit of a philosopher, as you'll see. So, the group that was there for the class, we were told we were going to make gnocchi. So, the first question, we asked him after he took and he poured a whole bag of flour onto the table. And he made like a volcano shape out of it. And he put four eggs in the center. And then he starts mixing it with his fingers. And he's mixing the dough and mixing the dough. So, the first question was, how can you tell if the dough is mixed enough? And he said, close your eyes and touch the dough. Think of something beautiful and touch the dough. And you'll know when it's ready. Of course, if you've done it many times before you know when it feels right, because you've been there. And the next thing he does is he starts rolling out the dough with his rolling pin. And then we asked the next question, Well, how do you know when the dough is thin enough? And he says, you can see the grain of the table, the grain of the wood will show through the dough and that's how you know that it's thin enough. Now, he said, in Italian cooking, for most dishes, you need to taste and adjust, taste and adjust, taste and adjust. It's an inexact method. And he said that's true for most every Italian dish you make, with one exception.  Italian pastry. For Italian pastry, he said, you must follow the recipe, you need to have the ingredients exact, you need to have the temperature in the oven exact, you need to have the time that it's in the oven exact, otherwise you ruin the pastry. 

So, an interesting question to consider, and applying this as a metaphor, is most of business, including technology, about making pasta, or pastry? [dramatic pause]. It's about making pasta.  In other words, it's handling ambiguity. So, if you prefer to work with a pastry approach and have an exact recipe, you're going to get very frustrated in your business career. Learning how and developing strategies for dealing with ambiguity will help you navigate that transition. Things at the university tend to be fairly exact. Rubrics are fairly exact. Exams are exact. But work that we do in corporations, in nonprofit organizations, and governments tends to be rather inexact. And there's a fair amount of discovery, experimentation and exploring, that you just plain have to figure out. That's a challenge. 

So, why is it important to be able to handle ambiguity? Well, everybody in leadership expects you to be able to handle the ambiguous situation. And ambiguous projects, especially when they're projects from clients, have a greater probability of being ambiguous.  

Your initial encounter with client ambiguity may come during your interview for the job.  I'll illustrate this with the Amazon Interview case, below, where the candidate was deliberately asked ambiguous questions to see how they would handle it, to see what their thought process was, how they broke it down, and made sense out of it.  Because when you're doing client projects, that's the thought process you're going to need to follow.  It's also an opportunity to prove your value-add at times. How do you make sense out of the chaos? Can you demonstrate that?

The Amazon Interview Case 

A question was posted on, presumably in Dec. 2016 [3], by a developer who had applied to Amazon and did not pass the on-site interview. They asked Quora readers for advice about what to do next. One of the responders was Matt Kellner, a software engineer from the Seattle area [4]. 1.5 million people have read his answer to-date.

Matt indicated in the follow-up comments that he previously worked for Amazon, and that he has been on both sides of the interview table. He notes that the line of questioning where an interviewee “locked up” is typical at tech companies. It’s deliberately ambiguous. It creates a mini-crisis to see how the applicant handles it. Read his response in full, below.

Also read the other replies, as there are interesting perspectives on tough interview questions. The bottom line is that companies are looking for candidates who can handle ambiguity. It’s not only for solving open-ended problems internal to the organization; often customers do not know exactly what they want and are not particularly adept at providing focused requirements. In fact, one could argue that in the realm of customers, whether internal or external, perfect requirements don’t exist. So handling ambiguity is fundamental to tech-related work.

Here are the discussion questions:

  1. Have you encountered ambiguous questions during interviews? How have you handled them?
  2. What do you think about Matt’s advice on how to handle ambiguous questions?  What resonates and what doesn’t?
  3. If the goal is to have a successful job interview (i.e., generate an offer of employment), how will you handle the inevitable ambiguous or open-ended question?
  4. What do you find frustrating about this, and how do you propose to deal with that?  
  5. If a goal of crisis preparedness is to build more resiliency, or the ability to “bounce back”, how will you prepare for the interview crisis? 

Here is the excerpt:


What should I do after failing at Amazon's on-site interview?

This question previously had details. They are now in a comment.

11 Answers

Matt Kellner, Software Engineer, Video Game Historian and Part-Time Farmer

“I wouldn’t call you a “loser”[5], but it does sound like you did one thing that interviewers don’t like to see in technical interviews. When you get a technical question in an interview, that question is almost always going to start out vague - it’ll have some important pieces missing such that you actually CAN’T adequately solve the problem without more info.

In a case like that, the interviewer is being very deliberate about the ambiguity. They may be simulating a request from a customer where the customer knows what they want but didn’t do a good job of communicating it. The point here is to see how you handle ambiguity: Do you make assumptions? Do you lock up? Do you go down a path toward a solution without understanding the question? Or do you try to identify what you don’t know and start asking questions to clarify?

I’ve interviewed a lot of people who did what you described: They just stood there staring at my question, not talking at all, not letting in on their thought process, etc. The problem is, you might be thinking through the problem, but the interviewer can’t tell what you’re doing, so to them, it looks like you just lock up.

We generally understand that people get stage fright in some situations. The interview is a stressful, grueling process for most people, so it’s not surprising that you’d have a case where you just lock up - some people don’t even realize they’re doing it. That’s why a good interviewer will try to prompt you, will ask you to describe what you’re thinking, will give you small bits of guidance, etc.. We WANT you to succeed, and it’s in our best interest to do whatever we can to help you succeed.

But there’s only so much we can do. If it becomes obvious that you’re completely lost on a problem, or you have to be hand-held through the whole thing, or you’re just not going to “perform” no matter how much help we give, that’s not a good sign of how you’ll do in a real work environment. The real work environment, especially at Amazon, is often more stressful and faster-moving, and your success or failure on a real problem could make the difference between success and failure of the product you’re working on.

So, here’s what I always recommend to interview candidates:

·         Brush up on your problem-solving skills. Remember that most interviews are about how well you can understand and solve problems, not just how good your code looks.

·         Always, in some way, repeat back the question you’re asked. It’s up to you to decide how best to do this - you can paraphrase the question, you can draw a diagram on the board, you can describe it in terms of behaviors, etc. But make it clear that you do (or do not) understand what the interviewer is asking you to do.

·         If you find a point where you CANNOT do that, ask the interviewer to clarify. For example, if I ask you to divide two numbers, and you realize that I haven’t specified what should happen when the second number is zero, just ask me - something like “What should this method do when I try to divide by zero? Throw an exception? What kind of exception should we throw?” etc.

·         When the interviewer clarifies a question like that, you might then realize you have even more questions. Explore those. Take notes on the board as you get these clarifications, as they will inform your solution.

·         Keep things moving. Keep talking, think out loud, draw, write, whatever you need to do to keep your interviewer informed as to your thought process.

·         Finally, don’t worry about finishing your solution. Most of the time, it’s not about actually writing and fully testing a finished piece of code - if you get there, that’s great, and a good interviewer will probably add a bit of complexity, quiz you on testing or on the operational complexity of the solution, etc.. But most of the time, they’re more interested in seeing how you approach problems, how you think and how willing you are to reach out for help. (This includes cases where you know what you want to do but don’t necessarily know the correct syntax to do it in whatever language you’re working in. If you can describe what you’re doing, the interviewer can usually play technical reference for you.)

If it helps any, I’ve done a number of interviews (on both sides - being interviewed and giving the interview) where we never got around to writing a single line of code. Instead, we spent the whole time just discussing the problem. :) I personally think those kinds of interviews are the most fun.

Hope this helps. :) Last piece of advice: Don’t be afraid to apply at Amazon again in the future. You’re unlikely to have been “blacklisted” - let some time pass (say, 6 months or a year), brush up on your skills, and if you’re still interested in trying again, apply to new positions. I can’t guarantee that they WILL interview you again, but they certainly might.

 I remember that someone in the comments took exception to where I said “Instead, we spent the whole time just discussing the problem”, as it seemed (to that person) that I was saying interviews really are only about conversations and not technical. I want to clarify this point:

What I meant there was that we sometimes ask (or are asked) questions that are really huge in nature. For example, in one of my own job interviews, I was asked how I would design a banking system that could handle deposits, withdrawals and account transfers. When I started exploring edge cases, concurrent users, data storage mechanisms, account locking and fraud detection, etc., the scope of the problem blew up to enormous proportions. What started as a simple exercise in defining some common-sense interfaces for these individual functions turned into a wide-ranging discussion that touched on both general banking concepts and on distributed systems design. Given that where I work is concerned with such things, this was on-point.

Once I was hired, the person who did that part of my interview mentioned that he fully intended for the discussion to grow like that, as the “test” he was giving me was on whether I would just try to solve the direct problem, or if I would start looking at the “big picture”. I’ve found in many cases that this is one primary difference between “junior” and “senior” engineers - since I was applying for a senior position, this level of discussion was appropriate.

My point here is to illustrate what is meant by some interviews not involving coding, even though you may be interviewing for a coding position. Software engineering, particularly the type that Amazon does, entails a lot more than just basic problem solving and algorithms. As you move up the chain, you can’t avoid taking on more in the way of large-scale design, testing, project management and other things that you may not be used to doing. And these interviews, in part, are done to see how ready you are to step outside your comfort zone.

(Update 2, April 8, 2018) I was reviewing this question and found another answer I hadn’t seen before, stating that you should be happy that you didn’t get hired at Amazon, and linking to a Google site documenting “management abuse” and such. I’d just like to say that, as a former employee of Amazon, I strongly disagree with this person’s answer (they disabled comments, so I can’t respond directly). While I cannot speak for all departments at Amazon, I worked in a total of four of them while I was there, most of that in AWS, and my experience there was nothing like the majority of reports on that site.

There are a couple of groups at Amazon that most people agree are less than ideal to work for, but my experience was largely positive, and the difficulties I encountered had more to do with the “frugality” aspect of Amazon’s culture - most notably in running “lean” teams that do more work per person than at most other companies. But I was proud to be a part of the groups I worked for - I did some of the best work of my career (so far) in those groups - and I left the company on good terms and would consider going back in the future.

I won’t tell you not to check out that “FACE of Amazon” site, but keep in mind, it appears they have posted only universally negative stories there, and thus the site seems to suffer from confirmation bias. Their language and their use of an anonymous email system suggests that they are not interested in positive stories for balance. If you’re concerned about what it would be like to work at Amazon, I would encourage you to look also at sites like Glassdoor to view ratings from a more representative sample of current and former employees, as well as by job type and title.

In my experience, Amazon has very high standards for its employees, and it CAN and WILL burn you out if you let it. However, they do also respect you when you tell them what your limits are - if you can still do your job, you should not fear letting your manager know that you can only work 40 hours a week.”


About the Author

Matt Kellner

Software Engineer, Video Game Historian and Part-Time Farmer

Software Engineer1997-present

Studied at California Polytechnic State University, San Luis Obispo

Lives in Puget Sound

[1] Merriam-Webster Dictionary,   

[2] From my "Letters to a Young Manager," series, 

[3] The earliest comment is from Dec. 8, 2016, after expanding all the comments, here: . The original question posting does not appear to be available on

[5] See the comment by Heck Evergreen, Senior Software Dev Engineer in 3/10 Top Companies (2005-present), answered Jun 4, 2017

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."

Wednesday, December 4, 2019

Lessons from Disaster Response

An Episcopal priest on a podcast is talking about families and growing up.  She notes the inevitable frictions that occur.  These frictions, or things going wrong, is an opportunity to grow.  In fact she says that if there were no friction, there would be no growth, because we would just stay the same.[1]

For the past twenty years I have been immersed in technology and organizations that use it to provide disaster relief.  Disaster planning is something that every IT leader needs to worry about.   This is more and more true as our dependence on our systems to get things done grows.  But I’m not talking about this kind of disaster planning, which has a natural parallel to preparing for crises in the disaster prone parts of the world.  

I’m interested in what we can learn from large-scale disaster response that we may apply to the running of IT itself.  What are the IT leadership lessons we can glean from disasters.  What is the growth we can experience from rubbing up against these frictions?

The obvious place to start is to recognize that disasters are inevitable. The planning in relief organizations is not about if a disaster occurs, it’s about when.  The question we are asked in disasters is: are we good enough to respond fast enough to save lives and help people bounce-back.

The first lesson is about speed.  In a disaster, the number one priority is speed.  Having a well-designed process that takes a week to deploy won’t cut it. People are hurting now.  Response  needs to be in hours, not weeks or months.  And relief organizations, both in government and non-government forms, need to get faster each time; if they are to grow in impact, they must learn from the frictions.

 When I first joined the International Red Cross/Red Crescent in Geneva as the Head of IT, I first developed some goals and a strategy for change.  It’s one of the top reasons a new CIO is hired, to make change happen.  I drew the analogy to the Queen Mary ocean liner, this large ship designed to move a large group of people and process as surely and safely as possible. But now we needed to work together to turn it around in the Rhône River because the river around us is moving in a new direction. And we need to catch up. That means acting more like speedboats than ocean liners.  

The second lesson is about good enough. Doing things right is another top concern of relief organizations.  But the speed demanded of us counts more than the quality.  A colleague commented during the Indonesia Tsunami response that we didn’t have time for all the meetings and discussions involved in good business as usual in an NGO.  We needed to make decisions on the spot.  And, she noted, nothing fell apart.  The good enough way worked.

We often design systems to handle each and every contingency we can think of.  That’s the way of deep knowledge organizations.  It’s also the way of organizations who are accustomed to doing things a certain way. As a result, we customize vendor systems to handle the exceptions without first criticizing the current processes.  We fall victim to what Michael Hammer called “paving the cow path.”  Most commercial off-the-shelf systems are good enough.

The third lesson is to build bounce-back.  Resiliency in communities is the ability to bounce-back after a disaster.  Working with vulnerable communities we are constantly reminded of the resiliency in the human spirit.  A positive outlook can go a long way.  But so can systems that can bend and change under stress.  The reeds bends more than the oaks in a storm.  This also applies to projects.  A resilient project is one that adapts to discovery along the way.  Agile approaches get at this.  So does chunking things down to small components.  If you want to bounce back from a project failure, have smaller project phases that are cheaper to throw-away and begin anew.  

A question I often ask my students is, "Does your system fail gracefully?"  The implication is that it will fail. Forty years of IT experience has taught me that IT failures are not a matter of "if", but rather "when".  How will your app behave when it fails?  Will it help the user recover with a meaningful error message that goes the extra yard of what action the user may want to take next?  That's a resilient system.

We can also apply this to a relatively new area of IT, autonomous vehicles or self-driving cars.  The question is not about how well the systems perform and learn to perform better in avoiding accidents.  It’s about planning for the inevitable failure of the software and hardware from time to time.  That takes imagination and humility.  How do you design for that?  Expect the storm, and adjust with grace, hope and a human touch.

[1] Barbara Crafton, “Sunday Sermon,” June 22, 2014, Trinity Church Boston,  She says “The way we grow primarily is by friction; you don't really grow much in the sunny meadows of sweet harmony; they mostly help you stay the same.  Where you grow is by bumping up against each other.  Where you grow is by rubbing up against each other in an uncomfortable way.”  [7:24 - 7:47]

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."

Tuesday, October 1, 2019

Good Follow-ship

During a recent call, a colleague lamented that employee engagement was at an all time low, with 70% not engaged at work (Gallup), and yet there are over 60,000 books on leadership (she Googled Amazon).  She went on to say, if so few employees are engaged then we are failing at leadership (and not learning from all these books).   Are managers getting bad advice? Probably not, but as we discussed, the lack of soft skills, modeling behaviors, that encourage connections between people is the culprit.

I learned early as an adult the hard way, and often the best way, of failing at communication.  I found that communication has two parts, a message and a reception, and without both, there is no communication.  I’ve spent half my professional life working on problems of digital communications across far flung organizations, especially helping bridge the digital divide in emerging countries.   When two people make a connection, it’s a beautiful thing to witness the budding conversation.  But getting connected is no guarantee that communication can occur. 

Why is that?  While a message and a connection are required, there can be no person-to-person communication without listening, understanding and empathizing.  And that’s not a one-way street; it’s about both listening more deeply, both seeking to understand the other, and putting yourselves in each other’s shoes.  That’s where the most engaging conversations, what David Whyte calls the courageous conversations, can begin.

This person-to-person communication is not just something that leaders in an organization can do, it is also for the followers, who make up most of the organization, must do.  It means being a better follower, and practicing what I call good follow-ship. 

A brief story may help...

A few summers ago my wife and I traveled to Cairo to be a judge in the Imagine Cup student competition. While we were there, we made plans to meet with a colleague who offered to show us his home country, the "real" Egypt he said. Farouk was one of our long-term Field Office Regional Tech's. He reported up through my US headquarters IT group.

As is the custom of his country, Farouk extended a hospitality that is rare in my part of the world. He took us on a tour of the old city in Cairo, we visited the Egyptian Museum, and we traveled to Alexandria. He made all the arrangements and would take no more than our thanks in return. It was humbling.

The traffic in Cairo is, by western standards, insane. There are few traffic lights in the city, and no one pays any attention to them. The painted lines on the street and highways are at best guidelines; if four cars can fit in three lanes, they do. One evening in Cairo we parked near the edge of the old city and walked to dinner. At the first main street, we experienced the drivers of Egypt, up close and personal. How were we ever going to cross this street? My New York instincts were to look for a gap in the traffic, and run for it. But there were no gaps.

Though I was the boss, Farouk took charge. "Hold my hands," he said, "follow my lead, and don't look!" It was a strange experience, a throw-back to early childhood, grabbing Dad's hand before crossing the street; depending on him to get us safely across. "Go now," he said, taking five steps forward and stopping, then five more. Cars were swerving around us like a river around three rocks. "Hold on," he admonished, "do what I say; now go." In a dance I did not understand, he guided across the sea of chaos, to the other side.

When we caught our breath, and heart rates slowed down, I asked him how he got us across. In New York, we would have been killed. But these were Cairo rules. "When you step out," he said, "the drivers must take responsibility not to hit you." "...but you need to know when to step out," he added.

This story was a lesson I'll never forget, precisely because I needed to forget. I had to put aside my experience and preconceived notions of how to cross a busy street, and trust someone else to guide me through their country's rules. Letting others lead you and teach you is part of becoming a good leader. It is especially true of learning about other cultures--we will never get it as well as those who have it in their blood. This also applies to our areas of expertise. Sometimes we need to practice good follow-ship.

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."

Wednesday, May 22, 2019

Why ERP's are wrong for NGOs

The first edition of this paper was written in May 2012 and updated in May 2016.  It has since been a part of the IT Think Papers on my web site.  Though some of the figures have changed (some better and some worse), the conclusions still ring true for the nonprofits I continue to follow, now as faculty member of a university, where I teach IT leadership and management.  I’ve made further revisions to anonymize portions and rerelease it here.

Considering an Enterprise Resource Planning system (ERP, for short) is a natural event for nonprofit organizations at a certain stage of their development.  International NGOs (INGO) such as CARE, IFRC and Save the Children and UNICEF have had similar experiences with varying degrees of success and shortfalls.  Why are such run-the-business systems so difficult?  Here are some reasons to consider :
  1. Too expensive. One of the first UN organizations to complete an ERP project spent an estimated $20M+ to roll out SAP to their 120 country offices.  Even limited ERP projects, with nonprofit discounts, take $2M+ to complete.  80-90% of the costs are for business process and implementation consulting.
  2. Takes too long.  The typical ERP project at large NGOs takes 2.5 years or more to complete.  For 5-year strategic plans it means that the impact of the new system is limited.  One INGO  spent 3+ years to complete its donor ERP, from Blackbaud, missing its strategy window.  Confounding this is that technology changes and user needs are evolving faster than 3-year ERP development cycles, ensuring that the system once released will likely be out of date.
  3. Failure rate is too high. According to the well-known Standish Chaos database, fully two-thirds of large IT projects fail.  One-third fail outright, while one-third are over time or budget.   A recent McKinsey report verifies this .  Smaller, chunked-down IT projects have higher success rates.
  4. Don't meet financial objectives. The Nucleus ROI study found 57% of large SAP projects (a leading ERP) don’t achieve the ROI cited in the beginning as the justification for doing the project in the first place.   Furthermore, an analysis of 100 corporations that have implemented SAP shows that this group has significantly lower profits than peers. 
  5. Too hard to change. The pain and effort to upgrade business processes and customize the ERP system results in a system that is resistant to change, in terms of time, cost and will. As a result the ERPs that often replaces a legacy system (or three), becomes itself a new legacy system.
  6. Not optimized for the web. Many of the large ERP systems, like SAP, have not been optimized for the web and browser world.  Web features have often been bolted on to antiquated back-office architectures. Web users notice the difference from popular web apps immediately.
  7. Almost impossible to please all departments. Given the enterprise nature of ERP systems, they cut across many departments.  Getting many departments to agree on features and functions often means a tendency to lower common denominators of need rather than an optimal solution for an individual department needs.  And yet each department wants its needs met, which often results in a high degree of customization.
  8. Expensive to customize.  The short-term cost to customize large ERP systems is high, with as much as 50% of the project costs going to the analysis, definition and development of components.  However, the longer-term cost may be even higher, especially if the customizations are many.  Each upgrade to the system that the vendor provides will most likely require an upgrade to the customized components, at the organization’s cost.  This often results in a lagging behind the current market system, which gets worse over time.
  9. User satisfaction is lower than for SaaS applications.  Applications that are built for a web-centered world are more likely to meet user’s expectations for how modern applications should work.  Users increasingly come to the organization with a strong experience base of using web-based applications in their education and personal lives.  ERP systems often do not measure up and don’t function in a browser as users expect them to work.
  10. Many INGOs have a poor record of implementing large systems.  At one such organization, an HR SAP project, Logistics project and Web Reconfiguration project were all cases in point where large applications were over-budget and over-time estimates by significant margins. One  could take the approach of investing more in getting big systems right, or one could take the approach of chunking projects down to more fit-for-purpose solutions.  Fortunately, the organization chose the latter.
  11. Not share-able with Field Offices.  For ERPs to be shareable across organizations, their customizations often need to be repeated for different country/location needs.  There is no multi-tenant model for ERPs that is comparable to SaaS applications, where each feature enhancement is made to the core code available to all.
  12. Total cost of ownership (TCO) of ERP’s is affordable by only the largest organizations. For-profit organizations typically have 5 times more IT spending power per employee than large nonprofits .  With 25% of systems cost as a typical annual maintenance and support estimate, a $1M system can generate $250K in recurring costs, and this figure rises with the degree of customization.  Over five years, the TCO can more than double the cost of a project.  My advocacy position--with a bit of intentional hyperbole-- is that traditional ERPs will bankrupt Nonprofits and field and country offices, customized ERPs even more so.
  13. The data integration model of ERPs is overrated.  One of the strongest selling features of an ERP is the shared database that underlies the system, providing the highest amount of systems and data integration.  However, the real-time integration needs of many organizations are rare.  If the data definitions are clear, the transfer of data among smaller applications is usually small enough for infrequent transfers of information.  Sometimes a manual transfer of data is sufficient.  Often a simple export and import of data will do. One-way integration may be needed.  Two-way integration should be avoided.  The key questions are: how often does the data change, how much of the data changes, and how many users are impacted?  A small scope demands a small solution.
As a result of these considerations, the strategic direction for applications at NGOs is clear.  The arguments are stronger for small, fit-for-purpose, web-based, loosely coupled applications.  A more agile organization is built on systems that are quick and easy to change, and faster to deliver.  Those applications that meet a specific need are more likely to succeed.  This is the most prudent use of NGO donors’ funding.

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."

Wednesday, April 17, 2019

Circuit Breakers

On October 19,1987 I was sitting at my desk in California when a colleague called from Chicago.  
“Have you seen the market news?” he asked.
“No...but I’m looking now”
“It’s fallen 20% and counting…”
I pulled up my market application on my PC and stared in disbelief.[1]

There were a variety of factors that caused the crash of ‘87.[2]  One was program trading, computer programs used by hedge funds and others to execute groups of trades directly and rapidly.  

In response to the crash, regulators developed policies known as “circuit breakers” so that stock exchanges could halt trading when large swings in prices crossed a threshold. They were essentially alert systems that watched the trading systems and took rapid action to pause. This temporarily stopped the chaos and allowed people to step in and restore order.

The metaphor of a circuit breaker comes from the common electrical circuit box in many homes.  If you’ve ever experienced a power outage, you know that sometimes running the toaster, dishwasher, and a space heater in the kitchen can overload the electrical circuit, cause the breaker-switch to “pop” and turn off the flow of current to that room.  That’s the breakers purpose: to stop the electricity flow from increasing to the point of burning up the wires (or the house!) And, as a colleague pointed out, the tripped circuit breaker is an important indicator that something went wrong that needs attention.

It’s evident why the circuit breaker name stuck for the “trading curb” systems that the Securities and Exchanges Commission (SEC) put in place following the market crash.[3]  I believe this is a useful metaphor for the current day and the concerns about runaway artificial intelligence (AI) programs.  

The key feature of this metaphor is that a rapid overload in the wrong direction can be automatically halted so human beings can step in to correct things.  It also implies a dash of humility and avoids the hubris of insisting we can build the failsafe system.

This is not fundamentally different from the “endless loop” interrupt or bottom of an else-if chain in programs.[4]  The graceful exit when unexpected situations occur is good design, as is the expectation that the system will at some point fail.  Most programmers would pick an orderly shutdown over a system crash.  How the system fails is important. This same expectation needs to temper our AI designs.  Self-healing and machine learning systems aside, a key interrupt in systems needs to be the human judgment interrupt.

What I’m suggesting is that our AI systems should be designed and programmed with this feature in mind.  When a system starts to cross a threshold that we’ve agreed is dangerous, the system halts and asks for some human judgment.  Defining those thresholds and boundaries may be hard work and such standards may be elusive, but as has been said in other realms, “I’ll know it, when I see it.”  The point is if we build the need for human judgment pauses into our systems --human interrupts, if you will, we just may avoid the system barreling forward at great speed and potential damage.

[1] At the time, I was Client Marketing Director for Lotus Signal, a PC-based real-time stock market application.  Lotus bought Dataspeed in 1985 primarily to get this technology.
[2] See the discussion in Wikipedia on “Black Monday (1987)”, .  Also see “Algorithmic Trading,”
[3] See “Trading Curbs”,
[4] The story I heard when learning to code was about the software engineer who put at the end of an exhaustive else-if chain the message “this can’t happen.” which of course was displayed one evening as the program merrily ran.  

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."

Monday, April 1, 2019

A Tale of Two Resumes

I review many students' resumes each semester, as part of the IT Leadership & Management course I teach, as an advisor to students immersed in career planning, and as a practitioner who has interviewed and hired many candidates.  The conversations that result convince me that having two resumes may help in your career search.

The first may be formatted to ease the AI review by an increasing number of applicant tracking system (ATS) apps HR departments are using to sort thru the many applicants.  This is often playing the game of matching phrases in your resume with keywords in the job description. For an interesting article on this, see beat the robots.

The second resume may apply more marketing intelligence to make your uniqueness pop. Since I arrived at UMSI, I've been encouraging the use of "about me" sidebars, and highlighting the impact you had in each job, as well as what you learned. As a manager, I care less about what you did than about the difference you made--on the organization as well as yourself.  For some interesting (some off-the-wall) and memorable designs, see the Piktochart site.  Start with the tamer #24, 40 and 44 for some ideas.

These two approaches may be two ends of the spectrum, but they recognize that (a) you are more than a collection of keywords, and (b) in a crowd of applications, standing out takes some creativity.

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."

Friday, August 10, 2018

Working Backwards

One of the hardest things for technology people to do is be succinct, replace the details with a story about benefits, and to ask for the sale.  After all, isn't this what marketing and sales people do?  Perhaps it's a surprise, but that is what you need to do if you hope to get a technology project, new product or new way of doing business, approved by a senior management team.

And please be brief!  It's not that senior managers have short attention spans, they need to get to the bottom line and make a decision.  Your job is to help them do that.

Each semester, I ask students to prepare a presentation for a mock senior management team.  Masters students know how to do the research and write the effective paper.  The goal of a senior management presentation is different.  It's about making the "ask," the bottom line decision you need them to make to proceed, and that's based on how much it will cost, how long it will take and how many people we need to commit to do it. Along the way, it's good to identify the benefits we will realize, the risks the organization will be taking to do this, and how you plan to mitigate those.  All of this should take less than 15 minutes with 10 slides plus Q&A.

Senior managers will want to know you are competent (what are your credentials and have you done this before) and that you've done your homework.  The 30-page report is impressive; it's not likely anyone will read it... beyond the executive summary.

A student team did a fine job talking about their research and the conclusions of their study.  When it came time for the Q&A, the CEO said, rather bluntly, "What the hell are you asking me to do?"  That's not the likely response to a presentation in an academic environment.  But it is what you should expect from a CEO, whether from a corporation or nonprofit organization.

A CEO told me once, "I can't use the students schools a graduating today; it takes me too long to retrain them."

"Retrain them how," I asked?

"To be able to talk with customers, manage a project with a diverse and dispersed team, and get to the bottom line of things!"

So to learn effective presentations, we start with the bottom line and work backwards.  That way, we are keeping the end goal in mind.  Here are ten slide headlines for a hypothetical project, in reverse order from an approved and funded proposal:

  1. The ask - what Senior Management Team decision are you asking for?
  2. Time and cost - how much time will it take to complete your project proposal and at what cost (one-time and recurring)?
  3. Risks - what risks are there to your project's success and what are the mitigations you see?
  4. Benefits - what benefits accrue from going with your recommendations; what's the value proposition?
  5. Your recommendation - Among the options, which are you recommending?
  6. Options - what are the alternatives you studied and the pros and cons of each
  7. Approach/method taken - how did you go about your investigation, sources consulted, tested conducted, etc.
  8. Assumptions made - what is the scope that you narrowed, what is your scenario?
  9. Problem - what problem are you trying to solve? what are the key questions you are trying to answer? 
  10. Intros - who are the members of your team and what are your credentials?

Everything else may be important, but it is appendix.  Now reverse the order.  Go.

"The postings on this site are my own and don't necessarily represent positions, strategies or opinions of any of the organizations with which I am associated."