The Morality of Rules
How can we apply our moral and ethical judgments to AI and machine learning?
Research - Psychology
@ MIT Media Lab
Duration
6 months
Jan 2019 - June 2019
This research project was part of my work as a research associate at the MIT Media Lab in partnership with MIT and Harvard psychology.
I worked with Dr. Sydney Levine, a postdoctoral scholar, in collaboration with Dr. Iyad Rahwan, Dr. Laura Schulz, Dr. Fiery Cushuman, and Dr. Joshua Tenenbaun
Tools used
Qualtrics
MTurk
Adobe Illustrator
Adobe Photoshop
R
Roles/Responsibilities
Experimental design
Survey design
Data analysis
Graphic/visual design
Overview:
To explore the mechanisms behind human moral and ethical judgments and decision making, we investigated a proposed guiding theory with the ultimate goal of being able to define a computational model in order to help advance AI and machine learning.
Similar work includes Moral Machine, run by the same research group from the MIT Media Lab
Associated & partner institutions
Waiting in Line
(When) is it okay to cut in line?
Waiting in line is more complicated that you think! How do we determine whether or not it is okay for someone to cut to the front and violate the “rules” of a line? How do we account for moral judgment in cases when the “rules go wrong”? This study begins to determine a utility calculus for an account of moral decision making by balancing utility gain of the line cutter against the utility lost of those waiting in line.
Background
Waiting in line?
It's so simple, right? Even ducklings can do it.
But maybe it is a little bit more complicated. Imagine standing in line to order food and someone just needs to ask the cashier for a napkin. Or you are in line for the bathroom but someone feels like they are going to throw up. You would probably let these people cut to the front of the line. In fact, if you think about it, there are a lot of cases in which you might let a person cut to the front of the line without waiting.
Rules of standing in line
By asking participants on MTurk open ended questions via a Qualtrics survey about lines and rules for waiting in line in a deli, we compiled a list of the most representative responses:
-
Generally, the line moves from front to back - the first person is helped, then the second, etc.
-
The first person who arrives in the deli gets their sandwich first. The second gets theirs second, and so on.
-
The rules of standing in line is to stand behind the earlier person in front of you. Wait in the line and move up as others move up until you are to speak to the cashier.
-
Each person has to wait their turn. They will each be served in order of arrival and they have to wait for the cashier to be ready.
-
You have to stay in your place, single file, and you cannot cut.
So, how do we figure out when it is okay to break these rules?
Proposed theory: Contractualism
an act is morally permissible if all the parties relevantly affected by
the act could reasonably agree to it (or at least not reasonably reject it)
In other words, we would find it okay for someone to cut to the front of the line on the basis of taking into account the interests of all the people who might be affected by the cutting such as other people in the line or in the room.
Outline
1. Do we indeed use agreement to gauge whether or not an action is okay when rules don’t account for it?
2. How do we know who is going to agree?
Questions
Goals
1. Determine permissibility based on agreement
2. Determine agreement based on utility calculus
Study Progression
Study 1a : Moral judgment of deli vignettes - is contractualism present?
Study 1b: Whose permission is needed for line cutting?
Study 2: Determining agreement of relevant parties
Study 3: Determining underlying factors to agreement
Study 4: Probing underlying factors w/ additional vignettes
Waiting in Line - Study 1:
Can we gauge if contractualism is present in moral judgments of line cutting?
Line scenarios
We first came up with three different scenarios where people stand in line and had potential for someone to cut in line. Though we planned to eventually include all of these scenarios in our study, we focused first on the deli scenario (people standing in line to order at the counter at a deli).
Visual aids for the study created in Photoshop
1. Deli
2. Airport bag check
3. Bathroom
Deli vignettes
We came up with 14 vignettes portraying instances in which someone would want to cut in line at the deli. I created gaphics for each vignette to use as visual aids during the study to facilitate understanding of context.
Examples of vignettes (view full list here)
Catering
Spoon
Family
Diabetic
A customer wants to ask a series of questions about a catering order that he will pick up next week.
A customer who is eating soup at the deli dropped his spoon on the floor and needs another one.
Two children and their mother who is married to a customer who is currently placing an order with the cashier walk in.
A customer walks in who is diabetic and urgently needs sugar
Study Format
Goal
Gauge permissibility and anger levels of for each vignette (indicative of agreement regarding the cutting)
To probe the presence of contractualism, which hinges on the agreement of relevant parties, we chose to measure levels of anger and permissibility, which based on previous literature serve as an indicator of agreement and therefore the beginnings of contractualism.
Question format
Participants, who were recruited via Amazon Mechanical Turk, were presented with a Qualtrics survey. For each of the 14 vignettes, they were asked either about the permissibility of the cutting or how angry it would make them, not both. This ensured that our data for each measurement was independently collected from the either, avoiding confounding factors.
Imagine that there are a few people who are waiting in line at a deli to order sandwiches for lunch. There is only one person (the cashier) working at the deli.
A customer who is eating soup at the deli dropped his spoon on the floor and needs another one. Is it OK for that person to ask the cashier for a new spoon without waiting in line? (Yes/No)
How angry would you be if that person asks the cashier for a new spoon without waiting in line? [0 -100]
Results & Insights
1. Confirmed adequate variety of vignettes (range of level of permissibility and anger levels)
2. Permissibility (OK Judgment) and anger seem to have an inverse relationship
(the more angry it makes you, the less OK it is to cut)
OK Judgment/Permissibility
0 = No (not OK)
1 = Yes (OK)
Anger
0 = Not angry at all
100 = Extremely angry
Data analyzed and graphed in R
Waiting in Line - Study 2:
Whose permission is needed for someone to cut?
Study Format
As a follow up to study 1, we wanted to figure out whose permission participants thought would be needed for someone to cut in line. This would help us determine the relevant parties, furthering our understanding the role of contractualism in these contexts.
Participants were presented with the same vignettes and either the anger or OK question. They were also presented with an extra open-ended free response question:
"whose permission would be needed for that person to cut in line?"
Results & Insights
Most relevant parties
Top 3 responses (out of 448 responses):
1. “Cashier”: 102 / 448 = (22.77%)
2. “Every”/“all” people in line: 38 + “people in front”: 18 = 56 / 448 (12.5%)
3. “Front”of the line = 35 / 448 (7.81)
“People in front” : 18 / 448= 3.69%
“(First)Person in front”: 17 / 448= 3.79%
Waiting in Line - Study 3:
Is agreement used to judge permissibility of cutting?
Study format
We used these three relevant parties as the options in our next iteration of the survey which asked participants if these parties would agree to letting the person cut in line, trying to probe underlying contractualism reasoning directly.
By analyzing the correlation between these agreement responses and permissibility and anger responses, we can begin to answer out first question of whether or not agreement is used in judgments of when it is ok to violate the rules of the line.
Results & Insights
1. Across all three parties, agreement is very strongly and significantly correlated with both anger level and permissibility.
2. Negative relationship with anger: more agreement from parties = less anger about cutting
3. Positive relationship with permissibility: more agreement from the parties = higher permissibility of the cutting
What does it mean?
1. Strong evidence for our theory of contractualism!
2. Suggests that agreement plays a part in judging exceptions that the rules do not account for
3. Going forward, we will just be using the OK/permissibility judgement since OK and anger are so strongly and tightly correlated
Waiting in Line - Study 4:
What are the underlying factors driving contractualism?
Where we are
1. Do we indeed use agreement to gauge whether or not an action is okay when rules don’t account for it? ✅
> Determine permissibility based on agreement ✅
2. 📍 How do we know who is going to agree? ⬅️ ⬅️ We are here
> Determine agreement based on utility calculus
Initial questions:
Study format
In order to investigate how we determine agreement, we hypothesized several underlying factors that may contribute to the agreement judgment:
1. Delay time caused by the cutting
2. Effect (well-being) on the entire line
3. Well-being of the first
4. Well-being of the middle person in line
5. Well-being of the last person in line
6. Well-being of the cutter
7. Well-being of everyone in the deli.
We asked participants to answer these questions for the 14 vignettes:
Imagine that there are a few people who are waiting in line at a deli to order sandwiches for lunch. There is only one person (the cashier) working at the deli. A customer who is eating soup at the deli dropped his spoon on the floor and needs another one. This customer decides to cut to the front of the line.
1.How long would the first person in line be delayed? (seconds)
[free response/ fill in blank]
2.Think about the well-being of all the people in line combined. How would they be affected if that person asks the cashier for a new spoon without waiting in line?
[-50 (A lot worse off) to 50 (a lot better off) scale]
3.How much worse off/better off is the first person in line? [-50 to 50]
4.How much worse off/better off is the middle person in line? [-50 to 50]
5.How much worse off/better off is the last person in line? [-50 to 50]
6.How much worse off/better off is the person who cut in line? [-50 to 50]
7.Think about the person who cut in line. How much worse off/better of would it be for people who come to the deli if everyone who was in this situation cut in line? [-50 to 50]
Results & Insights
Correlations between chosen factors with OK judgment
(middle and last person correlations were essentially the identical to that of the first person and are not displayed)
Based on these correlations, we selected the factors that seemed fairly distinct from one another (e.g. not as correlated with each other as the first, middle, and last person in line were) and that we were also fairly certain that we could systematically manipulate
Factors/predictors of interest:
1. Delay time
2. Benefit to the cutter
3. Benefit to the line (as a whole -- to encompass first, middle, and last person, since they were so highly correlated)
Intermission
At this point, we reevaluated the current vignettes. After looking at how our current vignettes ranked for each of the three factors, we decided to add several more that we believed would score near the extremes of each predictor (high and low delay time, high and low benefit to cutter, high and low benefit to the line)
Click here to view the full list of original and added vignettes
Waiting in Line - Study 5:
How strong are these predictors?
Study Format
With the original vignettes and new ones added, we asked about the three predictors.
One notable change that we made is that we asked about the effect on the line imagining there was NO DELAY in an attempt to separate these two factors; delay and effect on the line.
In addition to these questions, we also asked once again if it was OK or not for the person to cut, just to confirm the correlation with permissibility.
Results & Insights
Correlations between three predictors and the permissibility
1. The well-being of everyone in line has the strongest and most significant correlation with permissibility.
2. Delay time is the second strongest and most significant.
3. Well being of the cutter does not have significant correlation.
At this point, I left the research group upon graduation and relocation.
Next steps are discussed below. Research is ongoing.
Waiting in Line - Next steps:
How can we dig deeper?
Further analyses
Conduct analyses such as correlations with different combinations of predictors in addition to each of them separately. This would get us closer to defining an accurate utility calculus and computational model of contractualism in moral judgment of rule violations.
Line length
Test with different line lengths to see what effect that may have on judgments. Do patterns change as lines get longer or shorter? Does contractualism have limits? This is important in order get as a full of an understanding of moral judgments and decision making as possible.
