Unintentional Consequences of What We Build
Unintended Consequences of What We Build
Notes-taker(s): Annabelle Backman
Tags for the session – technology discussed/ideas considered:
Discussion notes, key understandings, outstanding questions, observations, and, if appropriate to this discussion: action items, next steps:
- Negative human behavior applied to/within technology, i.e., badly behaving users
- Technology that behaves badly itself, i.e., biased AI algorithms
- This is technology codifying negative human behavior
Online disinhibition leads to worse human behavior
Do we have a responsibility to prevent users from themselves? No consensus.
- Aggregating forces, snowball effects create pseudo-coercive effects; “opt-in” may not truly mean “opt-in”
- Users should be held accountable for the contracts they agree to.
- Users happily click through consent prompts then complain when their data is used in unexpected ways.
- Regulation is expensive, inhibits startups and new competition.
- Need to actually fix the way people think or don’t think.
- Are we always responsible? Or never responsible?
- Responsibility to advise and inform.
What happens when things go wrong? Example: Aadhar being misused, leading to fraud
- Similar to SSN; a lot of very useful properties, natural for people to use it
- Need to consider if all the properties of a system/solution are desirable.