Unintentional Consequences of What We Build

From IIW

Unintended Consequences of What We Build

Day/Session:Tuesday 5C

Convener:Annabelle Backman

Notes-taker(s): Annabelle Backman

Tags for the session – technology discussed/ideas considered:


Discussion notes, key understandings, outstanding questions, observations, and, if appropriate to this discussion: action items, next steps:

Two modes:

  • Negative human behavior applied to/within technology, i.e., badly behaving users
  • Technology that behaves badly itself, i.e., biased AI algorithms
    • This is technology codifying negative human behavior

Online disinhibition leads to worse human behavior

Do we have a responsibility to prevent users from themselves? No consensus.

  • Users can’t be responsible for something they don’t understand, e.g., massive Terms of Use.
    • Aggregating forces, snowball effects create pseudo-coercive effects; “opt-in” may not truly mean “opt-in”
  • Users should be held accountable for the contracts they agree to.
    • Users happily click through consent prompts then complain when their data is used in unexpected ways.
  • Regulation is expensive, inhibits startups and new competition.
  • Need to actually fix the way people think or don’t think.
  • Are we always responsible? Or never responsible?
    • Responsibility to advise and inform.

What happens when things go wrong? Example: Aadhar being misused, leading to fraud 

  • Similar to SSN; a lot of very useful properties, natural for people to use it
  • Need to consider if all the properties of a system/solution are desirable.