Black Box Algorithms & “Personalized” Services

Black Box Algorithms

Wednesday 1I Convener: Kris Alman

Notes-taker(s): Tom Leon

Tags for the session - technology discussed/ideas considered:

How do the algorithms that private companies are creating impact the balance of power and control of personalization vs person-centered applications in Education, Health, and other industries

Discussion notes, key understandings, outstanding questions, observations, and, if appropriate to this discussion: action items, next steps:

Resources:

Black Box Society – Frank Pasquale (http://www.greenapplebooks.com/book/9780674368279)

Audrey Watters – blog post “Identity, Education and Power” (https://medium.com/identity-education-and-power/identity-power-and-education-s-algorithms-a766527bb6cd#.vugr7k73q)

Initial Statement:

Man made algorithms that are making decisions on people (credit decisions, education decisions, etc.)

• Asking questions about what is happening, specifically in education
• There is a company called Knewton, who has teamed up with Pearsons, Houghton Mifflin, etc. These are companies that are known for creating curriculum; as more materials go online they need to close the loop on how to do assessments.
• In doing personalized learning, they are de-linking the teacher in the equation. Based on how they are doing, they are moving people (children/students) up and down the learning channel.

Education and HealthCare are deeply personal

• Education: FERPA and COPPA
  • DOE governs FERPA, FTC governs COPPA
• Affects where we go, what opportunities we have, etc.
• If there are black box algorithms that are becoming part of the big data brokers, you don’t know what info and decisions are being made.

Patient Privacy Rights: Org that looks at issues of HIPAA

• • HHS governs HIPAA
• How do we have access to our medical records and where do we go from there
• ? How do we deal with the privatization of these services that are integral to our futures?
• ? When you are talking about agency, children have little capacity to make their own decisions. How do they gain agency – can they?
• ? Is the concern that the power is being taken out of the teacher’s hands in guiding the student?
• What are the means that you as a parent/legal guardian have to address the student’s path through their course?

Kris gave and example of an algorithm affecting a pacemaker that is addressing AFib. Should people have information into how these work so that they can understand how CONCEPT: is this similar to Redlining where a person is deliberately excluded? How do you get out of this How do you know, how do you check? Who is profiling you and how do you change what is being profiled?

Whatever metadata might be created out of a company that is proprietary may include personality traits.

• Keystroking pacing
• How long does it take to accomplish something
  • The soft skill of “Grit” – does a student keep at trying to accomplish a goal
  • Businesses can do with this what they want, and the metadata is not governed by COPPA/FERPA/HIPAA

There is no Private Right of Action

• BlueCross/BlueShield example
• Customers of BCBS have no right to sue because of these breaches

EXAMPLE: There is health insurance that is regulated and other that is not regulated.

• Big Companies used ERISA (Employee Rights law)
  • They would go to United, Kaiser, Cigna, etc. and purchase directly with these companies as administrators
  • Because they did this, nobody is minding these plans offered by the big companies
  • ? If companies can gather information about their employees, they can use it against their employees?
  • THINK Tim Armstrong/AOL calling out the Million Dollar Baby
  • Do you get put on the list of people who cost too much?

? WHAT should we do about these black box algorithms?

Is it a technological issue?

Is it a policy issue?

Person-Centered vs Personalized

• How do you put the student at the center
• Need to humanize it – move it back to being Person-Centric and not just personalized

What does it mean to be “Personalized”?

• Is this a corporatized way to use language that sounds good on paper but is not really quantifiable?

Education: Is it working? Algorithm says that a student is doing well, but teacher disagrees

Should the box be open? Should people have insight into what is going on inside the box?

Respect and Transparency. They should have some insight into what the algorithms are expressing

Tools

Market Forces should give the power back to the parents/students/patients

There is a tension, an asymmetry

What is the key to Value. What are the problems, and how do they manifest themselves in the market? These need to be embedded.

There is a cultural aspect.

• Who are the stakeholders? Not just shareholders ~ Community
• Today this is not necessarily about just making money.

If the stakeholders (Parents/Patients) want to see something more Person-Centered

Need more options.

What if we took a different approach?

• If you started to talk about design principals and values that could be embedded in the product culture
• What would that look like for these different sectors – Education / Health Etc.

How can these laws of product designs be drafted and adopted by innovators?

How do we have corporations respond to stakeholders

• I am not sure about your Personalization; how do we move to Person-Centered

How do startups compete with the monopolies?

• The startups will start driving more Person-Centered services that respect the individual, that are transparent and are meeting the individual needs

CHALLENGE: How do we get buy-in from the stakeholder