The Unintended Consequences of Technology: Second Order Effects


Mark Smyth

“Every action has a consequence, and each consequence has another consequence… Be careful when making changes.” – Josh Kaufman

Frank Chen of Andreessen Horowitz spoke at the AI Summit in London in June 2018 on an area he calls “The Autonomy Ecosystem”1. The centrepiece of the presentation is the addition of a new artefact to tomorrow’s world – the self-driving car. The narrative to date around autonomous vehicles has predominately been focused on two significant consequences; human drivers being made redundant by technology and lives being saved through reduction in errors. In many instances, those two points are contrasted almost as cost and benefit. The debate of whether or how we proceed with the revolution of transport appears to rest on a simple net equation. Technology does not live in a vacuum however and the car touches a lot of ecosystems. As we come to understand the implications from the advent of new technologies, we should look towards the farther-reaching consequences and understand the role we have in shaping effects.

Snakes, Towers and Profanity

When we talk about action and consequence, there is a recurring theme of well-intentioned intervention resulting in an adverse outcome. The Cobra Effect describes an attempted solution making the problem worse. The anecdote goes that the British rulers in India, concerned about fatalities from venomous cobras, offered a reward for each dead cobra turned in. While initially successful in the slaughter of thousands of the predators, the laxity of the criteria began to be exploited by people who bred cobras in captivity to claim the reward. Upon the realisation that the system was being abused, the bounty was scrapped. This led to the bred cobras being released from captivity and the population increased above historic levels. While the cobra example lies closer to fable than fact, we know that something similar was did happen with rats in Vietnam under French colonial rule.

Unintended consequences can be positive as well – John Mansfield describes the building of the Leaning Tower of Pisa as an example2. Our first order effect is in the slant of the structure because of the poor foundations giving way. The second order effect, and unintended consequence, is that the Tower has become a popular tourist attraction which brings enormous revenue to the town.

Grounding the Cobra Effect in technology, the best example is probably the Scunthorpe Problem. The issue was named after an incident in 1996 in which AOL’s profanity filter prevented residents of the town of Scunthorpe in Lincolnshire, northern England, from creating accounts with AOL because of the substring. More recently, Google’s SafeSearch filters removed search results from local businesses containing Scunthorpe in their names.

Sociologist Robert K. Merton applied a systematic analysis3 to the problem of unintended consequences being created and listed 5 causes:

  1. Ignorance: Not understanding how the system works
  2. Error in calculation: Analytical errors using the information to hand
  3. Immediacy overriding the long-term: Focusing on short-term gain
  4. Basic values or principles: Being opposed to actions on moral grounds
  5. Self-defeating prophecy: Inadvertently change the circumstances to block the goal

The Autonomy Ecosystem

So what do the second order effects of the autonomous vehicle look like? Firstly, a brief precursor on the landscape of the future. Chen describes a city in which we have the fifth level of autonomous driving – there are no human drivers and everything is fully autonomous.4 As opposed to individuals owning their car, a fleet ownership model, or transportation-as-a-service, exists where people subscribe to use a pool of vehicles. Here are some of the second order effects:

  • Organ Donation: In the United Stated currently, 1 in 5 organ donations comes from the victims of road traffic accidents. An estimated 94 percent of motor-vehicle accidents involve a form of driver error. There would be significantly less road traffic accidents and accordingly, a significant shortfall in organ donations.
  • Insurance and Compensation: Your premium would be non-existent, or taken as a portion of your contribution to whichever pool you were a part of. Injuries Boards (in the UK & Ireland) and courts, where so many civil liability cases arising from traffic accidents are heard, see time freed up. Is there a possibility more opportunistic or fraudulent claims move to workplaces and public spaces instead?
  • Policing Resources: The amount of resources spent on traffic monitoring could be diverted elsewhere if speeding and drink-driving become redundant issues. Revenue stream from fines would dry up however. The burden of traffic accidents being removed also stands to significantly reduce the load on emergency services.
  • TV Ad Spend: Approximately 20% of primetime television and 10% of overall television advertising spend comes from cars and tangential products. Would collective ownership of autonomous vehicles decrease that or do they substitute current advertising as we know it for the various “car pools” consumers could pay to ride in? While the spending might not entirely vanish, it would certainly change in focus.
  • Film Industry: Hollywood might have to give some thought to what they do instead of car chase sequences in action films. The dynamic of driving off into the sunset isn’t quite the same either.
  • Traffic Management: In an entirely automated system, street directions could be changed in an instant to cater for traffic volume and direction. If you have seen the Road Zipper at work on Golden Gate Bridge, think of a similar model of changing traffic direction at a whim across the entirety of the road network. Congestion decreases relatively as routes are optimised. The response times of emergency services improves substantially as roads can be opened to navigate swiftly to the scene of an incident.
  • Land Usage: 14% of the land mass of Los Angeles is used for car parking. Given no one individual is using it exclusively and parking it at a shop/home, autonomous cars spend significantly less time stationary as they operate for pools. There are significant implications for the economics land usage from the increase in supply.

Finding Your Second Order Effects

Diagnosing your second order effects is challenging; if we could predict easily unintended consequences, there would be significantly less negative ones. Kaufman asks three questions about identifying second order effects:

  • What are the probable consequences of the system change you are considering?
  • What are the probable consequences of those consequences?
  • What can you do to minimise the probability and cost of unintended consequences?

Bigger Questions

For something seemingly as simplistic as the introduction of a singular piece of technology in the form of the self-driving car, there are a significant number of complications in the second order effects arising from it. We will see implications realised in the fields of medicine, insurance, film, urban planning and advertising.

Increasingly, we have a responsibility to be aware of the effects which technology implementation can have irrespective of intent. There will be significantly more complex ethical questions in the future and understanding the second order effects is of paramount importance. When considering the applications of the programming of ‘morals’ for artificial intelligence, the use of data to predict customer behaviour or the editing of DNA, we have far-reaching societal questions.

Emerging Technology in PPB

Part of our job as the Emerging Technology team is to understand and consider some of those second order effects to mitigate unintended consequences. While we discuss the comparatively novel implementation of a single technology, the consequences are increased by an order of magnitude depending on the complexity of the ecosystem it enters. The autonomous car is a case study which has been teased out and works as a convenient example for us to refer to. We will conceivably see robotic process automation, artificial intelligence, edge processing and the Internet of Things touch our industry over the next handful of years. As we look at emerging technologies being introduced, both on a micro and macro level, there is a responsibility on our part as implementers to understand what happens thereafter.

Sources and Further Reading:
  1. Frank Chen’s discussion is broken down into a series of videos available to view on the Andreessen Horowitz blog:
  2. Mansfield, John (2010) “The Nature of Change or the Law of Unintended Consequences: An Introductory Text to Designing Complex Systems and Managing Change”. World Scientific.
  3. Merton, Robert K. (1936) “The Unanticipated Consequences of Purposive Social Action.” American Sociological Review, Vol. 1, No. 6, pp. 894–904.
  4. Levels of autonomy in driving:
The Unintended Consequences of Technology: Second Order Effects

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s