Harvey Rosenfield and Laura Antonini are consumer advocates.
Operating in the shadows of the online marketplace, specialized tech companies you’ve likely never heard of are tapping vast troves of our personal data to generate secret "surveillance scores” — digital mug shots of millions of Americans — that supposedly predict our future behavior. The firms sell their scoring services to major businesses across the U.S. economy.
People with low scores can suffer harsh consequences.
CoreLogic and TransUnion say that scores they peddle to landlords can predict whether a potential tenant will pay the rent on time, be able to “absorb rent increases,” or break a lease. Large employers use HireVue, a firm that generates an “employability" score about candidates by analyzing “tens of thousands of factors,” including a person’s facial expressions and voice intonations. Other employers use Cornerstone’s score, which considers where a job prospect lives and which web browser they use to judge how successful they will be at a job.
Brand-name retailers purchase “risk scores” from Retail Equation to help make judgments about whether consumers commit fraud when they return goods for refunds. Players in the gig economy use outside firms such as Sift to score consumers’ “overall trustworthiness.” Wireless customers predicted to be less profitable are sometimes forced to endure longer customer service hold times.
Auto insurers raise premiums based on scores calculated using information from smartphone apps that track driving styles. Large analytics firms monitor whether we are likely to take our medication based on our propensity to refill our prescriptions; pharmaceutical companies, health-care providers and insurance companies can use those scores to, among other things, “match the right patient investment level to the right patients.”
Surveillance scoring is the product of two trends. First is the rampant (and mostly unregulated) collection of every intimate detail about our lives, amassed by the nanosecond from smartphones to cars, toasters to toys. This fire hose of data — most of which we surrender voluntarily — includes our demographics, income, facial characteristics, the sound of our voice, our precise location, shopping history, medical conditions, genetic information, what we search for on the Internet, the websites we visit, when we read an email, what apps we use and how long we use them, and how often we sleep, exercise and the like.
The second trend driving these scores is the arrival of technologies able to instantaneously crunch this data: exponentially more powerful computers and high-speed communications systems such as 5G, which lead to the scoring algorithms that use artificial intelligence to rate all of us in some way.
The result: automated decisions, based on each consumer’s unique score, that are, as a practical matter, irreversible.
That’s because the entire process — the scores themselves, as well as the data upon which they are based — is concealed from us. It is mostly impossible to know when one has become the casualty of a score, let alone whether a score is inaccurate, outdated or the product of biased or discriminatory code programmed by a faceless software engineer. There is no appeal.
Surveillance scoring bears a faint resemblance to credit scoring in the 1960s. In that pre-computer era, private investigators working for banks, retailers and insurance companies tailed consumers and scoured newspapers for information about arrests, promotions, sexual orientation, drinking habits and cleanliness to decide a consumer’s creditworthiness –— until Congress established rules in the 1970s giving consumers the right to review and question their credit scores.
Today’s data snoops obtain infinitely more information about their targets, and in real time. And the impact of surveillance scoring is far more pernicious.
The tech industry insists that its every advance improves our lives. But that’s a myth. Surveillance scoring enables companies to cloak old-school discrimination in an aura of technological infallibility and wonder.
Secret surveillance scores create micro-markets in which some consumers are no longer welcome. They divide Americans into “haves” and “have-nots,” with the losers relegated to the status of second-class citizens.
Consumers need a 21st-century solution to this emergent threat. Congress, awash in tech money, is mired in an outdated legal paradigm: “disclosure” of privacy policies and “consent” via a click. No one pretends that these industrial age contract law concepts will do anything to curb data larceny, let alone regulate or bar secret surveillance scores.
We petitioned the Federal Trade Commission to investigate and regulate surveillance scoring. The commission’s response? A blog post urging the firms that develop and apply scores to regulate themselves.
This is only the latest example of Washington’s capitulation to the tech industry, whose continuous loop of privacy violations, abject apologies and payment of inconsequential penalties confirms that consumers cannot rely on the federal government for protection.
Secret surveillance scoring places us at the precipice of the “singularity,” a dystopian turning point after which machines will make judgments about humans that will determine our fate. We either seize control of our future, or risk losing it.
Read more:
Jeff Bezos: How did tech CEOs do on Capitol Hill? Google ‘robber barons.’
Alexandra Petri: Congress agrees big tech creepy, can’t agree how