überall or computer says no

A fascinating feature article by Frank Pasquale, writing for Æon magazine, called the Digital Star Chamber is an excellent articulation of some of the fears that harboured when comes the faith we’ve vested in the unaccountable algorithms and the predictive models that produce. The story is told through the lens of the new class of instaserfs who’ve entered into a contractual relationship, albeit voluntarily, that’s very much of balance and crushing for the entrepreneurial spirit.
Not only is the potential for mischaracterisation tremendous, not without human bias and prejudice but subject to the same slant as racial profiling and stereotyping despite being computer generated—the collected demographics and dossiers of markets have real world consequences beyond targeted advertisements, like the ability to get a loan or not due to the deportment of one’s neighbours or predictive purchasing habits, allowed on an aeroplane or into a foreign country, with whom one is mated, what kind of medical treatment that one receives, or even how one’s own shingle, enterprise, is framed. Such judgements were always there is formulae that companies employ for determining a good or bad risk, but now even the criteria themselves are obscured and there is no chance for appeal. Of course, in a broad sense algorithms must provide a a functional gauge and reliable measure, despite our instinct to style ourselves other than average, otherwise there would be no longevity to such routines and recipes and they are forever being manipulated in mysterious ways. It is nothing new or novel to compartmentalise individual behaviour for study, nor unfortunately is it a new development to blindly trust the results insofar as they are mechanical and supposedly non-judgemental.  Our reliance and deference, however, becomes very perilous considering how exposed we have allowed ourselves to become transparent and vulnerable as opposed to the voyeuristic and inscrutable number-crunching systems that stalk our every real and assumed step.  What do you think?  How ought such prowling agents be held to account?