automatingsociety.algorithmwatch.orgAutomating Society Report

2020 Menu /// Policy Recommendations Research Belgium Denmark Estonia Finland France Germany Greece Italy Netherlands Poland Portugal Slovenia Spain Sweden Switzerland United Kingdom Stories Belgium Denmark Estonia Finland France Germany Greece Italy Netherlands Poland Portugal Slovenia Spain Sweden Switzerland United Kingdom Team Press Events Videos 2020 Introduction Policy Recommendations European Union Automating Society 2020 – Launch Event Belgium Denmark Estonia Finland France Germany Greece Italy Netherlands Poland Portugal Slovenia Spain Sweden Switzerland United Kingdom Download PDF (20MB) INTRODUCTION Life in the automated society: How automated decision-making systems became mainstream, and what to do about it by Fabio Chiusi On a cloudy August day in London, students were angry. They flocked to Parliament Square by the hundreds, in protest – their placards emblazoned with support for unusual allies: their teachers, and an even more unusual target: an algorithm. Due to the COVID-19 pandemic, schools closed in March in the United Kingdom. With the virus still raging throughout Europe over the summer of 2020, students knew that their final exams would have to be canceled, and their assessments – somehow – changed. What they could not have imagined, however, was that thousands of them would end up with lower than expected grades as a result. Students protesting knew what was to blame, as apparent by their signs and chants: the automated decision-making (ADM) system deployed by the Office of Qualifications and Examinations Regulation (Ofqual). It planned to produce the best data-based assessment for both General Certificates of Secondary Education and A-level results, in such a way that the distribution of grades follows a similar pattern to that in other years, so that this year’s students do not face a systemic disadvantage as a consequence of circumstances this year”. The government wanted to avoid the excess of optimism that would have resulted from human judgment alone, according to its own estimates : compared to the historical series, grades would have been too high. But this attempt to be as far as possible, fair to students who had been unable to sit their exams this summer” failed spectacularly, and, on that grey August day of protest, the students kept on coming, performing chants, and holding signs to express an urgent need for social justice. Some were desperate, some broke down and cried. Stop stealing our future”, read one placard, echoing the Fridays for Future protests of climate activists. Others, however, were more specifically tailored to the flaws of the ADM grading system: Grade my work, not my postcode”, we’re students, not stats”, they read, denouncing the discriminatory outcomes of the system. Finally, a chant erupted from the crowd, one that has come to the future of protest: Fuck the algorithm”. Scared that the government was casually – and opaquely – automating their future, no matter how inconsistent with their skills and efforts, students screamed for the right not to have their life chances unduly affected by bad code. They wanted to have a say, and what they said should be heard. Algorithms are neither neutral” nor objective” even though we tend to think that they are. They replicate the assumptions and beliefs of those who decide to deploy them and program them. Humans, therefore, are, or should be, responsible for both good and bad algorithmic choices, not algorithms” or ADM systems. The machine may be scary, but the ghost within it is always human. And humans are complicated, even more so than algorithms. The protesting students were not as naive as to believe that their woes were solely the fault of an algorithm, anyway. In fact, they were not chanting against the algorithm” in an outburst of technological determinism; they were motivated by an urge to protect and promote social justice. In this respect, their protest more closely resembles that of the Luddites. Just as the labor movement that crushed mechanized looms and knitting frames in the 19th Century, they know that ADM systems are about power, and should not be mistaken for being an allegedly objective technology. So, they chanted justice for the working class”, asked for the resignation of the Health Secretary, portrayed the ADM system as classism at its finest”, blatant classism”. Eventually, the students succeeded in abolishing the system which put their educational career and chances in life at risk: in a spectacular U-turn, the UK government scrapped the error-prone ADM system and utilized the grades predicted by teachers. But there’s more to this story than the fact that the protesters won in the end. This example highlights how poorly designed, implemented, and overseen systems that repro-duce human bias and discrimination fail to make use of the potential that ADM systems have, such as leveraging comparability and fairness. More clearly than many struggles in the past, this protest reveals that we’re no longer just automating society. We have automated it already – and, finally, somebody noticed. From Automating Society to the automated society When launching the first edition of this report, we decided to call it Automating Society”, as ADM systems in Europe were mostly new, experimental, and unmapped – and, above all, the exception rather than the norm. This situation has changed rapidly. As clearly shown by the many cases gathered in this report through our outstanding network of researchers, the deployment of ADM systems has vastly increased in just over a year. ADM systems now affect almost all kinds of human activities, and, most notably, the distribution of services to millions of European citizens – and their access to their rights. The stubborn opacity surrounding the ever-increasing use of ADM systems has made it all the more urgent that we continue to increase our efforts. Therefore, we have added four countries (Estonia, Greece, Portugal, and Switzerland) to the 12 we already analyzed in the previous edition of this report, bringing the total to 16 countries. While far from exhaustive, this allows us to provide a broader picture of the ADM scenario in Europe. Considering the impact these systems may have on everyday life, and how profoundly they challenge our intuitions – if not our norms and rules – about the relationship between democratic governance and automation, we believe this is an essential endeavor. This is especially true during the COVID-19 pandemic, a time in which we have witnessed the (mostly rushed) adoption of a plethora of ADM systems that aim to contribute to securing public health through data-based tools and automation. We deemed this development to be so important that we decided to dedicate a preview report” to it, published in August 2020 within the scope of the ‘Automating Society’ project. Even in Europe, when it comes to the deployment of ADM systems, the sky is the limit. Just think of some of the cases introduced in this report, adding to the many – from welfare to education, the health system, to the judiciary – that we already reported on in the previous edition. In the following pages, and for the first time, we provide updates on the development of these cases in three ways. Firstly, through journalistic stories, then, through research-based sections cataloging different examples, and, finally, with graphic novels. We felt that these ADM systems are – and increasingly will be – so crucial in everyone’s lives that we needed to try and communicate how they work, and what they actually do to us, in both rigorous and new ways, to reach all kinds of audiences. After all, ADM systems have an impact on all of us. Or at least they should. We’ve seen, for example, how a new, automated, proactive service distributes family benefits in Estonia. Parents no longer even need to apply for benefits: from birth, the state collects all the information about each newborn and their parents and collates it...