The path to qual­i­ty cri­te­ria for the use of algo­rithms

Our #algo­rules project, in coop­er­a­tion with the Ber­tels­mann Stiftung, has been run­ning for sev­er­al months – time to take stock. The goal is to devel­op a cat­a­logue of qual­i­ty cri­te­ria for algo­rith­mic process­es, which can serve as a ref­er­ence point for com­plaints and reviews.

Here you can get an overview of our past, present and future activ­i­ties.

What has hap­pened so far:

Research and work­ing paper on exist­ing qual­i­ty cri­te­ria cat­a­logues
There is no need to rein­vent the wheel. Espe­cial­ly in the USA, numer­ous orga­ni­za­tions have already attempt­ed to estab­lish qual­i­ty cri­te­ria for algo­rithms or to update exist­ing cat­a­logues of cri­te­ria. In the first phase of the project we thus asked our­selves, what approach­es to qual­i­ty cri­te­ria for algo­rithms already exist and what can we learn from them for fur­ther devel­op­ment?

We exam­ined three sets of prin­ci­ples: the FAT/ML Conference’s “Prin­ci­ples for Account­able Algo­rithms and a Social Impact State­ment for Algo­rith­mus”, the Future of Life Institute’s “Asilo­mar AI Prin­ci­ples”, and the ACM US Pub­lic Pol­i­cy Council’s “Prin­ci­ples for Algo­rith­mic Trans­paren­cy and Account­abil­i­ty,” and found that a large num­ber of use­ful qual­i­ty cri­te­ria for algo­rithms are already in place, from trans­paren­cy and fair­ness to respon­si­bil­i­ty and trace­abil­i­ty. What is lack­ing, how­ev­er, is an exam­i­na­tion of the spe­cif­ic eth­i­cal ques­tions raised by the relent­less use of algo­rithms in more and more areas of life, as well as con­crete data on the imple­men­ta­tion of any set of prin­ci­ples or cri­te­ria.

A work­ing paper (avail­able here) sum­ma­rizes the results of our efforts.

The first expert work­shop in Berlin and the ini­ti­a­tion of the #algo­rules process
On the basis of these find­ings, we cre­at­ed a draft cat­a­logue of qual­i­ty cri­te­ria. An expert work­shop was con­vened in Berlin on 25 May, 2018 in order to dis­cuss the con­tent and direc­tion of our pro­pos­als, which prof­it­ed from the shar­ing of var­i­ous per­spec­tives and expe­ri­ences. Algo­rithms are used by the pri­vate and pub­lic sec­tors in a wide range of fields, from com­mu­ni­ca­tions ser­vices, finan­cial risk assess­ment and tar­get­ed adver­tis­ing to health and police appli­ca­tions and edu­ca­tion. At the same time, the devel­op­ment process of algo­rith­mic sys­tems con­sists of many phas­es, each of which involves very dif­fer­ent actors. Meet­ing with key stake­hold­ers from a wide range of fields also marked the offi­cial start of our work on the #algo­rules. The par­tic­i­pants pro­vid­ed a num­ber of impor­tant impuls­es for the fur­ther devel­op­ment of the cat­a­logue, both for the pre­am­ble and for the cri­te­ria them­selves.

Since the con­clu­sion of this first work­shop, our cat­a­logue now con­tains over a dozen qual­i­ty cri­te­ria. From proven tools such as impact assess­ments and trans­paren­cy and account­abil­i­ty mech­a­nisms to basic prin­ci­ples such as fair­ness and sus­tain­abil­i­ty, we are work­ing on a coher­ent set of prac­ti­ca­ble require­ments that will now be fur­ther devel­oped and refined.

Consul­ta­tion process
The next phase in the devel­op­ment of the #algo­rules cat­a­logue con­sists of a con­sul­ta­tion process. We are cur­rent­ly in the process of inter­view­ing around forty high-rank­ing experts from sci­ence, pol­i­tics, busi­ness, civil soci­ety, media, and other rel­e­vant fields. Not only do we want to gath­er valu­able input for improv­ing and focussing the con­tent of our cat­a­logue. The expe­ri­ences of con­sul­tants from a range of dif­fer­ent sec­tors will also prove indis­pens­able to our imple­men­ta­tion strat­e­gy. Which require­ments must such a set of qual­i­ty cri­te­ria ful­fill in order to be accept­ed by var­i­ous actors in their respec­tive fields? Who are the impor­tant mul­ti­pli­ers? How can the cat­a­logue be com­bined with exist­ing pro­fes­sion­al eth­i­cal guide­lines? We antic­i­pate excit­ing answers to these ques­tions.

Pre­sen­ta­tion at expert round tables
We have also been gath­er­ing impor­tant feed­back by pre­sent­ing our project at rel­e­vant expert events and pan­els. For exam­ple, we recent­ly deliv­ered a pre­sen­ta­tion on the #algo­rules at a work­shop on smart assis­tance sys­tems host­ed by the Stiftung Neue Ver­ant­wor­tung (SNV), as well as at an event at the Hanns-Sei­del-Stiftung in Munich deal­ing with the use of algo­rithms in the process of TÜV tech­ni­cal cer­ti­fi­ca­tion.

Such for­mats offer us the oppor­tu­ni­ty to dis­cuss the project and the qual­i­ty cri­te­ria cat­a­logue with­in a larg­er group. In addi­tion, they allow us to get a feel­ing for the applic­a­bil­i­ty of the indi­vid­ual points in clear­ly defined con­texts, be it in med­i­cine or in con­nec­tion with con­sumer pro­tec­tion. We then incor­po­rate this feed­back in the fur­ther devel­op­ment of the cat­a­logue and for the devel­op­ment of an imple­men­ta­tion strat­e­gy.

Exper­tise on the com­mit­ment of suc­cess­ful pro­fes­sion­al ethics
Draw­ing on lessons drawn from exist­ing qual­i­ty cri­te­ria cat­a­logues, we hope to devel­op an effec­tive strat­e­gy for the prop­er appli­ca­tion of our #algo­rules. A close­ly relat­ed ques­tion, for exam­ple, con­cerns how any pro­fes­sion­al ethic or code can be made bind­ing. Why should respon­si­ble actors adhere to a more or less abstract col­lec­tion of require­ments? How can a cat­a­logue of qual­i­ty cri­te­ria for algo­rith­mic process­es become a bind­ing stan­dard, anal­o­gous, for exam­ple, to the press code for jour­nal­ists?

The authors of our work­ing paper “Ethics for Algo­rith­mists — What we can learn from suc­cess­ful pro­fes­sion­al codes” try get to the bot­tom of these ques­tions. Togeth­er, Alexan­der Fil­ipovic, Clau­dia Pagani­ni and Christo­pher Koska analyze six pro­fes­sion­al fields and their codes of ethics: med­i­cine, jour­nal­ism, social work, adver­tis­ing, pub­lic rela­tions and engi­neer­ing.

The authors iden­ti­fy ten suc­cess fac­tors for pro­fes­sion­al ethics and offer a prac­ti­cal ori­en­ta­tion on how they can be applied to the field of algo­rithm design. This is done against the back­ground of a detailed descrip­tion and his­tor­i­cal clas­si­fi­ca­tion of the grow­ing pro­fes­sion­al field of “algo­rith­mists.”

Cur­rent Activ­i­ties

Sec­ond expert work­shop in Berlin — devel­op­ing an imple­men­ta­tion strat­e­gy
In order to gath­er more feed­back on how our cri­te­ria could be imple­mented, we organized a sec­ond expert work­shop in Berlin in autumn, 2018. As with the first work­shop, we invit­ed rep­re­sen­ta­tives from dif­fer­ent sec­tors to gain as broad an insight as pos­si­ble into the devel­op­ment and appli­ca­tion of algo­rith­mic sys­tems. The dif­fer­ent per­spec­tives have proven help­ful in focus­ing and improv­ing the qual­i­ty cri­te­ria cat­a­logue and in devel­op­ing an imple­men­ta­tion strat­e­gy based on the results of the con­sul­ta­tion process and the rec­om­men­da­tions of Fil­ipovic, Koska and Pagani­ni.

Out­look

Broad par­tic­i­pa­tion process
By enter­ing a dia­logue with cho­sen experts from their fields in our work­shops and con­sul­ta­tions, we have laid a solid foun­da­tion for the devel­op­ment of our cat­a­logue of qual­i­ty cri­te­ria. In order for this cat­a­logue to achieve the nec­es­sary level of soci­etal accep­tance, it should also be sub­ject­ed to a broad­er debate. Thus, there is now a par­tic­i­pa­tion process in place that will involve a larg­er group of inter­est­ed peo­ple in the devel­op­ment of our cri­te­ria. We will con­tin­ue to pub­lish up-to-date infor­ma­tion about this here and on Twit­ter.

The #algo­rules process is in full swing. We look for­ward to con­tin­u­ing our coop­er­a­tion with the Ber­tels­mann Stiftung and all those involved and will keep you informed of our progress.

Back to arti­cle: Code of con­duct for the use of algo­rithms.