S12_ From principles to practice: a shared framework for responsive evaluation
S14_ Learning, Equity, and Innovation in EU Policy Evaluation: Methods and Practice for Vibrant Democracies [joint TWG 7 EU Policies and EEPF Proposal]
S12_ From principles to practice: a shared framework for responsive evaluation
S14_ Learning, Equity, and Innovation in EU Policy Evaluation: Methods and Practice for Vibrant Democracies [joint TWG 7 EU Policies and EEPF Proposal]
S13_ How is emerging artificial intelligence intersecting with evaluation and democracy? Towards new practice and adoption
Alix De Saint-albin, Bianca Montrosse-moorhead, Linda Raftree, Steffen Bohni Nielsen
Rationale and Objectives
Entire democratic ecosystems are being (re)shaped by emerging artificial intelligence (AI). Here ‘democratic ecosystems’ refer to the interconnected network of organizations, initiatives, technologies, and people working together to inform, strengthen, protect, and reform democratic practices, institutions, and infrastructure. AI is also revolutionizing evaluation -- an aspect of and for democratic ecosystems -- requiring us to re-examine fundamental evaluation issues.
Example questions include:
1. What is emerging AI and how is it used in evaluation?
2. What is ‘’Responsible AI” in evaluation, what elements does it include, and how does it intersect with democracy and democratic principles?
3. How is emerging AI reshaping evaluation supply and demand?
4. Where does professional judgment erode in AI-enabled evaluation, and how might a Responsible AI framework change this?
5. How are VOPEs responding to AI-driven change and how might they collaborate to address challenges related to emerging AI?
C. Several audiences will benefit (practitioners and consultants, including YEEs; researchers; and representatives from international organizations, national administrations, and NGOs).
This strand seeks contributions that share GenAI in evaluation case studies or case examples of use; discuss AI guidelines, charters, resources, etc developed or in development by VOPEs, evaluation organizations, or evaluation initiatives; or contribute to a shared understanding of responsible AI as it manifests within a range of evaluation theories and practices.
Preference will be given to proposals that explicitly demonstrate or discuss responsible AI, especially those that authentically include YEEs. ‘Responsible AI’ is understood as developing and using AI tools, products, and systems that are compliant with laws; uphold human rights provisions; are socially, culturally, and contextually sensitive; strengthen equity; uphold scientific standards; and strengthen evaluation as a profession and discipline.
Example questions include:
1. What is emerging AI and how is it used in evaluation?
2. What is ‘’Responsible AI” in evaluation, what elements does it include, and how does it intersect with democracy and democratic principles?
3. How is emerging AI reshaping evaluation supply and demand?
4. Where does professional judgment erode in AI-enabled evaluation, and how might a Responsible AI framework change this?
5. How are VOPEs responding to AI-driven change and how might they collaborate to address challenges related to emerging AI?
C. Several audiences will benefit (practitioners and consultants, including YEEs; researchers; and representatives from international organizations, national administrations, and NGOs).
This strand seeks contributions that share GenAI in evaluation case studies or case examples of use; discuss AI guidelines, charters, resources, etc developed or in development by VOPEs, evaluation organizations, or evaluation initiatives; or contribute to a shared understanding of responsible AI as it manifests within a range of evaluation theories and practices.
Preference will be given to proposals that explicitly demonstrate or discuss responsible AI, especially those that authentically include YEEs. ‘Responsible AI’ is understood as developing and using AI tools, products, and systems that are compliant with laws; uphold human rights provisions; are socially, culturally, and contextually sensitive; strengthen equity; uphold scientific standards; and strengthen evaluation as a profession and discipline.
Strand Overview
We will organize six sessions. Sessions 2 and 5 will be finalized through Call B.
1. “Level-Up”: Emerging AI and evaluation practice
Session introducing AI in evaluation basics, including current tools, practices, and implications
2. Living the tensions: Responsible AI evaluation case
Case studies to identify when and how AI in evaluation raises challenges for democratic principles, and how evaluators can respond to them using a Responsible AI framework
3. AI: A shock to the evaluation market?
Thematic panel to identify how and where AI is affecting evaluation supply and demand, and whether and how Responsible AI intersects
4. Evaluator competencies & capabilities: Where do we lose professional judgment in using AI?
Thematic panel to identify what evaluator competencies are impacted by AI, how AI forces evaluators to redefine their roles, judgment and responsibilities, and how a Responsible AI framing might guide professional standards and expected competencies
5. Professional evaluation organizations, groups, & initiatives: What is their evaluation + AI strategy?
Geographical case session to distill AI strategies developed by VOPEs, evaluation organizations, and evaluation initiatives and explore how a global conversation could advance shared frameworks for AI in evaluation
6. A world café on Responsible AI, evaluation, and democracy
Fishbowl to inform a Responsible AI in Evaluation strategy and potentially establish a global working group
Sessions form a coherent narrative exploring several interrelated themes (governance, ethics, practice, power, judgment, legitimacy) in service of a central concern: Will the international evaluation community shape AI use around these themes? Or will they allow those choices to be made for them by others?
1. “Level-Up”: Emerging AI and evaluation practice
Session introducing AI in evaluation basics, including current tools, practices, and implications
2. Living the tensions: Responsible AI evaluation case
Case studies to identify when and how AI in evaluation raises challenges for democratic principles, and how evaluators can respond to them using a Responsible AI framework
3. AI: A shock to the evaluation market?
Thematic panel to identify how and where AI is affecting evaluation supply and demand, and whether and how Responsible AI intersects
4. Evaluator competencies & capabilities: Where do we lose professional judgment in using AI?
Thematic panel to identify what evaluator competencies are impacted by AI, how AI forces evaluators to redefine their roles, judgment and responsibilities, and how a Responsible AI framing might guide professional standards and expected competencies
5. Professional evaluation organizations, groups, & initiatives: What is their evaluation + AI strategy?
Geographical case session to distill AI strategies developed by VOPEs, evaluation organizations, and evaluation initiatives and explore how a global conversation could advance shared frameworks for AI in evaluation
6. A world café on Responsible AI, evaluation, and democracy
Fishbowl to inform a Responsible AI in Evaluation strategy and potentially establish a global working group
Sessions form a coherent narrative exploring several interrelated themes (governance, ethics, practice, power, judgment, legitimacy) in service of a central concern: Will the international evaluation community shape AI use around these themes? Or will they allow those choices to be made for them by others?