About me ...
I am a third-year PhD candidate at the University of Amsterdam, supervised by Maarten de Rijke and Mohammad Alian Nejadi.
My research focus is on the evaluation of conversational systems, specifically from the user's perspective. I aim to improve user experience in their interaction with AI systems.
I am also interested in how to leverage LLMs for evaluation and studying how factors affecting humans in system evaluation, affect LLMs when used as evaluators.
Apart from that, I am an active member of Masakhane where I take part in promoting NLP for Low-resource African languages.
I completed my MSc degree from the African Institute for Mathematical Sciences - AMMI (Rwanda) supervised by Siva Reddy from McGill University. I received my BSc from Jomo Kenyatta University, Kenya.
Research Interests
User-centric evaluation of conversational systems
Evaluation of and with large language models
NLP for low-resource African languages
Publications
Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
SIGIR'24 conference
Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
PDFContext Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems
NAACL'24 conference
Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
PDFAsking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search
WWW'24 conference
Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
PDFCross-lingual Open-Retrieval Question Answering for African Languages
EMNLP'23 conference
Ogundepo Odunayo, ..., Clemencia Siro, et al.
PDFUnderstanding and Predicting User Satisfaction with Conversational Recommender Systems
ACM TOIS 2023 journal
Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
PDFAFRICANEWS: News Topic Classification for African News
AACL'23 conference (Area Chair Award)
Adelani, David Ifeoluwa, ..., Clemencia Siro, et al.
PDFEvaluating the Robustness of Machine Reading Comprehension Models to Low Resource Entity Renaming
AfricaNLP Workshop at ICLR'23 workshop
Clemencia Siro, Tunde Ajayi
PDFBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
TMLR 2023 journal
Srivastava, Aarohi,..., Clemencia Siro, et al.
PDFUnderstanding User Satisfaction with Task-oriented Dialogue Systems
SIGIR'22 conference
Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
PDFMasakhaNER: Named entity recognition for African languages
TACL 2021 journal
Adelani, David Ifeoluwa, ..., Clemencia Siro, et al.
PDFActivities
2024 - LLM4IREval workshop organizer at SIGIR'24
2023 - AfricaNLP workshop organizer at ICLR'23 and ICLR'24
2022: Guest speaker - Master AI University of Amsterdam: Evaluation of Dialogue System with Users
2022 - Present: Mentor for Master AI students at the University of Amsterdam
2019 - Present: Member of MasakhaneNLP, an NLP group for African languages