Home > Seminars > Paul Resnick - An Information-theoretic Measure of Classifier Accuracy When the Ground Truth is Subjective

Paul Resnick - An Information-theoretic Measure of Classifier Accuracy When the Ground Truth is Subjective

Start:

10/4/2018 at 3:30PM

End:

10/4/2018 at 4:45PM

Location:

138 DeBartolo

Host:

College of Engineering close button
headerbottom

Tim Weninger

Tim Weninger

VIEW FULL PROFILE Email: tweninge@nd.edu
Phone: 574-631-6770
Website: http://www.nd.edu/~tweninge/
Office: 353 Fitzpatrick Hall

Affiliations

College of Engineering Assistant Professor
Network science, data science, machine learning, databases, and information retrieval.
Click for more information about Tim
574-631-6770
Add to calendar:
iCal vCal
Many classification tasks have no objective ground truth. Examples include: which content or explanation is "better" according to some community? is this comment toxic? what is the political leaning of this news article? The traditional modeling approach assumes each item has an objective true state that is perceived by humans with some random error. It fails to account for the fact that people have greater agreement on some items than others. I will describe an alternative model where the true state is a distribution over labels that raters from a specified population would assign to an item. This leads to information gain (mutual information) as a theoretically justified and computationally tractable measure of a classifier's quality, and an intuitive interpretation of information gain in terms of the sample size for a survey that would yield the same expected error rate.

Seminar Speaker:

Paul Resnick

University of Michigan

Dr. Paul Resnick is the Michael D. Cohen Collegiate Professor of Information and Associate Dean for Research at the University of Michigan School of Information. He was a pioneer in the fields of recommender systems and reputation systems. He recently started the Center for Social Media Responsibility, which encourages and helps social media platforms to meet their public responsibilities.