Home > John P. Lalor - Learning Latent Parameters Without Human Response Patterns: Item Response Theory with Artificial Crowds

John P. Lalor - Learning Latent Parameters Without Human Response Patterns: Item Response Theory with Artificial Crowds


9/26/2019 at 3:30PM


9/26/2019 at 4:45PM


131 DeBartolo


College of Engineering close button

David Chiang

David Chiang

VIEW FULL PROFILE Email: dchiang@nd.edu
Phone: 574-631-9441
Website: http://www.nd.edu/~dchiang/
Office: 179 Fitzpatrick Hall of Engineering


College of Engineering Associate Professor
Natural language processing, machine learning, and digital humanities.
Click for more information about David
Add to calendar:
iCal vCal

Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern data, presenting a significant bottleneck for large data sets like those required for training deep neural networks. In this I will introduce an approach to learning IRT models using response patterns generated from artificial crowds of neural network models. I will demonstrate the effectiveness of learning IRT models using machine-generated data through quantitative and qualitative analyses for tasks in NLP and vision. I will demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. 

I'll conclude by introducing Dynamic Data selection for Curriculum Learning via Ability Estimation (DDaCLAE), a curriculum learning strategy that leverages these learned IRT models to probe model ability at each training epoch to select the best training examples at that point in time. DDaCLAE adds data at a rate commensurate with the model's capability, in contrast to scheduled curricula that add data at a predetermined rate. Experimental results demonstrate that DDaCLAE is more efficient and effective than existing curriculum learning methods, improving test set accuracy while reducing training set size by up to 88%.

Seminar Speaker:

John P. Lalor

University of Notre Dame

John is an Instructor in the IT, Analytics, and Operations Department in the Mendoza College of Business at the University of Notre Dame and an ABD PhD candidate at the University of Massachusetts Amherst in the College of Information and Computer Science. At UMass John worked with Dr. Hong Yu in the Bio-NLP lab. Prior to UMass, John worked as a software developer at Eze Software in Chicago and as an IT Audit Associate for KPMG. John received his MS in Computer Science at DePaul University, and his bachelor's degree in IT Management with a minor in Irish Language & Literature from the University of Notre Dame.