Does Knowledge Distillation Really Work? NYU & Google Study Provides Insights on Student Model Fidelity | Synced
A research team from New York University and Google Research explores whether knowledge distillation really works, showing that a surprisingly large discrepancy often remains between the predictive...
Source: Synced | AI Technology & Industry Review
A research team from New York University and Google Research explores whether knowledge distillation really works, showing that a surprisingly large discrepancy often remains between the predictive distributions of the teacher and student models, even when the student has the capacity to perfectly match the teacher.