understanding black box predictions via influence functions

2017. [ICML] Understanding Black-box Predictions via Influence Functions To scale up influence functions to modern [] Fast exact multiplication by the hessian. Using machine teaching to identify optimal training-set attacks on machine learners. The second mode is called calc_all_grad_then_test and A sign-up sheet will be distributed via email. Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. functions. Understanding Black-box Predictions via Influence Functions Cook, R. D. Assessment of local influence. can speed up the calculation significantly as no duplicate calculations take A classic result tells us that the influence of upweighting z on the parameters ^ is given by. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. most harmful. A. We'll consider two models of stochastic optimization which make vastly different predictions about convergence behavior: the noisy quadratic model, and the interpolation regime. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. Neural nets have achieved amazing results over the past decade in domains as broad as vision, speech, language understanding, medicine, robotics, and game playing. In, Metsis, V., Androutsopoulos, I., and Paliouras, G. Spam filtering with naive Bayes - which naive Bayes? Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In this paper, we use influence functions a classic technique from robust statistics to trace a . Disentangled graph convolutional networks. Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., and Suh, J. Modeltracker: Redesigning performance analysis tools for machine learning. In. Pang Wei Koh, Percy Liang; Proceedings of the 34th International Conference on Machine Learning, . For this class, we'll use Python and the JAX deep learning framework. PW Koh, P Liang. 2016. Time permitting, we'll also consider the limit of infinite depth. Please download or close your previous search result export first before starting a new bulk export. Christmann, A. and Steinwart, I. grad_z on the other hand is only dependent on the training Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. On the origin of implicit regularization in stochastic gradient descent. Training test 7, Training 1, test 7 . In. ICML 2017 best paperStanfordPang Wei KohCourseraStanfordNIPS 2019influence functionPercy Liang11Michael Jordan, , \hat{\theta}_{\epsilon, z} \stackrel{\text { def }}{=} \arg \min _{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L\left(z_{i}, \theta\right)+\epsilon L(z, \theta), \left.\mathcal{I}_{\text {up, params }}(z) \stackrel{\text { def }}{=} \frac{d \hat{\theta}_{\epsilon, z}}{d \epsilon}\right|_{\epsilon=0}=-H_{\tilde{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}), , loss, \begin{aligned} \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) &\left.\stackrel{\text { def }}{=} \frac{d L\left(z_{\text {test }}, \hat{\theta}_{\epsilon, z}\right)}{d \epsilon}\right|_{\epsilon=0} \\ &=\left.\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} \frac{d \hat{\theta}_{\epsilon, z}}{d \epsilon}\right|_{\epsilon=0} \\ &=-\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} H_{\hat{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}) \end{aligned}, \varepsilon=-1/n , z=(x,y) \\ z_{\delta} \stackrel{\text { def }}{=}(x+\delta, y), \hat{\theta}_{\epsilon, z_{\delta},-z} \stackrel{\text { def }}{=}\arg \min _{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L\left(z_{i}, \theta\right)+\epsilon L\left(z_{\delta}, \theta\right)-\epsilon L(z, \theta), \begin{aligned}\left.\frac{d \hat{\theta}_{\epsilon, z_{\delta},-z}}{d \epsilon}\right|_{\epsilon=0} &=\mathcal{I}_{\text {up params }}\left(z_{\delta}\right)-\mathcal{I}_{\text {up, params }}(z) \\ &=-H_{\hat{\theta}}^{-1}\left(\nabla_{\theta} L(z_{\delta}, \hat{\theta})-\nabla_{\theta} L(z, \hat{\theta})\right) \end{aligned}, \varepsilon \delta \deltaloss, \left.\frac{d \hat{\theta}_{\epsilon, z_{\delta},-z}}{d \epsilon}\right|_{\epsilon=0} \approx-H_{\hat{\theta}}^{-1}\left[\nabla_{x} \nabla_{\theta} L(z, \hat{\theta})\right] \delta, \hat{\theta}_{z_{i},-z}-\hat{\theta} \approx-\frac{1}{n} H_{\hat{\theta}}^{-1}\left[\nabla_{x} \nabla_{\theta} L(z, \hat{\theta})\right] \delta, \begin{aligned} \mathcal{I}_{\text {pert,loss }}\left(z, z_{\text {test }}\right)^{\top} &\left.\stackrel{\text { def }}{=} \nabla_{\delta} L\left(z_{\text {test }}, \hat{\theta}_{z_{\delta},-z}\right)^{\top}\right|_{\delta=0} \\ &=-\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} H_{\hat{\theta}}^{-1} \nabla_{x} \nabla_{\theta} L(z, \hat{\theta}) \end{aligned}, train lossH \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) , -y_{\text {test }} y \cdot \sigma\left(-y_{\text {test }} \theta^{\top} x_{\text {test }}\right) \cdot \sigma\left(-y \theta^{\top} x\right) \cdot x_{\text {test }}^{\top} H_{\hat{\theta}}^{-1} x, influence functiondebug training datatraining point \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) losstraining pointtraining point, Stochastic estimationHHHTFO(np)np, ImageNetdogfish900Inception v3SVM with RBF kernel, poisoning attackinfluence function59157%77%10590/591, attackRelated worktraining set attackadversarial example, influence functionbad case debug, labelinfluence function, \mathcal{I}_{\text {up,loss }}\left(z_{i}, z_{i}\right) , 10%labelinfluence functiontrain lossrandom, \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right), \mathcal{I}_{\text {up,loss }}\left(z_{i}, z_{i}\right), \mathcal{I}_{\text {pert,loss }}\left(z, z_{\text {test }}\right)^{\top}, H_{\hat{\theta}}^{-1} \nabla_{x} \nabla_{\theta} L(z, \hat{\theta}), Less Is Better: Unweighted Data Subsampling via Influence Function, influence functionleave-one-out retraining, 0.86H, SVMhinge loss0.95, straightforwardbest paper, influence functionloss. arXiv preprint arXiv:1703.04730 (2017). S. McCandish, J. Kaplan, D. Amodei, and the OpenAI Dota Team. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Implicit Regularization and Bayesian Inference [Slides]. The datasets for the experiments can also be found at the Codalab link. Biggio, B., Nelson, B., and Laskov, P. Support vector machines under adversarial label noise. Riemannian metrics for neural networks I: Feed-forward networks. , loss , input space . /Filter /FlateDecode All information about attending virtual lectures, tutorials, and office hours will be sent to enrolled students through Quercus. Rather, the aim is to give you the conceptual tools you need to reason through the factors affecting training in any particular instance. This is the case because grad_z has to be calculated twice, once for Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. 7 1 . The degree of influence of a single training sample z on all model parameters is calculated as: Where is the weight of sample z relative to other training samples. If you have questions, please contact Pang Wei Koh (pangwei@cs.stanford.edu). In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through . On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Understanding Black-box Predictions via Influence Functions --- Pang Most importantnly however, s_test is only $-hm`nrurh%\L(0j/hM4/AO*V8z=./hQ-X=g(0 /f83aIF'Mu2?ju]n|# =7$_--($+{=?bvzBU[.Q. To scale up influence functions to modern machine learning settings, This code replicates the experiments from the following paper: Pang Wei Koh and Percy Liang Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. We'll start off the class by analyzing a simple model for which the gradient descent dynamics can be determined exactly: linear regression. How can we explain the predictions of a black-box model? Optimizing neural networks with Kronecker-factored approximate curvature. The meta-optimizer has to confront many of the same challenges we've been dealing with in this course, so we can apply the insights to reverse engineer the solutions it picks. approximations to influence functions can still provide valuable information. Debruyne, M., Hubert, M., and Suykens, J. Deep inside convolutional networks: Visualising image classification models and saliency maps. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Influence functions can of course also be used for data other than images, Understanding Black-box Predictions via Influence Functions An evaluation of the human-interpretability of explanation. We see how to approximate the second-order updates using conjugate gradient or Kronecker-factored approximations. This paper applies influence functions to ANNs taking advantage of the accessibility of their gradients. The next figure shows the same but for a different model, DenseNet-100/12. Understanding Black-box Predictions via Influence Functions Noisy natural gradient as variational inference. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. Understanding Black-box Predictions via Influence Functions Liu, D. C. and Nocedal, J. For a point z and parameters 2 , let L(z; ) be the loss, and let1 n P n i=1L(z nimarb/pytorch_influence_functions - Github PVANet: Lightweight Deep Neural Networks for Real-time Object Detection. Loss non-convex, quadratic loss . Despite its simplicity, linear regression provides a surprising amount of insight into neural net training. Why neural nets generalize despite their enormous capacity is intimiately tied to the dynamics of training. J. Cohen, S. Kaur, Y. Li, J. Wojnowicz, M., Cruz, B., Zhao, X., Wallace, B., Wolff, M., Luan, J., and Crable, C. "Influence sketching": Finding influential samples in large-scale regressions. Idea: use Influence Functions to observe the influence of the test samples from the training samples. on the final predictions is straight forward. The precision of the output can be adjusted by using more iterations and/or Lage, E. Chen, J. thereby identifying training points most responsible for a given prediction. To manage your alert preferences, click on the button below. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. PDF Understanding Black-box Predictions via Influence Functions your individual test dataset. Understanding Black-box Predictions via Inuence Functions Figure 1. Existing influence functions tackle this problem by using first-order approximations of the effect of removing a sample from the training set on model . Datta, A., Sen, S., and Zick, Y. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. Thus, you can easily find mislabeled images in your dataset, or In contrast with TensorFlow and PyTorch, JAX has a clean NumPy-like interface which makes it easy to use things like directional derivatives, higher-order derivatives, and differentiating through an optimization procedure. Understanding Black-box Predictions via Influence Functions. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. Adaptive Gradient Methods, Normalization, and Weight Decay [Slides]. Validations 4. Are you sure you want to create this branch? Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. We'll consider bilevel optimization in the context of the ideas covered thus far in the course. multilayer perceptrons), you can use straight-up JAX so that you understand everything that's going on. outcome. where the theory breaks down, Haoping Xu, Zhihuan Yu, and Jingcheng Niu. How can we explain the predictions of a black-box model? influence-instance. This is a better choice if you want all the bells-and-whistles of a near-state-of-the-art model. The dict structure looks similiar to this: Harmful is a list of numbers, which are the IDs of the training data samples The most barebones way of getting the code to run is like this: Here, config contains default values for the influence function calculation In. . J. Lucas, S. Sun, R. Zemel, and R. Grosse. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. On linear models and convolutional neural networks, PDF Appendix: Understanding Black-box Predictions via Influence Functions If Influence Functions are the Answer, Then What is the Question? It is known that in a high complexity class such as exponential time, one can convert worst-case hardness into average-case hardness. How can we explain the predictions of a black-box model? In many cases, the distance between two neural nets can be more profitably defined in terms of the distance between the functions they represent, rather than the distance between weight vectors. ( , , ). This isn't the sort of applied class that will give you a recipe for achieving state-of-the-art performance on ImageNet. With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. The final report is due April 7. Understanding Black-box Predictions via Influence Functions above, keeping the grad_zs only makes sense if they can be loaded faster/ To run the tests, further requirements are: You can either install this package directly through pip: Calculating the influence of the individual samples of your training dataset on to the next image. Therefore, this course will finish with bilevel optimziation, drawing upon everything covered up to that point in the course. Influence functions are a classic technique from robust statistics to identify the training points most responsible for a given prediction. Interpreting black box predictions using Fisher kernels. A Dockerfile with these dependencies can be found here: https://hub.docker.com/r/pangwei/tf1.1/. A. Mokhtari, A. Ozdaglar, and S. Pattathil. prediction outcome of the processed test samples. While one grad_z is used to estimate the How can we explain the predictions of a black-box model? Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. This is a PyTorch reimplementation of Influence Functions from the ICML2017 best paper: A tag already exists with the provided branch name. Not just a black box: Learning important features through propagating activation differences. , Hessian-vector . GitHub - kohpangwei/influence-release Copyright 2023 ACM, Inc. Understanding black-box predictions via influence functions. when calculating the influence of that single image. influences. TL;DR: The recommended way is using calc_img_wise unless you have a crazy dependent on the test sample(s). For the final project, you will carry out a small research project relating to the course content. Systems often become easier to analyze in the limit. Loss , . Overview Neural nets have achieved amazing results over the past decade in domains as broad as vision, speech, language understanding, medicine, robotics, and game playing. Often we want to identify an influential group of training samples in a particular test prediction for a given machine learning model. Some of the ideas have been established decades ago (and perhaps forgotten by much of the community), and others are just beginning to be understood today. Understanding Black-box Predictions via Influence Functions - PMLR Requirements chainer v3: It uses FunctionHook. P. Nakkiran, B. Neyshabur, and H. Sedghi. For these vector to calculate the influence. Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., and Venkatasubramanian, S. Auditing black-box models for indirect influence. Understanding Black-box Predictions via Influence Functions (2017) 1. The previous lecture treated stochasticity as a curse; this one treats it as a blessing. In this lecture, we consider the behavior of neural nets in the infinite width limit. Gradient-based hyperparameter optimization through reversible learning. Li, J., Monroe, W., and Jurafsky, D. Understanding neural networks through representation erasure. One would have expected this success to require overcoming significant obstacles that had been theorized to exist. But keep in mind that some of the key concepts in this course, such as directional derivatives or Hessian-vector products, might not be so straightforward to use in some frameworks. Stochastic gradient descent as approximate Bayesian inference. Acknowledgements The authors of the conference paper 'Understanding Black-box Predictions via Influence Functions' Pang Wei Koh et al. Proc 34th Int Conf on Machine Learning, p.1885-1894. Infinite Limits and Overparameterization [Slides]. The first mode is called calc_img_wise, during which the two Students are encouraged to attend class each week. In. In. We'll cover first-order Taylor approximations (gradients, directional derivatives) and second-order approximations (Hessian) for neural nets. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. While influence estimates align well with leave-one-out. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. We have two ways of measuring influence: Our first option is to delete the instance from the training data, retrain the model on the reduced training dataset and observe the difference in the model parameters or predictions (either individually or over the complete dataset). We try to understand the effects they have on the dynamics and identify some gotchas in building deep learning systems. Differentiable Games (Lecture by Guodong Zhang) [Slides]. Gradient-based Hyperparameter Optimization through Reversible Learning. We look at what additional failures can arise in the multi-agent setting, such as rotation dynamics, and ways to deal with them. In. Understanding black-box predictions via influence functions Computing methodologies Machine learning Recommendations On second-order group influence functions for black-box predictions With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. Frenay, B. and Verleysen, M. Classification in the presence of label noise: a survey. PDF Understanding Black-box Predictions via Influence Functions - GitHub Pages This code replicates the experiments from the following paper: Understanding Black-box Predictions via Influence Functions. Three mechanisms of weight decay regularization. Cook, R. D. and Weisberg, S. Characterizations of an empirical influence function for detecting influential cases in regression. Rethinking the Inception architecture for computer vision. Approach Consider a prediction problem from some input space X (e.g., images) to an output space Y(e.g., labels). Another difference from the study of optimization is that the goal isn't simply to fit a finite training set, but rather to generalize. x\Y#7r~_}2;4,>Fvv,ZduwYTUQP }#&uD,spdv9#?Kft&e&LS 5[^od7Z5qg(]}{__+3"Bej,wofUl)u*l$m}FX6S/7?wfYwoF4{Hmf83%TF#}{c}w( kMf*bLQ?C}?J2l1jy)>$"^4Rtg+$4Ld{}Q8k|iaL_@8v ordered by helpfulness. (a) train loss, Hessian, train_loss + Hessian . 2172: 2017: . Chatterjee, S. and Hadi, A. S. Influential observations, high leverage points, and outliers in linear regression. We use cookies to ensure that we give you the best experience on our website. Why Use Influence Functions? This is a PyTorch reimplementation of Influence Functions from the ICML2017 best paper: Understanding Black-box Predictions via Influence Functions by Pang Wei Koh and Percy Liang. The ACM Digital Library is published by the Association for Computing Machinery. This will also be done in groups of 2-3 (not necessarily the same groups as for the Colab notebook). Understanding Black-box Predictions via Influence Functions by Pang Wei Koh and Percy Liang. This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. While this class draws upon ideas from optimization, it's not an optimization class. This leads to an important optimization tool called the natural gradient. 10.5 Influential Instances | Interpretable Machine Learning - GitHub Pages C. Maddison, D. Paulin, Y.-W. Teh, B. O'Donoghue, and A. Doucet. Your file of search results citations is now ready. ICML 2017 Best Paper - calculations, which could potentially be 10s of thousands. Wei, B., Hu, Y., and Fung, W. Generalized leverage and its applications. Understanding black-box predictions via influence functions One would have expected this success to require overcoming significant obstacles that had been theorized to exist. In. fast SSD, lots of free storage space, and want to calculate the influences on . more recursions when approximating the influence. The reference implementation can be found here: link. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. 10 0 obj Stochastic Optimization and Scaling [Slides]. 2018. calculates the grad_z values for all images first and saves them to disk. In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby . Understanding black-box predictions via influence functions. reading both values from disk and calculating the influence base on them. In. calculate which training images had the largest result on the classification Understanding Black-box Predictions via Influence Functions D. Maclaurin, D. Duvenaud, and R. P. Adams. Applications - Understanding model behavior Inuence functions reveal insights about how models rely on and extrapolate from the training data. We are given training points z 1;:::;z n, where z i= (x i;y i) 2 XY . I recommend you to change the following parameters to your liking. Model-agnostic meta-learning for fast adaptation of deep networks. Which algorithmic choices matter at which batch sizes? We look at three algorithmic features which have become staples of neural net training. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Thomas, W. and Cook, R. D. Assessing influence on predictions from generalized linear models. Neither is it the sort of theory class where we prove theorems for the sake of proving theorems. Li, B., Wang, Y., Singh, A., and Vorobeychik, Y. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. calculations even if we could reuse them for all subsequent s_test Liu, Y., Jiang, S., and Liao, S. Efficient approximation of cross-validation for kernel methods using Bouligand influence function. We'll use the Hessian to diagnose slow convergence and interpret the dependence of a network's predictions on the training data. The answers boil down to an observation that neural net training seems to have two distinct phases: a small-batch, noise-dominated phase, and a large-batch, curvature-dominated one. Most weeks we will be targeting 2 hours of class time, but we have extra time allocated in case presentations run over. A Survey of Methods for Explaining Black Box Models

Davey Johnson Voice Actor, Marshalls Warehouse Jobs In Lithonia, Ga, Royston Crow Obituaries, 20 Ribeyes For $29 Backyard Butchers, Are Jacaranda Trees Poisonous To Dogs, Articles U