Machine Learning
Thoughts on Models with Regularization
Lately I’ve been reflecting on regularization. Early in my data science career I spent some time working with generalized additive models, but I started focusing more and more on traditional statistical methods. I am re-finding the value in regularization and expect to use more of it going forward.
Supervised Learning as Function Approximation
Supervised learning is perhaps the most central idea in Machine Learning. It is equally central to statistics where it is known as regression. Statistics formulates the problem in terms of identifying the distribution from which observations are drawn; Machine Learning in terms of finding a model that fits the data well.
Statistics and Machine Learning: Better Together!
My masters degree focused on Machine Learning, but when I got my first job as a data scientist, I quickly realized there was a lot I still needed to learn about Statistics. Since then I have come to appreciate the nuanced differences between Statistics and Machine Learning and I’m convinced they have a lot to offer one another!
Thoughts on Principal Components Analysis
This is a post with more questions than answers. I’ve been thinking about Principal Components Analysis (PCA) lately.
Object Detection with Deep Learning
One of the most interesting topics in the Coursera Deep Learning specialization is the “YOLO” algorithm for object detection. I often find it helpful to describe algorithms in my own words to solidify my understanding, and that is precisely what I will do here. Readers likely will prefer the original paper and its sequel.
Thoughts on the Coursera Deep Learning Specialization
I recently completed the Deep Learning specialization on Coursera from deeplearning.ai. Over five courses, they go over generic neural networks, regularization, convolutional neural nets, and recurrent neural nets. Having completed it, I would say the specialization is a great overview, and a jumping off point for learning more about particular techniques. I wouldn’t say I have an in-depth understanding of all the material, but I do feel like I could go off and read papers and understand them, which is maybe all I could expect.
Distribution of Local Minima in Deep Neural Networks
The “unreasonable effectiveness of deep learning” has been much discussed. Namely, as the cost function is non-convex, any optimization procedure will in general find a local, non-global, minimum. Actually, algorithms like gradient descent will terminate (perhaps because of early stopping) before even reaching a local minimum. For many experts in optimization, this seems like a bad thing. Concretely, it seems like the performance of networks trained in this way would be much worse than other optimization-based systems where we are in fact able to find the global minimum, such as logistic regression.
Computer Vision Cheat Sheet
I am currently working through Convolutional Neural Networks, the fourth course in the Coursera specialization on Deep Learning. The first week of that course contains some hard-to-remember equations about filter sizes and padding and striding and I thought it would be helpful for me to write it out for future reference.
Deep Learning Checklist
Recently I started the Deep Learning Specialization on Coursera. While I studied neural networks in my masters program (from Andrew Ng himself!), that was a long time ago and the field has changed considerably since then. I am supplementing the course by reading Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville, which I will refer to as GBC16.
Three Goals of Statistics: Description, Prediction, and Prescription
The great successes of Machine Learning in recent years are based on our ability to extrapolate and predict based on data. The next big step is learning and leveraging the relationship between cause and effect to prescribe what action to take.
gamdist
Generalized Additive Models in Python
