...
- Christopher Olah's explanations of concepts used in neural networks
- http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/
Theano
Books
- Neural networks and deep learning ← by Michael Nielsen
- Learning deep architectures for AI.pdf
- 2012 - Machine Learning for Hackers
- Rec'd by Pete Warden
- Deep Learning: A Practitioner's Approach
- Michael Nielsen - Neural Networks and Deep Learning
- This is so well written.
- Deep Learning: Methods and Applications
- Deep Learning - Ian Goodfellow, Aaron Courville, and Yoshua Bengio
...
- BURLAP
- TensorFlow
- Theano
- Torch
- https://code.google.com/p/word2vec/
- Mentioned by Pete Warden IIRC
Tensorflow
Major sites
Learning resources
- TensorFlow Tutorials
- 2016.02.28 - Pete Warden's Blog - TensorFlow for Poets
- Otoro tutorials
- WildML
- Indico?
- https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9
Tools
Forums
Papers
Articles
- 2015.11.09 - Slate - What Is “TensorFlow,” and Why Is Google So Excited About It?
- 2015.11.13 - Wired - Google's TensorFlow alone will not revolutionize AI
- by Erik Mueller (from IBM's Watson team)
- 2015.11.13 - Indico - The indico Machine Learning Team’s Take on TensorFlow
Before we call out some of the features from TensorFlow that are particularly relevant to deep learning, it is worth emphasizing that the most compelling thing about TensorFlow is the usability and architecture of the project. Even if no individual piece were revolutionary, the fact that all of the pieces work together to let us compose, compute, and visualize models is a real differentiating factor. Much the same way that Amazon’s EC2 itself isn’t revolutionary, the fact that it comes bundled with the suite of AWS niceties makes it an excellent product.
Here are some especially promising features:
Resource allocation.
Using the abstraction of computation graphs, TensorFlow maps the required computations onto a set of available devices. Graph and queue abstractions are powerful here, and there are many ways to solve the problem of allocating resources to the computation. TensorFlow implements what looks like a pretty sophisticated simulation and greedy allocation algorithm with methods to minimize communication overhead between resources. Other open source libraries, if they even allow you to use more than one compute device, tend to rely on the user to allocate resources statically.
Why is TensorFlow more promising? The obvious thing is scalability across distributed (possibly heterogeneous) resources, as in a datacenter or cloud deployment. The more subtle consequence is the allocation algorithm frees us from having to manage CPU vs. GPU devices, even on a single workstation…they’re all just resources to be used as greedily as possible.
Queues that allow portions of the graph to execute asynchronously.
This looks particularly useful for pre-fetching the next batch of data while the previous batch is computing. For example, using Titan X GPUs (indico’s weapon of choice) disk I/O is often the limiting factor for some of our models. Although we work around this limitation using threaded I/O, the TensorFlow approach looks even more robust. In addition to being conceptually simpler, putting data manipulation on the computation graph allows for better device utilization.
Visualization with TensorBoard.
As models get more complex, it is all too easy to skimp on model inspection and the practice of validating intuition. We believe visualization is really fundamental to the creative process and our ability to develop better models. So, visualization tools like TensorBoard are a great step in the right direction. We hope this will encourage the machine learning community in general to validate model internals, and drive towards new ways to train models and inspect performance.
Computations expressed as stateful dataflow graphs.
This abstraction allows models to be deployed across heterogeneous resources without rewriting models. Using a single workstation, we can exploit both CPUs and GPUs. This has the added benefit of making it easier to deploy to a heterogeneous compute environment (cloud, datacenter, etc).
Mobile Deployment.
TensorFlow is designed to work on a wide variety of hardware platforms ranging from high end multi-GPU rigs to smart phones. This enables developers to build and deploy machine learning applications on mobile devices. Advanced neural network applications such as language translation can be available without an internet connection.
- 2015.11.29 - LinkedIn Pulse - Google TensorFlow simple examples -- Think, Understand, IMPLEMENT :-)
- 2015.11.30 - FastML - What you wanted to know about TensorFlow
Papers
- http://deeplearning.net/reading-list/
- Google DeepMind
- Andrew Ng's papers
- Misc
Articles
- 2014.01.27 - Times.uk - The man with his fingers on the future (Demis Hassibis)
- 2014.01.29 - MIT Tech Review - Is Google Cornering the Market on Deep Learning?
- 2015.09.14 - MIT Tech Review - Deep Learning Machine Teaches Itself Chess in 72 Hours, Plays at International Master Level
- 2015.11.10 - Wired - TensorFlor, Google's Open Source AI, Signals Big Changes in Hardware Too
- 2015.12.08 - MIT Tech Review - Here’s What Developers Are Doing with Google’s AI Brain
- 2015.12.10 - MIT Tech Review - Facebook Joins Stampede of Tech Giants Giving Away Artificial Intelligence Technology
- 2015.12.16 - MIT Tech Review - Baidu’s Deep-Learning System Rivals People at Speech Recognition
- 2015.12.17 - MIT Tech Review - Can This Man Make AI More Human?
- 2016.03.29 - Michael Nielsen - Is AlphaGo Really Such a Big Deal?
Over the past few years, neural networks have been used to capture intuition and recognize patterns across many domains. Many of the projects employing these networks have been visual in nature, involving tasks such as recognizing artistic style or developing good video-game strategy. But there are also striking examples of networks simulating intuition in very different domains, including audio and natural language.
Because of this versatility, I see AlphaGo not as a revolutionary breakthrough in itself, but rather as the leading edge of an extremely important development: the ability to build systems that can capture intuition and learn to recognize patterns. Computer scientists have attempted to do this for decades, without making much progress. But now, the success of neural networks has the potential to greatly expand the range of problems we can use computers to attack.
Just because neural networks can do a good job of capturing some specific types of intuition, that doesn’t mean they can do as good a job with other types. Maybe neural networks will be no good at all at some tasks we currently think of as requiring intuition.
In actual fact, our existing understanding of neural networks is very poor in important ways. For example, a 2014 paper described certain “adversarial examples” which can be used to fool neural networks.
- Another limitation of existing systems is that they often require many human examples to learn from.
- Now we’ve got so many wonderful challenges ahead: to expand the range of intuition types we can represent, to make the systems stable, to understand why and how they work, and to learn better ways to combine them with the existing strengths of computer systems.
- AlphaZero
- World Models
...