Tutorial: Deep Randomized Neural Networks

Deep Neural Networks (DNNs) are a fundamental tool in the modern
development of Machine Learning. Beyond the merits of properly designed
training strategies, a great part of DNNs success is undoubtedly due to
the inherent properties of their layered architectures, i.e., to the
introduced architectural biases. In this tutorial, we analyze how far we
can go by relying almost exclusively on these architectural biases. In
particular, we explore recent classes of DNN models wherein the majority
of connections are randomized or more generally fixed according to some
specific heuristic, leading to the development of Fast and Deep Neural
Network (FDNN) models. Examples of such systems consist of multi-layered
neural network architectures where the connections to the hidden
layer(s) are left untrained after initialization. Limiting the training
algorithms to operate on a reduced set of weights implies a number of
intriguing features. Among them, the extreme efficiency of the resulting
learning processes is undoubtedly a striking advantage with respect to
fully trained architectures. Besides, despite the involved
simplifications, randomized neural systems possess remarkable properties
both in practice, achieving state-of-the-art results in multiple
domains, and theoretically, allowing to analyze intrinsic properties of
neural architectures (e.g. before training of the hidden layers�
connections). In recent years, the study of randomized neural networks
has been extended towards deep architectures, opening new research
directions to the design of effective yet extremely efficient deep
learning models in vectorial as well as in more complex data domains.
This tutorial will cover all the major aspects regarding the design and
analysis of Fast and Deep Neural Networks, and some of the key results
with respect to their approximation capabilities. In particular, the
tutorial will first introduce the fundamentals of randomized neural
models in the context of feedforward networks (i.e., Random Vector
Functional Link and equivalent models), convolutional filters, and
recurrent systems (i.e., Reservoir Computing networks). Then, it will
focus specifically on recent results in the domain of deep randomized
systems, and their application to structured domains (trees, graphs).