Tutorial: Deep Stochastic Learning and Understanding

This tutorial addresses the advances in deep Bayesian
learning for sequence data which are ubiquitous in speech, music, text,
image, video, web, communication and networking applications. Spatial
and temporal contents are analyzed and represented to fulfill a variety
of tasks ranging from classification, synthesis, generation,
segmentation, dialogue, search, recommendation, summarization,
answering, captioning, mining, translation, adaptation to name a few.
Traditionally, "deep learning" is taken to be a learning process where
the inference or optimization is based on the real-valued deterministic
model. The "latent semantic structure" in words, sentences, images,
actions, documents or videos learned from data may not be well expressed
or correctly optimized in mathematical logic or computer programs. The
"distribution function" in discrete or continuous latent variable model
for spatial and temporal sequences may not be properly decomposed or
estimated. This tutorial addresses the fundamentals of statistical
models and neural networks, and focus on a series of advanced Bayesian
models and deep models including recurrent neural network,
sequence-to-sequence model, variational auto-encoder (VAE), attention
mechanism, memory-augmented neural network, skip neural network,
temporal difference VAE, stochastic neural network, stochastic temporal
convolutional network, predictive state neural network, and policy
neural network. Enhancing the prior/posterior representation is
addressed. We present how these models are connected and why they work
for a variety of applications on symbolic and complex patterns in
sequence data. The variational inference and sampling method are
formulated to tackle the optimization for complicated models. The
embeddings, clustering or co-clustering of words, sentences or objects
are merged with linguistic and semantic constraints. A series of case
studies, tasks and applications are presented to tackle different issues
in deep Bayesian learning and understanding. At last, we will point out
a number of directions and outlooks for future studies. This tutorial
serves the objectives to introduce novices to major topics within deep
Bayesian learning, motivate and explain a topic of emerging importance
for natural language understanding, and present a novel synthesis
combining distinct lines of machine learning work.