Project final report guidelines

• Project presentation: 10-12 min/group

Assignment 3 in

Invited talk on Unsupervised MT by Xuan Phi

Lecture Slide

• Multitask-learning
• Fine-Tuning for Transfer Learning
• Meta-learning problem
• Two views of Meta-learning Problem
• Black-box meta learning (GPT3)
• Optimization-based meta learning (MAML)
• Non-parametric meta learning (ptotyical nets)
  

Slides on adversarial nets

Slides on adverarial attacks (prepared by Samson@Amazon)

Slides on deep generative models

Invited talk on Text Generation by Lin Xiang

Lecture Content

• Generative adversarial nets (GANs)
• Domain adversarial nets (DANs)

• Adversarial attacks in NLP

• Defense:

  • Training with adversarial examples
  • Consistency regularization
  • Cross-view consistency

• Variational inference

• Auto encoders

• Variational auto encoders

• Conditional VAEs

• Vector Quantized VAEs

• Variational Generative adversarial nets

Suggested Readings

• GAN Paper

• MUSE paper

• Adversarial Learning for Neural Dialogue Generation

• DANN paper

• ADDA paper

• Ian Goodfellow’s talk on adversarial examples

• NAACL tutorial on adversarial examples

• VAE paper

• VQVAE paper

Lecture Slide

Assignment 2 in

Assignment 3 out

Project final report guidelines

Lecture Content

• Pre-training and fine-tuning paradigm

  • CoVe
  • TagLM
  • ELMo
  • GPT
  • ULMfit
  • BERT (+ mBERT)
  • XLM
  • XL-Net
  • BART (+ mBART)
  • T5 (+ mT5)

• Evaluation benchmarks

  • GLUE
  • SQuAD
  • NER
  • SuperGLUE
  • XNLI

TA: Mathieu

Pre-train Fine-tune with HF

Suggested Readings

• Cove Paper

• ULMFit paper

• BERT Paper

• XLM paper

• ELMo paper

• Transformer XL paper

• XLNet paper

• BART paper

• mBART paper

Lecture Slide

Project Proposal due

Lecture Content

• Seq2Seq Variants (Pointer nets, Pointer Generator Nets)

  • Machine Translation
  • Summarization
  • Parsing
  • image/video captioning

• Transformer architecture

  • Self-attention
  • Positional encoding
  • Multi-head attention

Practical exercise with Pytorch

TA: Bosheng Ding

The Annotated Transformer

Suggested Readings

• Get To The Point: Summarization with Pointer-Generator Networks

• Pointer Networks

• Stack-Pointer Networks for Dependency Parsing

• A Unified Linear-Time Framework for Sentence-Level Discourse Parsing

• Attention Is All You Need

• The Illustrated Transformer

• Resurrecting Submodularity in Neural Abstractive Summarization

Lecture Slide

Project Proposal due

Lecture Content

• Information bottleneck issue with vanilla Seq2Seq
• Attention to the rescue
• Details of attention mechanism
• Attention variants
• Morphology in MT
• Subword level models

Practical exercise with Pytorch

TA: Xuan-Phi (NTU-NLP)

Practical exercise with Pytorch

• Neural machine translation tutorial in pytorch

Suggested Readings

• Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)

• Effective Approaches to Attention-based Neural Machine Translation

• Neural Machine Translation of Rare Words with Subword Units

• Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

• Unsupervised Machine Translation Using Monolingual Corpora Only

• Phrase-Based & Neural Unsupervised Machine Translation

Lecture Slide

Tutorial 4

Lecture Content

• Machine translation

  • Early days (1950s)
  • Statistical machine translation or SMT (1990-2010)
  • Alignment in SMT
  • Decoding in SMT

• Neural machine translation or NMT (2014 - )

• Encoder-decoder model for NMT

• Advantages and disadvantages of NMT

• Greedy vs. beam-search decoding

• MT evaluation

• Other applications of Seq2Seq

Suggested Readings

• Statistical Machine Translation slides, CS224n 2015 (lectures 2/3/4)

• Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)

• Statistical Machine Translation (book by Philipp Koehn)

• A Neural Conversational Model

• BLEU (original paper)

Lecture Slide

Tutorial 3

Assignment 1 in

Assignment 2 out (in NTU-Learn)

Lecture Content

• Basic RNN structures
• Language modeling with RNNs
• Backpropagation through time
• Text generation with RNN LM
• Issues with Vanilla RNNs
• Exploding gradient
• Gated Recurrent Units (GRUs) and LSTMs
• Bidirectional RNNs
• Multi-layer RNNs
• Sequence labeling with RNNs
• Sequence classification with RNNs

TA: Saiful Bari

Practical exercise with Pytorch

• Named Entity Recognition

Suggested Readings

• N-gram Language Models

• Karpathy’s nice blog on Recurrent Neural Networks

• Building an Efficient Neural Language Model

• On the difficulty of training recurrent neural networks

• Colah’s blog on LSTMs/GRUs

• Neural Architectures for Named Entity Recognition

• Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings

• Zero-Resource Cross-Lingual NER

Lecture Slide

Tutorial 3

Lecture Content

• Classification tasks in NLP
• Window-based Approach for language modeling
• Window-based Approach for NER, POS tagging, and Chunking
• Convolutional Neural Net for NLP
• Max-margin Training
• Scaling Softmax (Adaptive input & output)

Practical exercise with Pytorch

• CNN for word encoding

Suggested Readings

• Natural Language Processing (Almost) from Scratch

• Convolutional Neural Networks for Sentence Classification

• Character-Aware Neural Language Models

• Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

• Adaptive Softmax Paper

• Adaptive Input representation paper

• KNN-LM paper

Lecture Slide

Project Proposal Instructions

Tutorial 2

**Assignment 1 out (in NTU-Learn) **

Lecture Content

• Word meaning
• Denotational semantics
• Distributed representation of words
• Word2Vec models (Skip-gram, CBOW)
• Negative sampling
• Glove
• FastText

• Evaluating word vectors

  • Intrinsic evaluation
  • Extrinsic evaluation

• Cross-lingual word vectors

TA: Mathieu

Practical exercise with Pytorch

• Skip-gram training

• Visualization

Suggested Readings

• Word2Vec Tutorial - The Skip-Gram Model blog
• Efficient Estimation of Word Representations in Vector Space - Original word2vec paper
• Distributed Representations of Words and Phrases and their Compositionality - negative sampling paper
• GloVe: Global Vectors for Word Representation
• FastText: Enriching Word Vectors with Subword Information
• Linguistic Regularities in Sparse and Explicit Word Representations.
• Neural Word Embeddings as Implicit Matrix Factorization.

• Improving Distributional Similarity with Lessons Learned from Word Embeddings

• Survey on Cross-lingual embedding methods

• Slides on Cross-lingual embedding

• Adversarial autoencoder for unsupervised word translation

• LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space

• Evaluating Cross-Lingual Word Embeddings

• Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Lecture Slide

Tutorial 2

Lecture Content

• Why Deep Learning for NLP?

• From Logistic Regression to Feed-forward NN

  • Activation functions

• SGD with Backpropagation

• Adaptive SGD (Adagrad, adam, RMSProp)

• Regularization (Weight Decay, Dropout, Batch normalization, Gradient clipping)

• Introduction to Word Vectors

TA: Mathieu

Practical exercise with Pytorch

Pytorch notebook

• Backpropagation
• Dropout
• Batch normalization
• Initialization
• Gradient clipping

Suggested Readings

• SGD optimization blog

• An overview of gradient descent optimization algorithms

• Adam: a Method for Stochastic Optimization

• Neural Optimizer Search with Reinforcement Learning

Lecture Slide

Tutorial 1

Lecture Content

• What is Machine Learning?
• Supervised vs. unsupervised learning
• Linear Regression
• Logistic Regression
• Multi-class classification
• Parameter estimation (MLE & MAP)
• Gradient-based optimization & SGD

TA: Mathieu

Practical exercise with Pytorch

• Deep learning with PyTorch
• Linear Regression
• Logistic Regression

• Numpy notebook

• [Supplementary]

  • Numerical programming with Pytorch - Pytorch intro
    

Lecture Slide

Lecture Content

• What is Natural Language Processing?
• Why is language understanding difficult?
• What is Deep Learning?
• Deep learning vs. other machine learning methods?
• Why deep learning for NLP?
• Applications of deep learning to NLP
• Knowing the target group (background, field of study, programming experience)
• Expectation from the course
  

Python & PyTorch Basics

• Programming in Python

  • Jupiter Notebook and google colab
  • Introduction to python
  • Deep Learning Frameworks
  • Why Pytorch?
  • Deep learning with PyTorch

• [Supplementary]

  • Numerical programming with numpy/scipy - Numpy intro
  • Numerical programming with Pytorch - Pytorch intro

• What is RL?
• Key concepts: Rewards, Policy, Value Function
• What is Deep RL?

• Policy-based Deep RL

  • Deep Policy Network
  • Policy Gradient

• Deep Q-Learning

• Applications of Deep RL in NLP

  • Abstractive summarization
  • Dialogue generation
  • Question answering
  • Multimodal (image and video captioning)
  • Machine translation