Course Objectives

Natural language processing (NLP) is one of the most important fields in artificial intelligence (AI). It has become very crucial in the information age because most of the information is in the form of unstructured text. NLP technologies are applied everywhere as people communicate mostly in language: language translation, web search, customer support, emails, forums, advertisement, radiology reports, to name a few.

There are a number of core NLP tasks and machine learning models behind NLP applications. Deep learning, a sub-field of machine learning, has recently brought a paradigm shift from traditional task-specific feature engineering to end-to-end systems and has obtained high performance across many different NLP tasks and downstream applications. Tech companies like Google, Baidu, Alibaba, Apple, Amazon, Facebook, Tencent, and Microsoft are now actively working on deep learning methods to improve their products. For example, Google recently replaced its traditional statistical machine translation and speech-recognition systems with systems based on deep learning methods.

Optional Textbooks

  • Deep Learning by Goodfellow, Bengio, and Courville free online
  • Machine Learning — A Probabilistic Perspective by Kevin Murphy online
  • Natural Language Processing by Jacob Eisenstein free online
  • Speech and Language Processing by Dan Jurafsky and James H. Martin (3rd ed. draft)

Intended Learning Outcomes

In this course, students will learn state-of-the-art deep learning methods for NLP. Through lectures and practical assignments, students will learn the necessary tricks for making their models work on practical problems. They will learn to implement, and possibly to invent their own deep learning models using available deep learning libraries like Pytorch.

Our Approach

  • Thorough and Detailed: How to write from scratch, debug and train deep neural models

  • State of the art: Most lecture materials are new from research world in the past 1-5 years.

  • Practical: Focus on practical techniques for training the models, and on GPUs.

  • Fun: Cover exciting new advancements in NLP (e.g., Transformer, BERT).

Assessment Approach

Weekly Workload

  • Every two-hour lecture will be accompanied by practical problems implemented in PyTorch.
  • There will be NO office hour.
  • There will be 5% marks for class participation.

Assignments (individually graded)

  • There will be three (3) assignments contributing to 3 * 15% = 45% of the total assessment.
  • Late day policy
    • 2 free late days; afterwards,10% off per day late
    • Not accepted after 3 late days
  • Students will be graded individually on the assignments. They will be allowed to discuss with each other on the homework assignments, but they are required to submit individual write-ups and coding exercises.

Final Project (Group work but individually graded)

  • There will be a final project contributing to the remaining 50% of the total course-work assessment.
    • 1–3 people per group
    • Project proposal: 5%, update: 5%, presentation: 10%, report: 30%
  • The project will be a group or individual work depending on the student’s preference. Students will be graded individually. The final project presentation will ensure the student’s understanding of the project

Course Prerequisites

  • Proficiency in Python (using numpy and PyTorch). There is a lecture for those who are not familiar with Python.
  • College Calculus, Linear Algebra
  • Basic Probability and Statistics
  • Machine Learning basics

Teaching

Instructor

Teaching Assistants

Schedule & Course Content

 
 
 
 
 

Week 14: In-class Project presentation

02:00 PM - 5:30 PM 19 February 2022 SMPS LT5, NTU, Singapore
  • Project presentation: 15 min/group
 
 
 
 
 

Week 13: Adversarial NLP

03:45 PM - 5:30 PM 12 February 2022 SMPS LT5, NTU, Singapore

Assignment 3 in

Slides on adversarial nets

Slides on adverarial attacks (prepared by Samson@NUS-Salesforce)

Lecture Content

  • Generative adversarial nets (GANs)
  • Domain adversarial nets (DANs)
  • Transfer learning with DANs
  • Adversarial attacks in NLP

  • Defense:

    • Training with adversarial examples
    • Consistency regularization
    • Cross-view consistency
  • Limits & Future of Deep NLP

    • Multi-sentence processing
    • Multi-task learning
    • Multimodal learning
    • Model interpretability

Suggested Readings

 
 
 
 
 

Week 13: Multilingual NLP

02:00 PM - 3:30 PM 12 February 2022 SMPS LT5, NTU, Singapore

Lecture Slide

Lecture Content

  • Multilingual NLP
    • Why we need Multilingual NLP?
    • Low-resource NLP
    • Cross-lingual models
    • Multilingual models
 
 
 
 
 

Week 12: Contextual embeddings (ii)

02:00 PM - 5:30 PM 5 February 2022 SMPS LT5, NTU, Singapore

Lecture Slide

Project final report guidelines

Lecture Content

  • Why semi-supervsied?
  • Semisupervised learning dimensions
  • Pre-training and fine-tuning methods

    • XLNet
    • mBERT
    • XLM
    • mBART
    • T5
    • mT5
  • Evaluation benchmarks

    • GLUE
    • SQuAD
    • NER
    • SuperGLUE
    • XNLI

Suggested Readings

 
 
 
 
 

Week 11: Contextual embeddings and self-supervised learning

03:30 PM - 7:00 PM 29 January 2022 SMPS LT5, NTU, Singapore

Lecture Slide

Lecture Content

  • Why semi-supervsied?
  • Semisupervised learning dimensions
  • Pre-training and fine-tuning methods

    • CoVe
    • TagLM
    • ELMo
    • GPT
    • ULMfit
    • BERT
    • BART
  • Evaluation benchmarks

    • GLUE
    • SQuAD
    • NER
    • SuperGLUE
    • XNLI

TA: Mathieu

Pre-train Fine-tune with HF

Suggested Readings

 
 
 
 
 

Week 10: Seq2Seq Variants and Transformer

02:00 PM - 5:30 PM 22 January 2022 SMPS LT5, NTU, Singapore

Project Proposal Presentation (3-5 min video) due

Lecture Slide

Assignment 2 in (24 Jan 2pm)

Assignment 3 out

Lecture Content

  • Seq2Seq Variants (Pointer nets, Pointer Generator Nets)

    • Machine Translation
    • Summarization
  • Transformer architecture

    • Self-attention
    • Positional encoding
    • Multi-head attention

TA: Bosheng Ding

The Annotated Transformer

Suggested Readings

 
 
 
 
 

Week 9: Seq2Seg Models, Attentions, Subwords

2:00 PM - 5:30 PM 15 January 2022 SMPS LT5, NTU, Singapore

Lecture Slide

Tutorial 4

Lecture Content

  • Information bottleneck issue with vanilla Seq2Seq
  • Attention to the rescue
  • Details of attention mechanism
  • Sub-word models
  • Byte-pair encoding
  • Hybrid models

Practical exercise with Pytorch

TA: Lin Xiang

Suggested Readings

 
 
 
 
 

Week 8: Seq2Seg Models, Attentions, Subwords

2:00 PM - 5:30 PM 8 January 2022 SMPS LT5, NTU, Singapore

Lecture slides

Lecture Slide

Tutorial 4

Lecture Content

  • Information bottleneck issue with vanilla Seq2Seq
  • Attention to the rescue
  • Details of attention mechanism
  • Sub-word models
  • Byte-pair encoding
  • Hybrid models

Practical exercise with Pytorch

TA: Xuan Phi

Suggested Readings

 
 
 
 
 

Week 7: Machine translation and Seq2Seg Models

02:00 PM - 5:30 PM 18 December 2021 SMPS LT5, NTU, Singapore

Assignment 2 out

Lecture slides

Tutorial 3

Lecture Content

  • Machine translation

    • Early days (1950s)
    • Statistical machine translation or SMT (1990-2010)
    • Alignment in SMT
    • Decoding in SMT
  • Neural machine translation or NMT (2014 - )

  • Encoder-decoder model for NMT

  • Advantages and disadvantages of NMT

  • Greedy vs. beam-search decoding

  • MT evaluation

Suggested Readings

 
 
 
 
 

Week 6: Recurrent Neural Nets

02:00 PM - 5:30 PM 11 December 2021 Online (teams)

Lecture Slide

Tutorial 3

Lecture Content

  • Language modeling with RNNs
  • Backpropagation through time
  • Text generation with RNN LM
  • Sequence labeling with RNNs
  • Sequence classification with RNNs
  • Issues with Vanilla RNNs
  • Gated Recurrent Units (GRUs) and LSTMs
  • Bidirectional RNNs
  • Multi-layer RNNs

Practical exercise with Pytorch (CNN and RNN for NER)

TA: Saiful Bari

Suggested Readings

 
 
 
 
 
 
 
 
 
 

Week 4: Word Vectors (Introduction)

02:00 PM - 5:30 PM 27 November 2021 Online via Teams

Lecture Slide

Tutorial 2

Project Proposal Instructions

Lecture Content

  • Word meaning
  • Denotational semantics
  • Distributed representation of words
  • Word2Vec models (Skip-gram, CBOW)
  • Negative sampling
  • FastText
  • Evaluating word vectors

    • Intrinsic evaluation
    • Extrinsic evaluation
  • Cross-lingual word embeddings

Assignment 1 out (in NTU-Learn)

Practical exercise with Pytorch

TA: Mathieu

Numpy notebook Pytorch notebook

Suggested Readings

 
 
 
 
 

Week 3: Neural Network Basics

02:00 PM - 5:30 PM 20 November 2021 Online via Teams

Lecture Slide

Tutorial 2

Lecture Content

  • From Logistic Regression to Feed-forward NN

    • Activation functions
  • SGD with Backpropagation

  • Adaptive SGD (adagrad, adam, RMSProp)

  • Regularization (Weight Decay, Dropout, Batch normalization, Gradient clipping)

Practical exercise with Pytorch

TA: Mathieu

Numpy notebook Pytorch notebook

  • Backpropagation
  • Dropout
  • Batch normalization
  • Initialization
  • Gradient clipping
 
 
 
 
 

Week 2: Machine Learning Basics

02:00 PM - 5:30 PM 13 November 2021 Online via Teams

Lecture Slide

Tutorial 1

Lecture Content

  • What is Machine Learning?
  • Supervised vs. unsupervised learning
  • Linear Regression
  • Logistic Regression
  • Multi-class classification
  • Parameter estimation (MLE & MAP)
  • Gradient-based optimization & SGD

Practical exercise with Pytorch

TA: Mathieu

 
 
 
 
 

Week 1: Introduction

02:00 PM - 5:30 PM 6 November 2021 Online via Teams

Lecture Slide

Lecture Content

  • What is Natural Language Processing?
  • Why is language understanding difficult?
  • What is Deep Learning?
  • Deep learning vs. other machine learning methods?
  • Why deep learning for NLP?
  • Applications of deep learning to NLP
  • Knowing the target group (background, field of study, programming experience)
  • Expectation from the course

Python & PyTorch Basics

TA: Mathieu