Natural language processing (NLP) is one of the most important fields in artificial intelligence (AI). It has become very crucial in the information age because most of the information is in the form of unstructured text. NLP technologies are applied everywhere as people communicate mostly in language: language translation, web search, customer support, emails, forums, advertisement, radiology reports, to name a few.
There are a number of core NLP tasks and machine learning models behind NLP applications. Deep learning, a sub-field of machine learning, has recently brought a paradigm shift from traditional task-specific feature engineering to end-to-end systems and has obtained high performance across many different NLP tasks and downstream applications. Tech companies like Google, Baidu, Alibaba, Apple, Amazon, Facebook, Tencent, and Microsoft are now actively working on deep learning methods to improve their products. For example, Google recently replaced its traditional statistical machine translation and speech-recognition systems with systems based on deep learning methods.
Optional Textbooks
In this course, students will learn state-of-the-art deep learning methods for NLP. Through lectures and practical assignments, students will learn the necessary tricks for making their models work on practical problems. They will learn to implement, and possibly to invent their own deep learning models using available deep learning libraries like Pytorch.
Our Approach
Thorough and Detailed: How to write from scratch, debug and train deep neural models
State of the art: Most lecture materials are new from research world in the past 1-5 years.
Practical: Focus on practical techniques for training the models, and on GPUs.
Fun: Cover exciting new advancements in NLP (e.g., Transformer, BERT).
Weekly Workload
5% marks for class participation.Assignments (individually graded)
3 * 15% = 45% of the total assessment.10% off per day lateFinal Project (Group work but individually graded)
1–3 people per group5%, update: 5%, presentation: 10%, report: 30%Assignment 3 in
Slides on adverarial attacks (prepared by Samson@NUS-Salesforce)
Lecture Content
Adversarial attacks in NLP
Defense:
Limits & Future of Deep NLP
Suggested Readings
Lecture Content
Project final report guidelines
Lecture Content
Pre-training and fine-tuning methods
Evaluation benchmarks
Suggested Readings
Lecture Content
Pre-training and fine-tuning methods
Evaluation benchmarks
TA: Mathieu
Suggested Readings
Project Proposal Presentation (3-5 min video) due
Assignment 2 in (24 Jan 2pm)
Assignment 3 out
Lecture Content
Seq2Seq Variants (Pointer nets, Pointer Generator Nets)
Transformer architecture
TA: Bosheng Ding
Suggested Readings
Lecture Content
Practical exercise with Pytorch
TA: Lin Xiang
Suggested Readings
Lecture Content
Practical exercise with Pytorch
TA: Xuan Phi
Suggested Readings
Assignment 2 out
Lecture Content
Machine translation
Neural machine translation or NMT (2014 - )
Encoder-decoder model for NMT
Advantages and disadvantages of NMT
Greedy vs. beam-search decoding
MT evaluation
Suggested Readings
Lecture Content
Practical exercise with Pytorch (CNN and RNN for NER)
TA: Saiful Bari
Suggested Readings
Lecture Content
Suggested Readings
Lecture Content
Evaluating word vectors
Cross-lingual word embeddings
Assignment 1 out (in NTU-Learn)
Practical exercise with Pytorch
TA: Mathieu
Numpy notebook Pytorch notebook
Suggested Readings
Lecture Content
From Logistic Regression to Feed-forward NN
SGD with Backpropagation
Adaptive SGD (adagrad, adam, RMSProp)
Regularization (Weight Decay, Dropout, Batch normalization, Gradient clipping)
Practical exercise with Pytorch
TA: Mathieu
Numpy notebook Pytorch notebook
Lecture Content
Practical exercise with Pytorch
TA: Mathieu
[Supplementary]
Lecture Content
Python & PyTorch Basics
TA: Mathieu
Programming in Python
[Supplementary]
