CS839 Modern Data Management and Machine Learning Systems

Modern Data Analytics and Machine learning (ML) are enjoying rapidly increasing adoption. However, designing and implementing the systems that support modern data analytics and machine learning in real-world deployments presents a significant challenge, in large part due to the radically different development and deployment profile of modern data analysis methods, and the range of practical concerns that come with broader adoption.

In this seminar course, we will describe the latest trends in modern data management and machine learning systems designs to better support the next generation of ML applications, and applications of ML to optimize the architecture and the performance of data management systems.

Class Format

• This is a seminar course. The format of this course will be a mix of lectures, seminar-style discussions, student presentations. Students will be responsible for paper readings, conference-style reviews, and completing a hands-on project. For projects, we will strongly encourage teams of three people.

Assignments

• Paper reviews/presentations (Individual assignments).
• Semester Project (Group assignment). We encourage students to find projects that relate to their ongoing research.
• There will be no midterm or final exams.

Misc

• The lecture schedule may be updated to accomodate obligations of the instructor.

---

Week	Date	Topic	Lecture Materials	Reading Material	Assignments
Introduction and Class Overview
1	1/21	Logistics and The Technical Aspects of Production ML	Lecture 1 (pdf)	MLSys: MLSys Whitepaper A Few Useful Things to Know About Machine Learning Polyzotis et al. Data Management Challenges in Production Machine Learning, SIGMOD 2017	Sign-up for paper presentations and scribes here.
Machine Learning Life-Cycle
1	1/23	Machine Learning Life-Cycle: A systems' percpective	Lecture 2 (pdf)
2	1/28	Machine Learning Life-Cycle: A systems' percpective	Paper Review and Discussion	Hidden Technical Debt in Machine Learning Systems (to review) Data Engineering Bulletin: Machine Learning Life-cycle Management Accelerating the Machine Learning Lifecycle with MLflow TFX: A TensorFlow-Based Production-Scale Machine Learning Platform MODELDB: A System for Machine Learning Model Management	Submit your reviews in Canvas.
2	1/30	No Class	No Lecture
Training Data Collection
3	2/4	Weak-Supervision: Automating The Collection of Training Data	Lecture 3 (pdf)	Weak Supervision Notes Notes on Self-Supervised Learning Multi-Task Learning Overview
3	2/6	Weak-Supervision: Automating The Collection of Training Data	Paper Review and Discussion	Training Complex Models with Multi-Task Weak Supervision (to review) A Theoretical Analysis of Contrastive Unsupervised Representation Learning Ludwig: a type-based declarative deep learning toolbox A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective AutoAugment: Learning Augmentation Strategies From Data	Submit your reviews in Canvas.
Data Validation
4	2/11	Managing Noisy Data	Lecture 4 (pdf) Additional Slides
4	2/13	Managing Noisy Data	Paper Review and Discussion	A Formal Framework For Probabilistic Unclean Databases (to review) Stronger Data Poisoning Attacks Break Data Sanitization Defenses How Hard is Inference for Structured Prediction? Sever: A Robust Meta-Algorithm for Stochastic Optimization MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius	Submit your reviews in Canvas.
Adaptive Data Management
5	2/18	Learnable Data Structures	Lecture 5 (pdf)		Register your Project Groups here.
5	2/20	Learnable Data Structures	Paper Review and Discussion	The Case for Learned Index Structures (to review) Learning Data Structure Alchemy Learning Multi-Dimensional Indexes The Periodic Table of Data Structures Learned Cardinalities: Estimating Correlated Joins with Deep Learning	Submit your reviews in Canvas.
Project Proposal Review
6	2/25	No Class	Proposal Submission Deadline (by end-of-day).		Please read the instructions for submitting your Project Proposal on Canvas.
6	2/27	Meetings with Instructor to Review Project Proposals		Sign-up sheet (15-minute slots):
Project Proposal Review
7	3/3	No Class (MLSys 2020)	No Lecture
7	3/5	Meetings with Instructor to Review Project Proposals		Sign-up sheet (15-minute slots):
ML in Database Systems
8	3/10	ML in Database Systems	Lecture 6 (pdf)
8	3/12	Optimizations for Feature Selection Workloads	Paper Review and Discussion	Materialization optimizations for feature selection workloads (to review) Learning generalized linear models over normalized data Factorized Databases In-Database Factorized Learning In-Database Learning with Sparse Tensors	Submit your reviews in Canvas.
Spring Break
Automated Machine Learning (AutoML)
10	3/24	Efficient and Robust Automated Machine Learning	Lecture 7 (pdf)	Lecture notes Bayesian optimization lecture notes Recorded lecture
10	3/26	Efficient and Robust Automated Machine Learning	Paper Review and Discussion	Lecture Notes Lecture notes Recorded lecture Reading Efficient and Robust Automated Machine Learning (to review) Neural Architecture Search with Reinforcement Learning Practical Bayesian Optimization of Machine Learning Algorithms Democratizing Data Science through Interactive Curation of ML Pipelines The Evolved Transformer	Submit your reviews in Canvas.
Distributed Model Training
11	3/31	Distributed Model Training	Lecture 8 (pdf)	Recorded lecture (part 1) Recorded lecture (part 2)
11	4/2	Distributed Model Training	Paper Review and Discussion	Scaling Distributed Machine Learning with the Parameter Server (to review) Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent Large Scale Distributed Deep Networks ATOMO: Communication-efficient Learning via Atomic Sparsification Scaling Distributed Machine Learning with In-Network Aggregation	Submit your reviews in Canvas.
Efficient Prediction Serving
12	4/7	Model Cascades	Lecture 9 (pdf)	Lecture notes(pdf)
12	4/9	Model Cascades	Paper Review and Discussion	Clipper: A Low-Latency Online Prediction Serving System (to review) Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference Focus: Querying Large Video Datasets with Low Latency and Low Cost Pretzel: Opening the Black Box of Machine Learning Prediction Serving Systems Accelerating Machine Learning Inference with Probabilistic Predicates	Submit your reviews in Canvas.
Model Compression
13	4/14	Low-Precision and Low-Rank Learning	Lecture 10 (pdf)	Lecture notes (pdf)
13	4/16	Low-Precision and Low-Rank Learning	Paper Review and Discussion	The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (to review) Training deep neural networks with low precision multiplications A Survey of Model Compression and Acceleration for Deep Neural Networks Understanding and Optimizing Asynchronous Low-Precision Stochastic Gradient Descent Distilling the Knowledge in a Neural Network	Submit your reviews in Canvas.
Relational Inductive Bias in Neural Networks
14	4/21	Graph (Neural) Networks	Lecture 11
14	4/23	Graph (Neural) Networks	Paper Review and Discussion	Relational inductive biases, deep learning, and graph networks (to review) A Comprehensive Survey on Graph Neural Networks Deep Graph Infomax Representation Tradeoffs for Hyperbolic Embeddings Label-Free Supervision of Neural Networks with Physics and Domain Knowledge	Submit your reviews in Canvas.
Project Presentations Week
15	4/28	Project Presentations Part 1
15	4/30	Project Presentations Part 2
Final Project Report
16	5/8	Final Project Report Due (by end-of-day)			Instructions for submitting your Final Project Report can be found here.

Reviews and Presentations	20%
Project Presentation	20%
Project Final Report	60%

Theo: by appointment @ Room CS4361

There will be no late dates for the project deliverables. However, you have the option to skip up to two reviews. Additional extensions may be granted in the case of a severe medical or family emergency.

The template of this website was created by HazyReseach@Stanford.