CS839 Data Management for Machine Learning Applications

We are seeing widespread investments in machine learning (ML) that enable computers to interpret what they see, communicate in natural language, answer complex questions, and interact with their environment. There is a hidden catch, however: all state-of-the-art ML systems rely on high-effort data management tasks like data exploration, data preparation and data cleaning. The goal of this seminar course is to study data management challenges that arise in the context of machine learning pipelines. The focus will be on cutting-edge problems in the context of ML pipelines, related to (1) data exploration and understanding, (2) data integration, cleaning, and validation, and (3) data preparation for ML models and serving of production ML applications. The seminar will be very interactive and collaborative. The topics covered and the depth of coverage will depend on the participants' input and interests. The goal of the course is to give you an indepth look at an important, emerging topic in data management research. This course will provide research opportunities in the areas of data management, human-computer interaction, and machine learning. Along the way, you will also pick up some practical experience in reading and presenting research papers, synthesizing research across desperate areas, using existing tools, and doing a course project that ideally will lead to a publishable paper.

Class Format

• This is a seminar course. Each class will consist of presentations and discussion. Students will be required to do a class project for the course (60%). A significant portion of the grade will be based on class participation, which includes paper presentations, contributions to paper reviews, and paper discussions (40%). Because of the interactive nature of the course, and space limitations, auditing is discouraged.

Prerequisites

• Mathematical maturity and a basic course in probability required. Background in algorithms, databases, machine learning, and graphical models suggested.

Assignments

• You will need to form groups of up to three people and work on certain assignments as described below.
• A research class project: Each group will work on a research project and file a single submission. The project will be broken down to five assignments: (1) initial research proposal, (2) Intermediate report, (3) final report, (4) final presentation (in class), and (5) poster presentation (people from the entire department will be invited).
• Questions, Comments, and Responses (QCRs): During each class you will need to provide 3 questions and 3 comments for each (mandatory) paper. QCs will be individual assignments. For each class one of the groups (see above) will be responsible to lead a discussion on the papers and provide answers to all posted questions during class. Comments and answers will then be summarized in a written report which will be submitted and shared with everyone in the class.
• There will be no midterm or final exams.

Misc

• Class time may be adjusted to accomodate external talks releated to the class.
• Google drive for deliverables: CS839 Drive

---

#	Date	Topic	Lecture Materials	Reading Material	Assignments
Introduction and Class Overview
1	1/23	Logistics and Data Management for Production ML	Lecture 1	Kaggle: The state of data science Polyzotis et al. Data Management Challenges in Production Machine Learning, SIGMOD 2017
2	1/25	DB and ML integration: A systems percpective	Lecture 2	Inside SystemML Kumar et al. Data Management in Machine Learning, SIGMOD 2017 System Challenges for AI
Data Exploration and Understanding
3	1/30	What to expect from data: Data driven visualizations	Lecture 3	Vartak et al. SeeDB, VLDB 2015 Enabling Data Science for the Majority (Read but no QCRs) Siddiqui et al. ZenVisage, VLDB 2017 (Optional) Gonzalez et al. Google Fusion Tables, SIGMOD 2010 (Optional)
4	2/1	From data cubes to feature-based analysis	Lecture 4	Jogklekar et al. Interactive Data Exploration with Smart Drill-Down, ICDE 2016 Sathe and Sarawagi Intelligent Rollups in Multidimensional OLAP Data, VLDB 2001 (Optional) Harinarayan et al. Implementing Data Cubes Efficiently, SIGMOD 1996 (Optional) Kahng et al. Visual Exploration of Machine Learning Results using Data Cube Analysis, HILDA Workshop @ SIGMOD 2016 (Optional)
5	2/6	Leave no relevant data behind: Data search	Lecture 5	Halevy et al. Goods: Organizing Google's Datasets, SIGMOD 2016 Hellerstein et al. Ground: A Data Context Service, CIDR 2017 (Optional) Rekatsinas et al. Finding Quality in Quantity, CIDR 2015 (Optional)
6	2/8	Data hubs: Version control for datasets	Lecture 6	Bhattacherjee et al. Principles of Dataset Versioning, VLDB 2015 Miao et al. ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows, HILDA Workshop@SIGMOD 2017 (Optional)
Data Preparation: Extraction, Integration, Cleaning
7	2/13	Knowledge Base Construction: From dark data to insights	Lecture 7	Shin et al. Incremental Knowledge Base Construction Using DeepDive, VLDB 2015 Dong et al. Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion, KDD 2014 (Optional) Carlson et al. Toward an Architecture for Never-Ending Language Learning , AAAI 2010 (Optional)
8	2/15	No Class (Theo @ SysML)
9	2/20	Creating Training Data	Lecture 8	Ratner et al. Snorkel: Rapid Training Data Creation with Weak Supervision, VLDB 2018 Weak Superivsion blog (Read but no QCRs) Ratner et al. Data Programming: Creating Large Training Sets, Quickly, NIPS 2016 (Optional) Carlson et al. Toward an Architecture for Never-Ending Language Learning , AAAI 2010 (Optional)	Example class projects posted here!
10	2/22	Data Integration: Entity Resolution 1	Lecture 9	Fellegi and Sunter A Theory for Record Linkage, Journal of the American Statistical Association 1969 Getoor and Machanavajjhala Entity Resolution: Tutorial, VLDB 2012 no QCRs Cohen et al. A Comparison of String Metrics for Matching Names and Records, IJCAI 2003 (Optional) Singla and Domingos Entity Resolution with Markov Logic, ICDM 2006 (Optional)
11	2/27	Data Integration: Entity Resolution 2	Lecture 10	Wang et al. Entity Matching: How Similar is Similar, VLDB 2011 Konda et al. Magellan: Toward Building Entity Matching Management Systems, VLDB 2016 (Optional) Das et al. Falcon: Scaling Up Hands-Off Crowdsourced Entity Mathing to Build Cloud Services, SIGMOD 2017 (Optional)	Project proposal due!
12	3/1	Data Integration: Data Fusion	Lecture 11	Rekatsinas et al. SLiMFast: Guaranteed Results for Data Fusion and Source Reliability, SIGMOD 2017 Li et al. Truth Finding on the Deep Web: Is the Problem Solved?, VLDB 2013 (Optional) Dong et al. Integrating conflicting data: the role of source dependence, VLDB 2009 (Optional)
13	3/6	Data Wrangling	Lecture 12	Kandle et al. Wrangler: Interactive Visual Specification of Data Transformation Scripts, CHI 2011 Raza and Gulwani Automated Data Extraction Using Predictive Program Synthesis., AAAI 2017 (Optional) Jin et al. Foofah: Transforming Data By Example, SIGMOD 2017 (Optional)
14	3/8	Data Cleaning: Error Detection	Lecture 13	Abedjan et al. Detecting Data Errors: Where are we and what needs to be done?, VLDB 2016 Wu and Madden Scorpion: Explaining away outliers in aggregate queries, VLDB 2013 (Optional) Chu et al. Holistic data cleaning: Putting violations into context., ICDE 2013 (Optional)
15	3/13	Data Cleaning: Error Repairing	Lecture 14	Rekatsinas et al. HoloClean: Holistic Data Repairs with Probabilistic Inference, VLDB 2017 Hellesrstein Quantitative Data Cleaning for Large Databases, 2008 (Optional) Fagin et al.Dichotomies in the Complexity of Preferred Repairs, PODS 2015 (Optional)
16	3/15	Data Cleaning and Machine Learning	Lecture 15	Krishnan et al. http://www.vldb.org/pvldb/vol9/p948-krishnan.pdf, 2016 Krishnan et al. BoostClean: Automated Error Detection and Repair for Machine Learning, 2017(Optional) Dolatshah et al.Cleaning Crowdsourced Labels Using Oracles For Supervised Learning, 2018 (Optional)
17	3/20	Management of Data under Uncertainty	Lecture 16	Dalvi et al. Probabilistic Databases: Diamonds in the Dirt, 2009 Dalvi et al. Efficient Query Evaluation on Probabilistic Databases, 2004(Optional)
18	3/22	Summary of Data Preparation	Lecture 17	No Readings.
RBMS and ML Integration
19	4/3	Relational vs Linear Algebra: One semi-ring to rule them all!	Lecture 18	Joglekar et al. Aggregations over Generalized Hypertree Decompositions, PODS 2016 Aberger et al. LevelHeaded: A Unified Engine for Business Intelligence and Linear Algebra Querying , ICDE 2018 Jananthan et al. Polystore Mathematics of Relational Algebra, 2017(Optional) Hutchison et al. LaraDB: A Minimalist Kernel for Linear and Relational Algebra Computation , BeyondMR 2017(Optional)	Intermediate Report due!
20	4/5	ML and Data Systems: SQL and UDFs	Lecture 19	Luo et al. Scalable Linear Algebra on a Relational Database System, 2017 Hellerstein et al. The MADlib Analytics Library or MAD Skills, the SQL, 2012
21	4/10	ML and Data Systems: Statistical Relational Learning Engines	Lecture 20	Zhang and Re DimmWitted: A Study of Main-Memory Statistical Analytics, 2014 Niu et al. Tuffy: Scaling up Statistical Inference in Markov Logic Networks using an RDBMS (Optional), 2011 Tran et al. Edward: A library for probabilistic modeling, inference, and criticism (Optional), 2017
22	4/12	DB-inspired ML Systems: From Linear Algebra to Execulation Plans and Rewrites	Lecture 21	Boehm et al. SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs, 2014
23	4/17	DB-inspired ML Systems: Compress, Scan, Index	Lecture 22	Elgohary et al.Scaling Machine Learning via Compressed Linear Algebra, 2017 Boehm et al. On Optimizing Operator Fusion Plans for Large-Scale Machine Learning in SystemML (Optional), 2011
24	4/19	ML serving: Feature engineering	Lecture 23	Zhang et al. Materialization optimizations for feature selection workloads, SIGMOD 2014 Nargesian et al. Learning Feature Engineering for Classification, IJCAI 2017 (Optional)
25	4/24	ML serving: Model hubs; repositories for your models	Lecture 24	Miao et al. ModelHub: Towards Unified Data and Lifecycle Management for Deep Learning, 2016
26	4/26	Project presentations 1
27	5/1	Project presentations 2
28	5/3	Poster presentations

Questions and Comments	20%
Responses and Discussion	20%
Project Proposal	10%
Project Intermediate Report	10%
Project Final Report	30%
Project Presentation and Poster	10%

Theo: by appointment @ Room CS4361

There will be no late dates for the project deliverables. However, you have the option to skip up to three QCs. Additional extensions may be granted in the case of a severe medical or family emergency.

The template of this website was created by HazyReseach@Stanford.