Statistics 37793

Topics in Deep Learning: Discriminative Models

Lecture: MW 1:50pm-3:10pm, on Zoom (link on Canvas).
Instructor: Yi Sun (yi.sun@uchicago.edu), office hours 3:10pm-4:10pm Wednesday on Zoom (link on Canvas)
Website: Access this site at bit.ly/3pJrjic
Project Discussion: On Slack (link on Canvas)

Overview and prerequisites

Description: This course will explore modern approaches to optimization, data augmentation, and domain shift for deep neural networks from both theoretical and empirical perspectives. Participation will require independent investigation with PyTorch as well as paper presentations.

Prerequisites: STAT 37601/CMSC 25025, STAT 37710/CMSC 35400, or equivalent. Contact me by email if you have questions about whether this course is appropriate for you.

Course Policies

Lectures: The course will begin with 3 weeks of lectures on modern approaches to initialization, optimization, and generalization for deep networks.

Projects: Students will be divided into groups to complete projects on the 6 topics below. Each student will be in a group for 2 projects. Papers and references for each topic are here.

• Week of Feb. 1: Batch normalization
• Week of Feb. 8: Evolution of the NTK
• Week of Feb. 15: Implicit bias of SGD
• Week of Feb. 22: Data augmentation
• Week of Mar. 1: Average-case robustness
• Week of Mar. 8: Adversarial robustness

Each group will read papers assigned for each of these topics, (optionally) search the literature for more recent papers on the subject, and finally choose a paper to base a project on. They will give presentations on a pre-approved subset of the papers in the first session of each week and present the results of their experiments in the second session of each week.

Groups should submit the following by email:

• 2 weeks before the first presentation: A project proposal and experimental plan. The project should either (1) reproduce the main result of a paper on a manageable dataset and attempt a small variation, or (2) investigate an original question suggested by the topic. If you attempt a project of type (2), please clear your plan with me on Slack in advance.
• 1 week before the first presentation: A 2-3 page progress report on the experimental results so far.
• End of week of the presentation: A 5-10 page project report.

You should also feel free to discuss the project with me over Slack at any time.

Compute: Please use Google Colab for compute. Here are a few additional resources:

• I recommend using Weights and Biases to track and share training runs.
• See here for a guide to using PyTorch and W&B to train a model on Colab.
• The course will use PyTorch. You may use TensorFlow if you already know it, but I advise most students to use PyTorch.

Grading: The course will be graded pass/fail based on project work.

Accessibility: The University of Chicago is committed to ensuring equitable access to our academic programs and services. Students with disabilities who have been approved for the use of academic accommodations by Student Disability Services (SDS) and need a reasonable accommodation(s) to participate fully in this course should follow the procedures established by SDS for using accommodations. Timely notifications are required in order to ensure that your accommodations can be implemented. Please meet with me to discuss your access needs in this class after you have completed the SDS procedures for requesting accommodations. Information on the SDS registration process is available here.

Recording: The Recording and Deletion Policies for the current academic year can be found in the Student Manual under Petitions, Audio, and Video Recording on Campus.

• Do not record, share, or disseminate any course sessions, videos, transcripts, audio, or chats.
• Do not share links for the course to those not currently enrolled.
• Zoom cloud recordings will be automatically deleted 90 days after the completion of the recording.

COVID-19: Students who have been exposed to or who are experiencing symptoms of COVID-19 should contact UChicago Student Wellness immediately to be tested, and reach out to their area Dean of Students to request accommodations for classes until:

• At least 10 days have passed since symptoms first appeared and;
• At least 3 days (72 hours) have passed since recovery, defined as resolution of fever without the use of fever-reducing medications and improvement in respiratory symptoms (cough, shortness of breath).

Getting help: Please come to office hours, email me, or chat on Slack if you would like to discuss the course material.