ECE 188: Machine Learning for the Arts - Spring 2019


Details Resources Policies Code Schedule Projects CMU Collaboration Accomodations Diversity and Inclusion

Course Description

This course explores the vital new domain of Machine Learning (ML) for the arts. Though born out of computer science research, contemporary ML techniques are reimagined through creative application to diverse tasks such as style transfer, generative portraiture, music synthesis, and textual chatbots and agents. Through direct, hands-on experience with state of the art ML tools, students will develop their skills in this nascent area and form critical perspectives on the strengths and limitations of current approaches.

As ML permeates multiple aspects of culture, industry, and scholarship, it is essential both to train the next generation of ML-literate artists and engineers, and to equip them with critical tools to evaluate these new techniques. How do computational tools augment, complicate, or supercede human creative endeavor? What new approaches to artistic production are possible with the advent of affordable graphics hardware and ML software?

This project-based course will be conducted primarily in python using free, open-source machine learning and scientific computing toolkits, running on cloud-based educational computing resources. In addition to hands-on experience with ML techniques, students will become familiar with cloud-based workflows, jupyter notebooks, and kubernetes containers. Architectures and topics covered include Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), LSTMs, Wavenets, Generative Adversarial Networks (GANs) and others. Students will be responsible both for technical implementation and creative value of course projects.

Prequisites: ECE16, ECE143, or equivalent course on Python. Students need to attend the bootcamp 3/21 to enroll in the course.






Work will be evaluated on the quality of concept, the degree of experimentation (both aesthetic and technical), and final realization (again, aesthetic and technical). I will share a rubric with the first project assignment.

Group Work

This course will be a mix of group work and individual work depending on each assignment. I want you each to develop your own personal research interests, but also to pool your resources and talents to produce the best projects possible.

Project Critiques and Group Discussions

We will have critique/group discussion for each of the projects this quarter.


Code examples are here: https://github.com/roberttwomey/ml-art-code


We do our processing on datahub.ucsd.edu. Here is their instruction manual:


Introduction to Art and ML (Week 1)

Day 1: Course and Syllabus 4/1/2019

Homework: Sign up for slack and post something you are interested in (a project, paper, github link) to #shiny.

Day 2: Introduction to ML and the Arts 4/3/2019

Lab 1: Get set up in our environment 4/5/2019

Text Generation (Week 2)

Day 3: Generative Text 4/8/2019

Day 4: Text part 2 4/10/2019

Lab 2: Project 1 Work Time

Time Series in ML (Week 3)

Day 5: Autoencoders, Embeddings, Sketch-RNN 4/15/2019

Day 6: Hands-On with VAE, Sketch-RNN 4/17/2019

Project 1 Due 4/18/2019, 11:59pm.

Lab 3: Project 1 discussion 4/19/2019

Drawing cont., Generative Audio (Week 4)

Day 7: Handwriting 4/22/2019

Day 8: Generative Audio 4/24/2019

Lab 3: Project 2 Work 4/26/2019

Audio Continued (Week 5)

Day 9: Generative Networks for Music 4/29/2019

Day 10: class cancelled 5/1/2019

sick day

Project 2 Due: 5/2/2019, 11:59pm. Submit online to github classroom: https://classroom.github.com/a/gP_-KCrL

Lab 4: Project 2 Discussion 5/3/2019

Audio, then Visual Processing (Week 6)

Day 11: Speech Generation 5/6/2019

Day 12: Visual Processing 5/8/2019

Visual Continued (Week 7)

Day 13: Work Day 5/13/2019

Day 14: Style Transfer, Continued 5/15/2019

Project 3 Due: 5/16/2019 at 11:59pm.

Submit online to github classroom: https://classroom.github.com/a/JPtQMEm9

Visual Continued (Week 8)

Day 15: Deep Dream and Gradient Ascent 5/20/2019

Assign Project 4: Generative Visual

Day 16: GANs 5/22/2019

Visual (Week 9)

Day 17: MEMORIAL DAY 5/27/2019

No class!

Day 18: Recognition 5/29/2019

Project 4 due 5/30/2019

Final Project Development (Week 10)

Lecture: Alternate Platforms

Final Presentations / Exhibition (Finals Week)

Final Work due for Exhibition


Project 1: Generative Text

Generative Text Assignment, due 4/18/2019, 11:59pm.

Submit online to github classroom: https://classroom.github.com/a/F_S6X9eN

Project 2: Sketching with ML

Sketching with ML Assignment, due 5/2/2019, 11:59pm.

Submit online to github classroom: https://classroom.github.com/a/gP_-KCrL

Project 3: Generative Audio

Generative Audio Assignment, due 5/16/2019, 11:59pm.

Submit online to github classroom: https://classroom.github.com/a/JPtQMEm9

Project 4: Generative Visual

Due 5/30/2019, 11:59pm. Open ended.

Submit online to github classroom: https://classroom.github.com/g/tngoWMjM

Final Project: Revisit One Project for Exhibit

Refine and enhance one of your earlier projects for the final exhibit during Finals Week.

CMU Collaboration

We will have a couple of opportunities to interact with a similar class running this Spring at Carnegie Mellon University, as well as making a joint, online, public-facing exhibition for excellent student work (opt-in). More info coming soon!


The Office for Students with Disabilities (OSD), an Academic Affairs department, is responsible for the review of medical documentation and the determination of reasonable accommodations based on a disability. Authorization for Accommodation (AFA) letters are issued by the OSD and given to undergraduate, graduate, and Professional School students directly. If you have an AFA letter, meet with the CSE Student Affairs representative, and schedule an appointment with your instructor by the end of Week 2 to ensure that reasonable accommodations for the quarter can be arranged.

Diversity and Inclusion

We are committed to fostering a learning environment for this course that supports a diversity of thoughts, perspectives and experiences, and respects your identities (including race, ethnicity, heritage, gender, sex, class, sexuality, religion, ability, age, educational background, etc.). Our goal is to create a diverse and inclusive learning environment where all students feel comfortable and can thrive.

Our instructional staff will make a concerted effort to be welcoming and inclusive to the wide diversity of students in this course. If there is a way we can make you feel more included please let one of the course staff know, either in person, via email/discussion board, or even in a note under the door. Our learning about diverse perspectives and identities is an ongoing process, and we welcome your perspectives and input.

We also expect that you, as a student in this course, will honor and respect your classmates, abiding by the UCSD Principles of Community https://ucsd.edu/about/principles.html. Please understand that others’ backgrounds, perspectives and experiences may be different than your own, and help us to build an environment where everyone is respected and feels comfortable.If you experience any sort of harassment or discrimination, please contact the instructor as soon as possible. If you prefer to speak with someone outside of the course, please contact the Office of Prevention of Harassment and Discrimination: https://ophd.ucsd.edu/