ECE 188: Machine Learning for the Arts - Fall 2019

Resources Policies Schedule Projects Accomodations Diversity and Inclusion


Course Description

This course explores the vital new domain of Machine Learning (ML) for the arts. Though born out of computer science research, contemporary ML techniques are reimagined through creative application to diverse tasks such as style transfer, generative portraiture, music synthesis, and textual chatbots and agents. Through direct, hands-on experience with state of the art ML tools, students will develop their skills in this nascent area and form critical perspectives on the strengths and limitations of current approaches.

As ML permeates multiple aspects of culture, industry, and scholarship, it is essential both to train the next generation of ML-literate artists and engineers, and to equip them with critical tools to evaluate these new techniques. How do computational tools augment, complicate, or supercede human creative endeavor? What new approaches to artistic production are possible with the advent of affordable graphics hardware and ML software?

This project-based course will be conducted primarily in python using free, open-source machine learning and scientific computing toolkits, running on cloud-based educational computing resources. In addition to hands-on experience with ML techniques, students will become familiar with cloud-based workflows, jupyter notebooks, and kubernetes containers. Architectures and topics covered include Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), LSTMs, Wavenets, Generative Adversarial Networks (GANs) and others. Students will be responsible both for technical implementation and creative value of course projects.

Prequisites: ECE16, ECE143, or equivalent course on Python.





Work will be evaluated on the quality of concept, the degree of experimentation (both aesthetic and technical), and final realization (again, aesthetic and technical). I will share a rubric with the first project assignment.

Group Work

This course will be a mix of group work and individual work depending on each assignment. I want you each to develop your own personal research interests, but also to pool your resources and talents to produce the best projects possible. Group work is encouraged, but not required.

Project Critiques and Group Discussions

We will have critique/group discussion for each of the projects this quarter.


Introduction to Art and ML (Week 1)

Day 1: Course and Syllabus 9/29

Homework: Post something you are interested in (a project, paper, github link) to #shiny.

Day 2: Introduction to ML and the Arts 10/2

Lab 1: Make sure you can log in

Homework: Read Andrej Karpathy’s The Unreasonable Effectiveness of RNNs (2015) http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Text Generation (Week 2)

Day 3: Generative Text 10/7

Day 4: Text part 2 10/9

Lab 2: GPT-2 Examples

Homework: Project proposal

Time Series in ML (Week 3)

Day 5: Chatbots 10/14

Lab: Chatbots

Day 6: Autoencoders, Embeddings, Sketch-RNN 10/16 Autoencoders, Embeddings, Sketch-RNN

Lab: Autoencoders and online Sketch-RNN demos

Homework: Project 1

Generative Audio (Week 4)

Project 1 Due 10/20 11:59pm

Day 7: Project 1 discussion 10/21

Day 8: Intro to Generative Audio 10/23

Audio Continued (Week 5)

Day 9: Generative Networks for Music 10/28

Day 10: Speech Generation 10/30

Lab: Hands-On with ML Speech




Homework: Project proposal

Visual Processing (Week 6)

Day 11: Visual Processing 11/4


Day 12: Style Transfer 11/6


Visual Continued (Week 7)


Project 2 Due: 11/12, 11:59pm, through github classroom.

Day 14: Critique: Project 2 11/13

Visual Continued (Week 8)

Day 15: Deep Dream and Gradient Ascent 11/18


Assign Project 3: Generative Visual

Day 16: GANs 11/20


Homework: Project 3 proposal

Visual (Week 9)

Day 17: Image Captioning and Segmentation 11/25


Day 18: Platforms 11/27

Final Project Development (Week 10)

Project 3 due: 12/1, 11:59pm.

Day 19: Project 3 Critique 12/2


Day 20: Creativity Metric Activity, Final Project Check-in 12/4

Final Presentations / Exhibition (Finals Week)

FINAL TIME: Wednesday December 11, 8-11am. Location TBD.


Project 1: Generative Text

Generative Text Assignment. Due 10/20/2019, 11:59pm.

Submit online to github classroom: https://classroom.github.com/g/sJIzmAcR

Project 2: Generative Audio

Generative Audio Assignment. Due 11/12/2019, 11:59pm.

Submit online to github classroom: https://classroom.github.com/g/ujfzX5Wp

Project 3: Generative Visual

Generative Visual Assignment. Due 12/1/2019, 11:59pm.

Submit online to github classroom: https://classroom.github.com/g/AMOrRaOj

Final Project: Revisit One Project for Showcase

Refine, enhance, extend one of your earlier projects for the showcase during Finals Week.

PROJECT DUE 12/11, 8-11am. Location TBD.

REPORT DUE 12/13, 11:59pm. Add the pdf to your github, please.

For the final project you will need to submit two things:


Enrollment Fall 2019

If you intend to enroll for Fall 2019, please fill out this questionnare: https://forms.gle/iHiggRiVbPUsWMm46, and enroll for the class through the EASy system.

CMU Collaboration

We will have a couple of opportunities to interact with a similar class running this Spring at Carnegie Mellon University, as well as making a joint, online, public-facing exhibition for excellent student work (opt-in). More info coming soon!

Datahub Info

We do our processing on datahub.ucsd.edu. Here is their instruction manual:


The Office for Students with Disabilities (OSD), an Academic Affairs department, is responsible for the review of medical documentation and the determination of reasonable accommodations based on a disability. Authorization for Accommodation (AFA) letters are issued by the OSD and given to undergraduate, graduate, and Professional School students directly. If you have an AFA letter, meet with the CSE Student Affairs representative, and schedule an appointment with your instructor by the end of Week 2 to ensure that reasonable accommodations for the quarter can be arranged.

Diversity and Inclusion

We are committed to fostering a learning environment for this course that supports a diversity of thoughts, perspectives and experiences, and respects your identities (including race, ethnicity, heritage, gender, sex, class, sexuality, religion, ability, age, educational background, etc.). Our goal is to create a diverse and inclusive learning environment where all students feel comfortable and can thrive.

Our instructional staff will make a concerted effort to be welcoming and inclusive to the wide diversity of students in this course. If there is a way we can make you feel more included please let one of the course staff know, either in person, via email/discussion board, or even in a note under the door. Our learning about diverse perspectives and identities is an ongoing process, and we welcome your perspectives and input.

We also expect that you, as a student in this course, will honor and respect your classmates, abiding by the UCSD Principles of Community https://ucsd.edu/about/principles.html. Please understand that others’ backgrounds, perspectives and experiences may be different than your own, and help us to build an environment where everyone is respected and feels comfortable.If you experience any sort of harassment or discrimination, please contact the instructor as soon as possible. If you prefer to speak with someone outside of the course, please contact the Office of Prevention of Harassment and Discrimination: https://ophd.ucsd.edu/