unl-ml-art

EMAR 349: Machine Learning for the Arts - Spring 2022

Description Schedule Resources Grading Policies References

Image

Description

As Machine Learning (ML) permeates multiple aspects of culture, industry, and scholarship, it is essential that the next generation of computational artists be ML-literate, with the capacity to critically evaluate and apply this evolving technology. This course explores the vital new domain of Machine Learning as applied to the arts. Born out of computer science research, ML techniques are reimagined through creative application to diverse generative art tasks. Through hands-on experience with state-of-the-art ML tools, students will develop their skills in this area and form critical perspectives on the strengths and limitations of current approaches.

This project-based course will be conducted primarily in python using free, open-source machine learning and scientific computing toolkits, running on cloud-based educational computing resources. In addition to hands-on experience with ML techniques, students will become familiar with cloud-based workflows and jupyter notebooks. Architectures and topics covered include Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), LSTMs, Wavenets, Generative Adversarial Networks (GANs) and others. Students will be responsible both for technical implementation and creative value of course projects.

Details

Course objectives

A student who successfully completes this course will:

Prequisites: Major in Emerging Media Arts and successful completion of EMAR161 Computational Media Studio II, or permission of instructor.

Resources

Schedule

(Subject to Change-Always check back for most up to date information)

Week Topic
1a Hello. Syllabus and Policies. Compute Setup
1b Generative Systems in the Arts, Basics of Neural Nets
- Course environment setup
- python/jupyter basics;
- Exercise: introductory python/jupyterhub/OOD exercise
2 Text Generation I
- Historical Approaches;
- RNNs, LSTMs, GRUs;
- text scraping and data cleaning;
- RNN exercise;
Assign Project 1
3 Text Generation II
- Transformers;
- Fine-Tuning;
- Exercise: GPT and fine-tuning;
- Due: Project 1 Proposal
4 Chatbots, Interactive Text
- Time Series in ML
- Exercise: interactive text
5 Time II: Autoencoders, Embeddings, Sketch-RNN
- Sketch-RNN exercise
- Due: Project 1
6 Audio I
- Intro to Generative Audio
- Generative Networks for Music
- Exercise: MIDI RNN
7 Audio II
- Music transformers
- Speech generation
- Exercise: speech generation
- Assign Project 2
8 Audio III
- Speech Recognition
-vocal cloning
- Exercise: speech interaction
- Due: Project 2 Proposal
9 Visual Processing
- CNNs
- Style Transfer and Deep Dream
- Exercise: style transfer/deep dream
10 GANs
- Generative Adversarial Networks
-Exercise: GAN
- Due: Project 2
- Assign Project 3
11 Visual II
- Segmentation and masked generation - GauGAN exercise - Due Project 3 Proposal
12 Text and Image
- Image Captioning, transformers and GANs
- Text to image translation (CLIP, DALL-E, guided diffusion)
- Exercise: text and image
- Due: Final Project Proposal
13 ML and Video
- Video processing exercise
- Due: Project 3
- Exercise: Final project proposal
14 Platforms and Applications
- infrastructure for ML and the Arts
- Final project work time
15 Workshopping Final Presentations
- Final project work time
16 Exhibition/Showcase Open Studios (Final Project, Talk, Documentation due) NO FINAL EXAM

Grading

Graded activities

*Work will be evaluated on the quality of concept, the degree of experimentation (both aesthetic and technical), and final realization (again, aesthetic and technical). Prompts and rubrics will be provided with more specific details regarding each assignment and breakdowns

Description of Assignments and Exams

Weekly Exercises We will have regular, weekly programming assignments employing the tools and techniques covered in class. These will be short activities with clearly stated creative prompts and technical requirements. Projects will be graded on satisfactory completion with additional credit for creative, technical, expressive extension beyond requirements.

Projects We will have three projects over the semester (at 15% each), covering three kinds of generative media: Text, Sound, and Image. Each project will be hosted on a github repository documenting project following a uniform template. This includes a statement of concept, source code, links to data resources, discussion of results, and future directions. When assigned, students will submit a proposal/concept for their project to receive instructor feedback, and then work to complete the project. Projects will be presented and critiqued in class and github respositories will be submitted for grading.

Final Project, Documentation, and Presentation At the end of the semester, you will propose and create a self-directed ML for the Arts project engaging a subject of your choice. You may either revisit a subject or idea that excited you from earlier in the semester, or explore a topic of interest that we have not covered in class. The format, workflow, and submission of this project will follow the process of the earlier projects. In week 16 we will have a showcase for these projects, including a short talk and exhibition of the resulting work. Projects will additionally be added to an online virtual gallery of ML Arts projects.

Participation Contributions to class discussions and active participation in small group work are essential to both the momentum of the course and the development of your ideas. This requires that you come to class prepared (having completed assigned reading and writing) and ready to participate in class activities. This course is based on collaborative, project-based learning and you are also expected to contribute as a responsible member of a group. See the participation evaluation in the Grading Scale below for more information.

Attendance

On-time attendance is required as well as playtesting inside and outside of section. Please notify your instructor in advance if you must be absent for illness or family emergency. Any absences must be cleared with the instructor, or justified with written documentation (e.g. letter from team, etc.). We do not differentiate between mental and physical health and in either case please be in communication for when you need to take a day off. After a student misses a week’s worth of classes each subsequent missed class will result in the reduction of the final grade by a full letter grade (i.e., A to B, B- to C-) Excessive tardiness or leaving early will also impact your grade and will follow the same rubric.

Please also note the JCSTF attendance policy:

Late work policy

An assignment may receive an F if a student does not participate in every phase of the development of the project and meet all deadlines for preliminary materials (proposals, drafts, etc.). Failure to submit any of the graded course assignments is grounds for failure in the course. If a final draft or project, plus required addenda, is not submitted in class on the date due, it will be considered late and will lose one letter grade for each day or part of a day past due (A to B, etc.). Assignments are due in hard copy and or via email/link (online assignment). You must submit your assignments directly to the instructor. Any late submissions must be approved by your faculty instructor well in advance of the due date.

Grading Scale

A+ = 97-100 | A = 93-97 | A- = 90-93
B+ = 87-90 | B = 83-87 | B- = 80-83
C+ = 77-80 | C = 73-77 | C- = 70-73
D+ = 67-70 | D = 63-67 | D- = 60-63
F = below 60%

Here is a description of the kind of participation in the course that would earn you an A, B, C, etc. Your instructor may use pluses and minuses to reflect your participation more fairly, but this is a general description for each letter grade.

A – Excellent
Excellent participation is marked by near-perfect attendance and rigorous preparation for class. You respond to questions and activities with enthusiasm and insight and you listen and respond thoughtfully to your peers. You submit rough drafts on time, and these drafts demonstrate a thorough engagement with the assignment. You respond creatively to the feedback you receive (from both your peers and instructors) on drafts, making significant changes to your writing between the first and final drafts that demonstrate ownership of your own writing process. Finally, you are an active contributor to the peer- review and collaborative writing/making processes.

B – Good
Good participation is marked by near-perfect attendance and thorough preparation for discussion section. You respond to questions with specificity and make active contributions to creating a safe space for the exchange of ideas. You submit rough drafts on time, and these drafts demonstrate thorough engagement with the assignment. You respond effectively to the feedback you receive (from both your peers and instructor) on drafts, making changes to your work between the first and final drafts. You are a regular and reliable contributor to the peer-review and collaborative writing/making processes.

C – Satisfactory
Satisfactory participation is marked by regular attendance and preparation for class. You respond to questions when prompted and participate in classroom activities, though you may sometimes be distracted. You are present, with few absences, and have done some of the reading some of the time. You submit drafts on time and make some efforts toward revision between the first and final drafts of an assignment. You are involved in peer-review activities, but you offer minimal feedback and you may not always contribute fully to the collaborative writing/making process.

D – Unsatisfactory
Unsatisfactory participation is marked by multiple absences from section and a consistent lack of preparation. You may regularly be distracted by materials/technology not directly related to class. You submit late or incomplete work and revise minimally or only at a surface level between drafts. You are absent for peer-review activities, offer unproductive feedback, or do not work cooperatively in collaborative environments.

F—Failing
Failing participation is marked by excessive absences, a habitual lack of preparation, and failure to engage in the drafting, revision, and collaborative writing/making processes.

Academic Honesty Policy

Academic dishonesty will not be tolerated in this course. Any instances will result in an automatic grade of F in the course and possible disciplinary action under the Student Code of Conduct (https://studentconduct.unl.edu/student-code-conduct). For information on the University’s policy on academic dishonesty, please refer to the current Undergraduate Bulletin (https://registrar.unl.edu/academic-honesty).

We will use many open source projects to make our work. It is ok to use others’ code. However, you need to cite your sources, and you need to do transformative work/make it your own.

Policies

UNL Course Policies and Resources. Students are responsible for knowing the university policies and resources found on this page (https://go.unl.edu/coursepolicies):

Attendance

On-time attendance is required as well as work inside and outside of section. Please notify your instructor in advance if you must be absent for illness or family emergency. Any absences must be cleared with the instructor, or justified with written documentation (e.g. letter from team, etc.). We do not differentiate between mental and physical health and in either case please be in communication for when you need to take a day off. After a student misses a week’s worth of classes each subsequent missed class will result in the reduction of the final grade by a full letter grade (i.e., A to B, B- to C-) Excessive tardiness or leaving early will also impact your grade and will follow the same rubric.

Please also note the JCSTF attendance policy:

Addenda

CMU Collaboration

We will have a couple of opportunities to interact with a similar class running this Spring at Carnegie Mellon University, as well as making a joint, online, public-facing exhibition for excellent student work (opt-in). More info coming soon!

References