unl-ml-art

EMAR 349: Machine Learning for the Arts - Spring 2024

Description Schedule Resources Grading Policies References

Image

Description

As Artificial Intelligence and Machine Learning transform all aspects of culture, industry, and scholarship, it is essential to train the next generation of AI-literate artists and engineers in these emerging technologies, and to equip them with the critical context necessary to assess their consequence. How do computational tools augment, assist, or replace traditional human creative efforts? What new forms and expressions are fostered by these tools?

This project-based course will be conducted primarily in python using free, open-source machine learning and scientific computing toolkits, running on cloud-based educational computing resources. In addition to hands-on experience with ML techniques, students will become familiar with cloud-based workflows and jupyter notebooks. Architectures and topics covered include Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Diffusion, Transformer Models, Wavenets, Generative Adversarial Networks (GANs) and others. Students will be responsible both for technical implementation and creative value of course projects.

Details

Course objectives

A student who successfully completes this course will:

Prequisites: Major in Emerging Media Arts and successful completion of EMAR161 Computational Media Studio II, or permission of instructor.

Resources

Course Schedule

(Subject to Change-Always check back for most up to date information)

Week Topic
1 Hello. Syllabus and Policies. Compute Setup
Generative Systems in the Arts
- Course environment setup
- python/jupyter basics;
- Exercise: introductory python/jupyter/colab exercise
HW: RNN in 100 lines
2 Basics of Neural Nets. Text Generation I
- Historical Approaches;
- RNNs, LSTMs, GRUs;
- text scraping and data cleaning;
- RNN exercise;
Assign Project 1
3 Text Generation II
- Transformers;
- Fine-Tuning;
- Exercise: GPT and fine-tuning;
- Due: Project 1 Proposal
4 Chatbots, Interactive Text
- Time Series in ML
- Exercise: interactive text
5 Time II: Autoencoders, Embeddings, Sketch-RNN
- Sketch-RNN exercise
- Due: Project 1
6 Audio I
- Intro to Generative Audio
- Generative Networks for Music
- Exercise: MIDI RNN
7 Audio II
- Music transformers
- Speech generation
- Exercise: speech generation
- Assign Project 2
8 Audio III
- Speech Recognition
-vocal cloning
- Exercise: speech interaction
- Due: Project 2 Proposal
9 Visual Processing
- CNNs
- Style Transfer and Deep Dream
- Exercise: style transfer/deep dream
10 GANs
- Generative Adversarial Networks
-Exercise: GAN
- Due: Project 2
- Assign Project 3
11 Visual II
- Segmentation and masked generation - GauGAN exercise - Due Project 3 Proposal
12 Text and Image
- Image Captioning, transformers and GANs
- Text to image translation (CLIP, DALL-E, guided diffusion)
- Exercise: text and image
- Due: Final Project Proposal
13 ML and Video
- Video processing exercise
- Due: Project 3
- Exercise: Final project proposal
14 Platforms and Applications
- infrastructure for ML and the Arts
- Final project work time
15 Workshopping Final Presentations
- Final project work time
16 Exhibition/Showcase Open Studios (Final Project, Talk, Documentation due) NO FINAL EXAM

Course Grading Policy

Graded activities

*Work will be evaluated on the quality of concept, the degree of experimentation (both aesthetic and technical), and final realization (again, aesthetic and technical). Prompts and rubrics will be provided with more specific details regarding each assignment and breakdowns

Description of Assignments and Exams

Weekly Exercises We will have regular, weekly programming assignments employing the tools and techniques covered in class. These will be short activities with clearly stated creative prompts and technical requirements. Projects will be graded on satisfactory completion with additional credit for creative, technical, expressive extension beyond requirements.

Projects We will have three projects over the semester at 15% each. Each of these will explore one perspective on generative AI, and each will use the variety of generative media we have learned about in class (text, audio, speech, image, video, etc.). Projects will be submitted as github repositories following a uniform template. This includes a statement of concept, source code, links to data resources, discussion of results, and future directions. When assigned, students will submit a proposal/concept for their project to receive instructor feedback, and then work to complete the project. Projects will be presented and critiqued in class and github respositories will be submitted for grading.

Final Project, Documentation, and Presentation At the end of the semester, you will propose and create a self-directed ML for the Arts project engaging a subject of your choice. You may either revisit a subject or idea that excited you from earlier in the semester, or explore a topic of interest that we have not covered in class. The format, workflow, and submission of this project will follow the process of the earlier projects. In week 16 we will have a showcase for these projects, including a short talk and exhibition of the resulting work. Projects will additionally be added to an online virtual gallery of ML Arts projects.

Participation Contributions to class discussions and active participation in critiques and workshops are essential to the momentum of the course and the development of your ideas. This requires that you come to class prepared (having completed assigned reading or exercises), and ready to participate in class activities. Bring finished work for in-class crtiqiues. See the participation evaluation in the Grading Scale below for more information.

Attendance

On-time attendance is required as well as playtesting inside and outside of section. Please notify your instructor in advance if you must be absent for illness or family emergency. Any absences must be cleared with the instructor, or justified with written documentation (e.g. letter from team, etc.). We do not differentiate between mental and physical health and in either case please be in communication for when you need to take a day off. After a student misses a week’s worth of classes each subsequent missed class will result in the reduction of the final grade by a full letter grade (i.e., A to B, B- to C-) Excessive tardiness or leaving early will also impact your grade and will follow the same rubric.

Please also note the JCSTF attendance policy:

Late work policy

An assignment may receive an F if a student does not participate in every phase of the development of the project and meet all deadlines for preliminary materials (proposals, drafts, etc.). Failure to submit any of the graded course assignments is grounds for failure in the course. If a final draft or project, plus required addenda, is not submitted in class on the date due, it will be considered late and will lose one letter grade for each day or part of a day past due (A to B, etc.). Assignments are due in hard copy and or via email/link (online assignment). You must submit your assignments directly to the instructor. Any late submissions must be approved by your faculty instructor well in advance of the due date.

Grading Scale

A+ = 96.67-100 | A = 93.33-96.67 | A- = 90-93.33
B+ = 86.66-90 | B = 83.33-86.67 | B- = 80-83.33
C+ = 76.67-80 | C = 73.33-76.67 | C- = 70-73.33
D+ = 66.67-70 | D = 63.33-66.67 | D- = 60-63.33
F = below 60%

Here is a description of the kind of participation in the course that would earn you an A, B, C, etc. Your instructor may use pluses and minuses to reflect your participation more fairly, but this is a general description for each letter grade.

A – Excellent

Excellent participation is marked by near-perfect attendance and rigorous preparation for class work. You respond to questions and activities with enthusiasm and insight and you listen and respond thoughtfully to your peers. You submit assignments on time, adhering to posted requirements, and demonstrate a thorough engagement with the assignment. You respond creatively to any feedback you receive (from both your peers and instructor). You are an active contributor to classroom community.

B – Good
Good participation is marked by near-perfect attendance and thorough preparation for class. You respond to questions with specificity and make active contributions to class. You submit assignments on time, and demonstrate a thorough engagement with the assignment. You respond effectively to the feedback you receive (from both your peers and instructor). You are a regular and reliable contributor to classroom community.

C – Satisfactory
Satisfactory participation is marked by regular attendance and preparation for class. You respond to questions when prompted and participate in classroom activities, though you may sometimes be distracted. You are present in class, with few absences, and have done some of the work some of the time. You submit assignments and make some efforts toward revision proposals and final submission. You are involved in classroom activities, but you offer minimal feedback and you may not always contribute fully to classroom community.

D – Unsatisfactory
Unsatisfactory participation is marked by multiple absences from class and a consistent lack of preparation. You may regularly be distracted by materials/technology not directly related to class. You submit late or incomplete work. You are absent for classroom activities, or do not work cooperatively in collaborative environments.

F—Failing
Failing participation is marked by excessive absences, a habitual lack of preparation, and failure to engage in classroom activities and development processes.

Policies

Course-specific policies and rules.

Attendance Late Work Other People’s Code UNL Course Policies and Resources

Attendance

On-time attendance is required as well as work inside and outside of section. Please notify your instructor in advance if you must be absent for illness or family emergency. Any absences must be cleared with the instructor, or justified with written documentation (e.g. letter from team, etc.). We do not differentiate between mental and physical health and in either case please be in communication for when you need to take a day off. After a student misses a week’s worth of classes each subsequent missed class will result in the reduction of the final grade by a full letter grade (i.e., A to B, B- to C-) Excessive tardiness or leaving early will also impact your grade and will follow the same rubric.

Please also note the JCSTF attendance policy:

Late Work

An assignment may receive an F if a student does not participate in every phase of the development of the project and meet all deadlines for preliminary materials (proposals, drafts, work in progress, etc.). Failure to submit any of the graded course assignments is grounds for failure in the course. If a final draft or project, plus required addenda, is not submitted in class on the date due, it will be considered late and will lose one 5% for each day or part of a day past due (A to B, etc.). Assignments are due in class and via online submission, as specified in assignment descriptions. Any late submissions must be approved by your faculty instructor well in advance of the due date.

Other People’s Code

We will use many open source projects to make our work. It is ok to use others’ code. However, you need to cite your sources, and you need to do transformative work/make it your own.

UNL Course Policies and Resources

Students are responsible for knowing the university policies and resources found on this page (https://go.unl.edu/coursepolicies):

Land Acknowledgment

We acknowledge that the University of Nebraska is a land-grant institution with campuses and programs on the past, present, and future homelands of the Pawnee, Ponca, Oto-Missouria, Omaha, Dakota, Lakota, Arapaho, Cheyenne, and Kaw Peoples, as well as the relocated Ho-Chunk, Iowa, and Sac and Fox Peoples. Please take a moment to consider the legacies of more than 150 years of displacement, violence, settlement, and survival that bring us together here today. This acknowledgement and the centering of Indigenous Peoples is a start as we move forward together for the next 150 years

Addenda

CMU Collaboration

We may have a couple of opportunities to interact with a similar class running this Spring at Carnegie Mellon University, as well as making a joint, online, public-facing exhibition for excellent student work (opt-in). More info coming soon!

References