Course curriculum

  • 1

    Session 1: An Introduction to AI Music

    • Session 1 Recording

    • Main Topics: AI Music Case Studies, Course Roadmap

    • An Introduction to AI Music (50')

    • Session 1 PDF handout

    • Open Discussion
  • 2

    Session 2: Setting up your environment

    • Session 2 Recording

    • Main Topics: Environment Setup

    • GitHub Repository

    • Setting up your environment - Cloning the class repository (10')

    • Hands On

    • Setting up your environment - Hands On 1.1: Loading, visualizing, playing audio (10')

    • Setting up your environment - Hands On 1.2: Extracting Audio Features, RMS and ZCR (13')

    • Setting up your environment - Hands On 1.2: Extracting Audio Features, Spectrograms (13')

    • Setting up your environment - Hands On 2: Manipulating MIDI Data (20')
  • 3

    Session 3: Core Machine Learning Concepts for Music and Audio

    • Session 3 Recording

    • Main Topics: Audio vs Symbolic Music, Basics of Generative AI, Data Acquisition and Ethics

    • Statistical Basics of Generative Modeling in Artificial Intelligence (10')

    • Variational Autoencoders (16')

    • Hands On

    • Hands On 1.1: Lakh MIDI Dataset (12')

    • Hands On 1.2: Free Music Archive (6')

    • Hands On 2: Using RAVE
  • 4

    Session 4: Real-Life Collaborations between Artists and Engineers with Guest Speaker Jordan Rudess

    • Session 4 Recording

    • Main Topics: Human-Computer Interaction, Iterative Design, Continuous Deployment

    • Human-Computer Interaction & User-Centered Design

    • Open Discussion with Jordan Rudess
  • 5

    Session 5: Representation Learning for Music

    • Session 5 Recording

    • Deep Dive into MIDI & Spectrograms

    • Comparing Musical Representations & Encodec Deep Dive (14')

    • Understanding RVQ in Encodec (12')

    • Hands On

    • Hands On: Encodec (29')
  • 6

    Session 6: Autoregressive Music Generation

    • Session 6 Recording

    • Main Topics: Autoregressive modeling, the Transformer architecture, HuggingFace Hub

    • The Transformer architecture (15')

    • Understanding Anticipatory Music Transformers (13')

    • Hands On

    • Hands On: Using AMT to generate MIDI data (Part 1) (18')

    • Hands On: Using AMT to generate MIDI data (Part 2) (17')
  • 7

    Session 7: Autoregressive Music Generation (Part 2)

    • Session 7 Recording

    • Main Topics: MusicGen & Audio Generation with Transformers

    • Understanding MusicGen (8')

    • Hands On

    • Hands On: Using MusicGen to generate audio (38')
  • 8

    Session 8: Diffusion Models for Music Generation

    • Session 8 Recording

    • Main Topics: Diffusion Models, Latent Diffusion Models

    • Intro to Diffusion Models Part 1 (11')

    • Intro to Diffusion Models Part 2 (14')

    • Conditioning & Classifier-Free Guidance (10')

    • The UNet Architecture (6')

    • Inference-Time Optimization: DITTO (6')

    • Hands On

    • Hands On: Using Stable Audio Part 1 (15')

    • Hands On: Using Stable Audio Part 2 (18')
  • 9

    Session 9: Real-Time Generative AI & Commercial Applications of Generative AI in Music [Guest: Christian Steinmetz]

    • Session 9 Recording

    • Introduction to TorchScript (18')

    • 8-bit Linear Quantization (9')

    • ONNX and Graph Optimizations (15')

    • Main Topics: Landscape of companies in AI and Music, Available Commercial Products

    • Demo
  • 10

    Session 10: Final Project Planning

    • Session 10 Recording 1

    • Session 10 Recording 2

    • Session 10 Recording 3

    • Main Topics: Setting up a project specification, timeline, and scope

    • Training Part 1 (8')

    • Training Part 2 (13')

    • Training Part 3 (16')

    • Peer Review & Feedback
  • 11

    Session 11: Final Project Lab

    • Session 11

    • Lab Session: Guided Coding & Troubleshooting

    • Milestone Check-Ins
  • 12

    Session 12: Project Showcase & Next Steps

    • Session 12 Recording

    • Final Presentations

    • Next Steps