Open-Source Music Generation: Text-to-Music & Lyrics-to-Song - AI Build & Learn

About this event

​Welcome to AI Build & Learn, a weekly AI engineering stream where we pick a new topic and learn by building together. • RSVP on the Luma to get direct live links and better reminders • You can always find the live on YouTube: https://www.youtube.com/@sagecodes/streams ​This event is about generating music and audio with AI. As with the image and video events, there's no single model we're locked into — the point is to explore what's out there and actually try a few. We'll focus on open-source models, but you're welcome to bring commercial ones (Suno, Udio, and friends) if you want to compare — worth noting there isn't a fully open-source Suno equivalent yet, though the gap is closing. ​We'll look at the two main flavors: text-to-music (instrumental / sound design from a prompt) and lyrics-to-song (full tracks with vocals and accompaniment). Under the hood these lean on the same diffusion and transformer/language-model approaches as image and video, applied to audio. I'll research and try some of the best open-source options ahead of the stream, and we'll talk through the practical tradeoffs: quality, track length, controllability, speed, and licensing. ​Some things to look up to get started: ​Open-source models: • ​YuE (YuE AI): lyrics-to-song — full tracks up to \~5 min with synchronized vocals and accompaniment • ​ACE-Step: fast and controllable — a \~4-min song in seconds; diffusion + linear-transformer design • ​MusicGen (Meta / AudioCraft): versatile text-to-music with melody conditioning (note: CC BY-NC — non-commercial output license) • ​Stable Audio Open (Stability AI): great for ambient/textural audio, SFX, and samples (short clips, not full songs) ​Tooling: • ​AudioCraft (Meta) — MusicGen / AudioGen: https://github.com/facebookresearch/audiocraft (https://github.com/facebookresearch/audiocraft?utm_source=luma) • ​Hugging Face — audio models and pipelines: https://huggingface.co/models?pipeline_tag=text-to-audio (https://huggingface.co/models?pipeline_tag=text-to-audio&utm_source=luma) ​​​Resources • ​​​GitHub: https://github.com/sagecodes/ai-build-and-learn • ​​​Events Calendar: https://luma.com/ai-builders-and-learners • ​​​Slack (Discuss during the week): https://slack.flyte.org/ • ​​​Hosted by Sage Elliott: https://www.linkedin.com/in/sageelliott/ ​​In this stream • Intro to topic • ​​​​Community Discussion • Practical examples ​​​Community challenge (optional) ​​​Try spending 30–90 minutes during the week learning or building something related to the topic, then share what you’re working on in Slack. ​​​Note on Flyte / Union ​​​You may see Flyte used in some demos. Flyte is an open-source AI orchestration platform maintained by Union (where I work) for building scalable, durable, and observable AI workflows. You do not need to use Flyte to participate. • ​​​Union: https://www.union.ai/ • ​​​Flyte: https://flyte.org/ ​​​Drop a comment with ideas for future topics (agents, RAG, MLOps, robotics, frameworks, and more).