mori

Async & Parallel R with {mirai} | Charlie Gao | Data Science Lab

The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We’d love to have you! The Lab is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow. On this call, Libby Heeren is joined by Charlie Gao, who walks through async and parallel programming using the mirai package for R. Charlie is the author of {mirai} and {mori}! Charlie demonstrates how mirai lets you run R code in parallel across multiple cores or even distributed across remote machines. He covers the key differences between parallel and async programming, shows how to set up worker daemons (said like “demons”), scale resources dynamically, and connect to remote machines via SSH. Whether you’re running long computations, training models, or building Shiny apps, mirai helps you make the most of your computing resources without blocking your main R session:) Hosting crew from Posit: Libby Heeren, Isabella Velasquez Charlie Gao’s GitHub: https://github.com/shikokuchuo Charlie Gao’s Bluesky: https://bsky.app/profile/shikokuchuo.net Charlie Gao’s LinkedIn: https://www.linkedin.com/in/charliegao/ Charlie Gao’s Mastodon: https://fosstodon.org/@shikokuchuo Resources mentioned in the video and chat: mirai package website: https://mirai.r-lib.org/ mirai GitHub repository: https://github.com/r-lib/mirai AskDeepSeek chatbot for mirai documentation: https://mirai.r-lib.org/ (click “ask deep wiki” button) mirai - Promises (Shiny and Plumber): https://mirai.r-lib.org/articles/v02-promises.html mirai - Serialization: https://mirai.r-lib.org/articles/v03-serialization.html mirai - OpenTelemetry: https://mirai.r-lib.org/articles/v05-opentelemetry.html mirai stop_mirai function: https://mirai.r-lib.org/reference/stop_mirai.html mirai skill for Claude Code: https://github.com/r-lib/mirai/blob/main/.claude/skills/mirai/SKILL.md Plumber2 package: https://plumber2.posit.co/ Daemon (computing) on Wikipedia: https://en.wikipedia.org/wiki/Daemon_(computing) ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co Hangout: https://pos.it/dsh The Lab: https://pos.it/dslab LinkedIn: https://www.linkedin.com/company/posit-software Bluesky: https://bsky.app/profile/posit.co Thanks for hanging out with us! Timestamps of Questions / Topics: 00:00 Introduction 03:33 “What is async programming and how is it different from parallel?” 06:57 AskDeepSeek chatbot for the Mirai package 09:40 Setting up the Positron IDE with activity bar on top 15:25 “What was the motivation or need for developing Mirai?” 22:10 Mirai map function for parallel processing 25:25 “Do the contents of Mirai inherit definitions from the global environment?” 29:00 “What’s the difference between Mirai versus promises and future?” 31:50 Demonstrating sequential vs parallel processing 36:00 “Can you use Mirai with HPC?” 37:07 Dynamically scaling workers by adding and removing daemons 38:38 Setting up daemons with URLs for network connections 42:05 Launching workers over SSH to remote machines 43:09 SSH tunneling to connect workers without open ports 51:14 “Does Mirai allow R to perform the same thing as NumPy?” 51:54 “Does Mirai ship with some kind of task viewing dashboard?” 52:35 “Can we use parallel::detectCores() to see how many daemons we can use?” 53:58 “Would parallel processing be more useful than async processing in typical data science work?” 55:00 Mirai skill for Claude Code and AI agents

Charlie Gao