Build Your Own AI Voice Assistant with an ESP32 and Meta's Muse Gadgets SDK

For years, "build your own voice assistant" meant stitching together a wake-word engine, a speech-to-text service, a language model, a text-to-speech service, and a lot of glue code. Most of those projects ended up half-finished in a drawer.

In October 2026, Meta released Muse Gadgets: open-source firmware and SDKs that connect boards you build yourself to its Muse AI assistant. The code lives on GitHub at facebookincubator/muse-gadget-sdk, mostly under the Apache 2.0 license. It has three parts:

esp32/ — firmware for ESP32 microcontroller boards (displays, microphones, speakers, sensors, buttons)
linux/ — an SDK for Raspberry Pi and other Linux single-board computers
skills/ — ways to extend what the assistant can do on your device

Flash a supported board, pair it with the Muse app on your phone, and the assistant can listen, talk back, draw on a screen, and read the hardware around it. This guide walks through one complete build and then gives you five more to try.

How It Actually Works (Read This First)

The AI is not running on the chip. An ESP32-S3 has 512 KB of SRAM and a few megabytes of PSRAM — nowhere near enough for a large language model. Muse runs in Meta's cloud. Your gadget is a smart terminal:

You press a button and speak.
The board records audio and streams it to Muse over an encrypted connection.
Muse thinks, then sends back speech, text, and images.
The board plays the reply through its speaker and draws on its screen.

That has three practical consequences:

Every gadget needs Wi-Fi. ESP32 boards have no cellular radio. Away from home, run it off your phone's hotspot. Remember that the ESP32 only speaks 2.4 GHz Wi-Fi — if your router uses one combined network name for 2.4 and 5 GHz and pairing fails, that is the first thing to check.
No internet, no assistant. If Muse's servers are down, your gadget is a nice-looking paperweight.
Check the app works where you live. You need the Muse app and a Muse account. Before you buy hardware, confirm the app is available and you can sign in from India.

The fine print

Be clear-eyed about what this is. Meta's token terms say the SDK and tokens are not a supported product and access can change or be withdrawn at any time. The repo itself warns that custom firmware "can brick boards and void warranties." This is a tinkerer's playground, not a consumer product. If that sounds like fun, read on.

Build 1: The Talking AI Puck (Full Walkthrough)

Difficulty: 2/5 | Build Time: 2–3 hours (mostly downloads) | Soldering: None

Meta's recommended starter board is the Waveshare ESP32-S3-Touch-AMOLED-1.75C: an ESP32-S3 with a round 1.75" AMOLED touch screen, microphone, speaker, and a battery connector, all on one board. The firmware supports its full UI — push-to-talk, settings, and images on the round screen. On your desk it looks like a tiny glowing AI puck.

What you need

Item	Notes
Waveshare ESP32-S3-Touch-AMOLED-1.75C	The only part that matters. Check our development boards
USB-C data cable	Many cheap cables are charge-only and will not work for flashing
Small LiPo battery (optional)	For cordless use. Match the board's connector and polarity
A computer	Windows, macOS, or Linux
A phone	With the Muse app installed

That's it. No breadboard, no jumper wires.

Step 1: Get an SDK token

The token links your hardware to your Muse account. It ends up baked into the firmware, so the repo treats it as an identifier rather than a secret — but don't post it publicly or commit it to GitHub anyway. If it ever leaks, revoke it on the site and create a new one.

Step 2: Turn on Developer mode in the Muse app

Install the Muse app, sign in, and go to Settings > Devices > Developer mode. This lets the app find and pair hardware you built yourself.

Step 3: Install ESP-IDF v6.0.1

The firmware is built with Espressif's ESP-IDF, and the repo is strict about the version: v6.0.1, and other versions are not supported. If you have used the Arduino IDE before, ESP-IDF is the lower-level official toolchain. Our ESP-IDF vs Arduino guide explains the difference.

Easiest route: install VS Code, add the Espressif IDF extension, run its setup wizard with the Express option, and pick v6.0.1 when it asks for a version. It downloads around 2 GB, so start it and go make chai.

Command line (macOS/Linux), as documented in the repo:

git clone -b v6.0.1 --recursive https://github.com/espressif/esp-idf.git ~/esp/esp-idf-v6
~/esp/esp-idf-v6/install.sh esp32s3

Then, in every new terminal before you build:

. ~/esp/esp-idf-v6/export.sh

Check it worked:

idf.py --version

It must print v6.0.1. If you see build errors about missing headers later, a wrong ESP-IDF version is almost always the cause.

Step 4: Download the firmware

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
cd muse-gadget-sdk/esp32

Step 5: Connect the board and find its port

Plug the board in with your data cable, then:

macOS: ls /dev/cu.usbmodem*
Linux: ls /dev/ttyACM* — if you get "permission denied" later, run sudo usermod -a -G dialout $USER, then log out and back in
Windows: open Device Manager, expand Ports (COM & LPT), and note the COM number (e.g. COM5)

Replace PORT in the commands below with what you found. If nothing shows up at all, swap the cable before you try anything else.

Step 6: Back up the factory firmware

Flashing replaces whatever the board shipped with. Save a copy first so you can go back:

python -m esptool --chip esp32s3 -p PORT read-flash 0 ALL factory-backup.bin

This reads the entire flash chip and takes a few minutes. Keep factory-backup.bin somewhere safe.

Step 7: Add your token

This board is configured by layering settings files on top of each other: the repo's defaults, the Muse settings, and a profile written specifically for this Waveshare board (screen, touch, audio, and buttons). Open the configuration menu with those layers applied:

idf.py -B build-muse-waveshare-s3-175c -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-waveshare-s3-175c/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-waveshare-s3-175c" menuconfig

Navigate to ESP32 Device SDK > Muse Gadgets SDK token, paste your mgst_... token, save, and exit.

While you are in menuconfig, the repo strongly recommends turning on NVS encryption (CONFIG_HOMEHUB_NVS_ENCRYPTION), which protects the Wi-Fi credentials and pairing data stored on the board.

Step 8: Build

Same command, with build at the end:

idf.py -B build-muse-waveshare-s3-175c -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-waveshare-s3-175c/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-waveshare-s3-175c" build

The first build compiles all of ESP-IDF and takes several minutes. Later builds are much faster.

Step 9: Flash

Same again, with -p PORT flash at the end:

idf.py -B build-muse-waveshare-s3-175c -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-waveshare-s3-175c/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-waveshare-s3-175c" -p PORT flash

If it fails to connect: hold the BOOT button, tap RESET, release BOOT, and run the command again. This forces the ESP32-S3 into download mode.

When flashing finishes, the board reboots into the Muse firmware.

Step 10: Pair it

In the Muse app, go to Settings > Devices > Add Device (the + icon).
Your board shows up as MuseGadget-XXXXXX. Select it.
When the board asks you to confirm pairing, press its BOOT button.
Give it your Wi-Fi details when the app asks.

The status indicator tells you where it is: orange breathing means ready for setup, blue breathing means press the button to confirm, solid blue means connecting, and green means connected. Red blinking means something went wrong — run idf.py -p PORT monitor to read the log.

Pairing has no manufacturer verification, so do it at home on a network you trust, not on café Wi-Fi.

Step 11: Talk to it

Press the board's wake button (the firmware profile uses the top button), speak, and Muse answers through the speaker with pictures on the round screen.

Good first uses:

Kitchen helper: timers, unit conversions, "how much is 200 grams of maida in cups?"
Study buddy: quick definitions and explanations without unlocking your phone
Bedtime stories: with pictures on the round display

On boards with PSRAM, like this one, the firmware also includes a home-network tunnel, so Muse can reach devices on your local network through the gadget.

If you need to start over

Reset setup: hold the button for 5 seconds, then pair again in the app.
Changed or revoked your token? Reflash, hold the button for 5 seconds, and pair again. The board keeps using the old token until you do.
Complete wipe: idf.py -p PORT erase-flash. A normal reflash keeps your Wi-Fi and pairing; this does not.
Go back to stock: python -m esptool --chip esp32s3 -p PORT write-flash 0 factory-backup.bin

Five More Muse Builds to Try

The repo supports more than 30 boards, from bare dev kits to finished-looking gadgets. Support varies: some boards get the full UI with images, others are voice-only or display-only, and a few are marked experimental. Check the board table in the repo's esp32/README.md before you buy.

1. The wrist assistant — Seeed SenseCAP Watcher

A round-screen ESP32-S3 device with a microphone, speaker, and a scroll wheel. It has a helper script:

tools/muse/board.sh build watcher
tools/muse/board.sh flash watcher

Two warnings from the repo:

Back up the nvsfactory partition before flashing. Muse overwrites the factory data, and the restore steps are in the repo's devices/README.md.
Flashing goes slowly and quietly. The Watcher's USB bridge needs a slow, paced flash that prints nothing for about three minutes. Do not unplug it or press Ctrl+C — interrupting it can leave the board unable to boot.

Pair it, connect it to your phone's hotspot, and you have an assistant on your wrist — as long as your phone is nearby.

2. The pocket buddy — M5Stack StickS3

A stick-shaped ESP32-S3 with a small colour screen, speaker, microphone, and buttons, small enough for a shirt pocket:

tools/muse/board.sh build sticks3
tools/muse/board.sh flash sticks3

It runs off your phone's hotspot on the go, which makes it a fun travel companion for asking quick questions without pulling out your phone.

3. The e-ink desk display

E-paper uses almost no power between updates and stays readable in bright daylight, which makes it perfect for a morning briefing or to-do list on your desk. Two officially supported routes:

Small: Waveshare ESP32-S3-1.54inch-ePaper V2 — tools/board.sh waveshare-s3-epaper-154 build
Large: Seeed reTerminal E1002 colour e-ink display (one of Meta's three recommended starter devices) — tools/board.sh reterminal-e1002 build

Any other e-paper panel means writing your own display driver. Our SPI, I2C and UART guide is a good place to start if you want to try that.

4. The free one — Raspberry Pi home bridge

Already have a Raspberry Pi gathering dust? The Linux SDK turns it into a Muse gadget. Meta recommends the Raspberry Pi 5, but any Linux box can work. Because it is a full Linux computer, it can run custom commands on your behalf, which makes it a natural bridge to home automation. You still need an SDK token and Developer mode, just like the ESP32 builds. If the Pi is already yours, this build costs nothing.

5. The desk robot with a Muse brain

The most satisfying build of all. Take a small robot body — a couple of servo motors for a nodding head, a 3D-printed shell (see our enclosure guide), and a lithium cell — and give it a Muse brain.

The Waveshare 1.75C from Build 1 makes an ideal robot face: the round AMOLED becomes the eyes and avatar, and the built-in microphone handles listening. Design your enclosure around the round board, and the mechanical side becomes the fun part.

Want to start from scratch on a budget instead? The plain Espressif ESP32-S3-DevKitC-1 is supported too (tools/board.sh espressif-s3-devkitc-1 build). It has no screen, microphone, or speaker of its own, so you will be adding I2S audio and a display yourself — more work, but a great way to learn how every piece fits together. Our ESP32 vs ESP32-S3 vs ESP32-C3 comparison explains why the S3 is the right chip for voice projects.

Troubleshooting Quick Reference

Problem	Fix
Computer doesn't see the board	Try a different USB-C cable — most likely charge-only
"Failed to connect" while flashing	Hold BOOT, tap RESET, release BOOT, retry
Build errors about missing headers	Wrong ESP-IDF version — it must be v6.0.1
Board doesn't appear in the app	Turn on Developer mode; check Bluetooth is on
Stuck on solid blue	Wi-Fi problem — make sure you're on a 2.4 GHz network
Settings changes not taking effect	Delete the `build-...` directory and rebuild
Purple status light	Not paired — pair again in the app
Red blinking	Read the log with `idf.py -p PORT monitor`

The Bottom Line

Muse Gadgets is Meta saying: here's our assistant, here's the firmware, go build something strange. Getting started costs one dev board, a USB cable, and a free token. Some builds (the Waveshare puck, the SenseCAP Watcher, the Raspberry Pi bridge) work with ready-made configs today. Others (a custom e-ink display, the robot) need you to adapt the code, which is exactly the kind of weekend project this program was made for.

Pick one idea, grab the parts from our development boards range, and flash something this weekend. The worst outcome is a board you have to restore from backup and a good story. The best outcome is an AI companion you built yourself, sitting on your desk and talking back.

Links: firmware at github.com/facebookincubator/muse-gadget-sdk · tokens at gadgets.muse.ai

Disclaimer: This article was written with the help of AI. We checked the commands and steps against the official muse-gadget-sdk documentation, but the SDK is new and changes often, so always confirm against the repo before flashing your board.

esp32/ — firmware for ESP32 microcontroller boards (displays, microphones, speakers, sensors, buttons)
linux/ — an SDK for Raspberry Pi and other Linux single-board computers
skills/ — ways to extend what the assistant can do on your device

How It Actually Works (Read This First)

You press a button and speak.
The board records audio and streams it to Muse over an encrypted connection.
Muse thinks, then sends back speech, text, and images.
The board plays the reply through its speaker and draws on its screen.

That has three practical consequences:

Every gadget needs Wi-Fi. ESP32 boards have no cellular radio. Away from home, run it off your phone's hotspot. Remember that the ESP32 only speaks 2.4 GHz Wi-Fi — if your router uses one combined network name for 2.4 and 5 GHz and pairing fails, that is the first thing to check.
No internet, no assistant. If Muse's servers are down, your gadget is a nice-looking paperweight.
Check the app works where you live. You need the Muse app and a Muse account. Before you buy hardware, confirm the app is available and you can sign in from India.

The fine print

Build 1: The Talking AI Puck (Full Walkthrough)

Difficulty: 2/5 | Build Time: 2–3 hours (mostly downloads) | Soldering: None

What you need

Item	Notes
Waveshare ESP32-S3-Touch-AMOLED-1.75C	The only part that matters. Check our development boards
USB-C data cable	Many cheap cables are charge-only and will not work for flashing
Small LiPo battery (optional)	For cordless use. Match the board's connector and polarity
A computer	Windows, macOS, or Linux
A phone	With the Muse app installed

That's it. No breadboard, no jumper wires.

Step 1: Get an SDK token

Step 2: Turn on Developer mode in the Muse app

Install the Muse app, sign in, and go to Settings > Devices > Developer mode. This lets the app find and pair hardware you built yourself.

Step 3: Install ESP-IDF v6.0.1

Command line (macOS/Linux), as documented in the repo:

git clone -b v6.0.1 --recursive https://github.com/espressif/esp-idf.git ~/esp/esp-idf-v6
~/esp/esp-idf-v6/install.sh esp32s3

Then, in every new terminal before you build:

. ~/esp/esp-idf-v6/export.sh

Check it worked:

idf.py --version

It must print v6.0.1. If you see build errors about missing headers later, a wrong ESP-IDF version is almost always the cause.

Step 4: Download the firmware

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
cd muse-gadget-sdk/esp32

Step 5: Connect the board and find its port

Plug the board in with your data cable, then:

macOS: ls /dev/cu.usbmodem*
Linux: ls /dev/ttyACM* — if you get "permission denied" later, run sudo usermod -a -G dialout $USER, then log out and back in
Windows: open Device Manager, expand Ports (COM & LPT), and note the COM number (e.g. COM5)

Replace PORT in the commands below with what you found. If nothing shows up at all, swap the cable before you try anything else.

Step 6: Back up the factory firmware

Flashing replaces whatever the board shipped with. Save a copy first so you can go back:

python -m esptool --chip esp32s3 -p PORT read-flash 0 ALL factory-backup.bin

This reads the entire flash chip and takes a few minutes. Keep factory-backup.bin somewhere safe.

Step 7: Add your token

idf.py -B build-muse-waveshare-s3-175c -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-waveshare-s3-175c/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-waveshare-s3-175c" menuconfig

Navigate to ESP32 Device SDK > Muse Gadgets SDK token, paste your mgst_... token, save, and exit.

While you are in menuconfig, the repo strongly recommends turning on NVS encryption (CONFIG_HOMEHUB_NVS_ENCRYPTION), which protects the Wi-Fi credentials and pairing data stored on the board.

Step 8: Build

Same command, with build at the end:

idf.py -B build-muse-waveshare-s3-175c -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-waveshare-s3-175c/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-waveshare-s3-175c" build

The first build compiles all of ESP-IDF and takes several minutes. Later builds are much faster.

Step 9: Flash

Same again, with -p PORT flash at the end:

idf.py -B build-muse-waveshare-s3-175c -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-waveshare-s3-175c/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-waveshare-s3-175c" -p PORT flash

If it fails to connect: hold the BOOT button, tap RESET, release BOOT, and run the command again. This forces the ESP32-S3 into download mode.

When flashing finishes, the board reboots into the Muse firmware.

Step 10: Pair it

In the Muse app, go to Settings > Devices > Add Device (the + icon).
Your board shows up as MuseGadget-XXXXXX. Select it.
When the board asks you to confirm pairing, press its BOOT button.
Give it your Wi-Fi details when the app asks.

Pairing has no manufacturer verification, so do it at home on a network you trust, not on café Wi-Fi.

Step 11: Talk to it

Press the board's wake button (the firmware profile uses the top button), speak, and Muse answers through the speaker with pictures on the round screen.

Good first uses:

Kitchen helper: timers, unit conversions, "how much is 200 grams of maida in cups?"
Study buddy: quick definitions and explanations without unlocking your phone
Bedtime stories: with pictures on the round display

On boards with PSRAM, like this one, the firmware also includes a home-network tunnel, so Muse can reach devices on your local network through the gadget.

If you need to start over

Reset setup: hold the button for 5 seconds, then pair again in the app.
Changed or revoked your token? Reflash, hold the button for 5 seconds, and pair again. The board keeps using the old token until you do.
Complete wipe: idf.py -p PORT erase-flash. A normal reflash keeps your Wi-Fi and pairing; this does not.
Go back to stock: python -m esptool --chip esp32s3 -p PORT write-flash 0 factory-backup.bin

Five More Muse Builds to Try

1. The wrist assistant — Seeed SenseCAP Watcher

A round-screen ESP32-S3 device with a microphone, speaker, and a scroll wheel. It has a helper script:

tools/muse/board.sh build watcher
tools/muse/board.sh flash watcher

Two warnings from the repo:

Back up the nvsfactory partition before flashing. Muse overwrites the factory data, and the restore steps are in the repo's devices/README.md.
Flashing goes slowly and quietly. The Watcher's USB bridge needs a slow, paced flash that prints nothing for about three minutes. Do not unplug it or press Ctrl+C — interrupting it can leave the board unable to boot.

Pair it, connect it to your phone's hotspot, and you have an assistant on your wrist — as long as your phone is nearby.

2. The pocket buddy — M5Stack StickS3

A stick-shaped ESP32-S3 with a small colour screen, speaker, microphone, and buttons, small enough for a shirt pocket:

tools/muse/board.sh build sticks3
tools/muse/board.sh flash sticks3

It runs off your phone's hotspot on the go, which makes it a fun travel companion for asking quick questions without pulling out your phone.

3. The e-ink desk display

E-paper uses almost no power between updates and stays readable in bright daylight, which makes it perfect for a morning briefing or to-do list on your desk. Two officially supported routes:

Small: Waveshare ESP32-S3-1.54inch-ePaper V2 — tools/board.sh waveshare-s3-epaper-154 build
Large: Seeed reTerminal E1002 colour e-ink display (one of Meta's three recommended starter devices) — tools/board.sh reterminal-e1002 build

Any other e-paper panel means writing your own display driver. Our SPI, I2C and UART guide is a good place to start if you want to try that.

4. The free one — Raspberry Pi home bridge

5. The desk robot with a Muse brain

Troubleshooting Quick Reference

Problem	Fix
Computer doesn't see the board	Try a different USB-C cable — most likely charge-only
"Failed to connect" while flashing	Hold BOOT, tap RESET, release BOOT, retry
Build errors about missing headers	Wrong ESP-IDF version — it must be v6.0.1
Board doesn't appear in the app	Turn on Developer mode; check Bluetooth is on
Stuck on solid blue	Wi-Fi problem — make sure you're on a 2.4 GHz network
Settings changes not taking effect	Delete the `build-...` directory and rebuild
Purple status light	Not paired — pair again in the app
Red blinking	Read the log with `idf.py -p PORT monitor`

The Bottom Line

Links: firmware at github.com/facebookincubator/muse-gadget-sdk · tokens at gadgets.muse.ai

Disclaimer: This article was written with the help of AI. We checked the commands and steps against the official muse-gadget-sdk documentation, but the SDK is new and changes often, so always confirm against the repo before flashing your board.

Build Your Own AI Voice Assistant with an ESP32 and Meta's Muse Gadgets SDK

How It Actually Works (Read This First)

The fine print

Build 1: The Talking AI Puck (Full Walkthrough)

What you need

Step 1: Get an SDK token

Step 2: Turn on Developer mode in the Muse app

Step 3: Install ESP-IDF v6.0.1

Step 4: Download the firmware

Step 5: Connect the board and find its port

Step 6: Back up the factory firmware

Step 7: Add your token

Step 8: Build

Step 9: Flash

Step 10: Pair it

Step 11: Talk to it

If you need to start over

Five More Muse Builds to Try

1. The wrist assistant — Seeed SenseCAP Watcher

2. The pocket buddy — M5Stack StickS3

3. The e-ink desk display

4. The free one — Raspberry Pi home bridge

5. The desk robot with a Muse brain

Troubleshooting Quick Reference

The Bottom Line

Related Posts

5 IoT Projects You Can Build This Weekend

Home Automation with ESP32 and LoRa: A Step-by-Step Guide

Building a WiFi Weather Station with ESP32 and BME280

Build Your Own AI Voice Assistant with an ESP32 and Meta's Muse Gadgets SDK

How It Actually Works (Read This First)

The fine print

Build 1: The Talking AI Puck (Full Walkthrough)

What you need

Step 1: Get an SDK token

Step 2: Turn on Developer mode in the Muse app

Step 3: Install ESP-IDF v6.0.1

Step 4: Download the firmware

Step 5: Connect the board and find its port

Step 6: Back up the factory firmware

Step 7: Add your token

Step 8: Build

Step 9: Flash

Step 10: Pair it

Step 11: Talk to it

If you need to start over

Five More Muse Builds to Try

1. The wrist assistant — Seeed SenseCAP Watcher

2. The pocket buddy — M5Stack StickS3

3. The e-ink desk display

4. The free one — Raspberry Pi home bridge

5. The desk robot with a Muse brain

Troubleshooting Quick Reference

The Bottom Line

Related Posts

5 IoT Projects You Can Build This Weekend

Home Automation with ESP32 and LoRa: A Step-by-Step Guide

Building a WiFi Weather Station with ESP32 and BME280