Voice and files

malac can send you files and screenshots, type what you say, speak when it needs you, and hold a spoken conversation. All of it works in every client of the daemon: the terminal UI, the web UI, VS Code and the phone apps. Each voice feature appears once one of your providers has a model that can do it, and each can be turned off.

Files and screenshots

Ask for something to see, “send me a screenshot of the settings page” or “share the generated report”, and malac sends it with its send_to_user tool: files by path, or the newest images its tools returned, with a short caption. With the control service connected, malac screenshots its browser or your desktop and sends that; without it, it captures the screen with a command and sends the file.

What you get depends on the file:

• Images show in the conversation. Tap or click one to view it full screen.
• Other files show as a card with the name and size, which opens or downloads the file.
• In the terminal UI, each file is a row with its name and size. Click it, press o on it in the transcript, or type /open to open it in the system viewer. The line mode prints each file’s name, size and where it is.

Sent files are copies, kept in ~/.malac/shared/ for 30 days, so they still open after the original changes. Paired devices fetch them through the daemon like everything else, so they also arrive when the daemon runs on another machine.

The image viewer

Every image in a conversation opens full screen: screenshots in tool results, images attached to prompts, and images malac sent. The viewer fits the image to the screen, zooms and pans, and saves or shares it.

App	Zoom and pan	Save and share
Web UI	Wheel or trackpad pinch around the pointer, drag to pan, double-click for fit or actual size, + - 0. ← → move between a message’s images, Esc closes.	Download, and Share where the browser can
VS Code	Wheel or pinch around the pointer, drag to pan, double-click for fit or 100%, + - 0 (fit) 1 (100%), arrow keys pan, Esc closes	Open in editor (VS Code’s image preview), Save as…
Android	Pinch around your fingers, drag to pan, double-tap for fit or 2.5×, swipe down or back closes	Share, Save to Pictures
iPhone	Pinch or double-tap, drag to pan, swipe down closes. With a keyboard, ⌘= and ⌘− zoom, ⌘0 fits and Esc closes.	Share, save to Photos
Terminal UI	/open opens the image in the system viewer

The three voice features

Feature	What it does	Needs	On by default
Dictation	A microphone button in the composer: speak, and the text appears for you to edit and send	A transcription model (whisper-1, gpt-4o-mini-transcribe), or a chat model that takes audio (gpt-4o-audio-preview)	Yes
Speech	malac speaks when it needs you: approvals, questions, and when a long task finishes or something notable happens	A speech model (gpt-4o-mini-tts, tts-1), or your computer’s own voice (say on macOS, espeak-ng on Linux)	No
Talk mode	A spoken conversation: you talk, a voice model hands the work to malac and tells you what malac did and what it needs	A Realtime model (gpt-realtime)	Yes

malac finds these models in your providers’ model lists, by name. Only OpenAI-style providers have audio APIs: OpenAI itself, and gateways such as LiteLLM in front of OpenAI, Groq or a local server.

A gateway may name a model differently on its Realtime route. LiteLLM, for one, can take gpt-4o-mini-transcribe for transcription but want openai/gpt-realtime for talk mode, so if talk mode says the model is unknown, pick the provider-prefixed name (see Using an LLM gateway). A model whose name does not say what it does can be declared:

[models."lora/listener"]
audio = "transcription"   # or "input", "speech", "realtime"

Names with whisper or transcribe, tts or speech, realtime or audio are recognised without it.

Settings

Every app has a Voice section in its settings: a switch per feature, the model it uses, the voice, and the dictation language. They all write the same [voice] table in ~/.malac/config.toml:

[voice]
dictation = true
dictation_model = "openai/gpt-4o-mini-transcribe"   # unset: the first model that can
language = "en"                                     # unset: the model works it out
speech = true
speech_model = "openai/gpt-4o-mini-tts"             # or "system"
speech_voice = "coral"
talk = true
talk_model = "openai/gpt-realtime"
talk_voice = "marin"

Changes apply at once. malac voice shows what is available and in use:

$ malac voice
dictation  ready       openai/gpt-4o-mini-transcribe
                       also: openai/whisper-1
speech     ready       openai/gpt-4o-mini-tts · voice coral
talk       ready       openai/gpt-realtime · voice marin
language   en

malac voice set speech true changes a setting from the terminal, and malac voice set speech_model none goes back to the first model that can. You can also ask malac: “turn speech on”. Every key is in the configuration reference.

Where speech plays

Speech plays on one device: of those with Speak on this device on, the one you used last. Answer an approval on your phone and the next sentence plays there; go back to the laptop and it follows you.

When speech is on, malac’s agents also get a say tool, which they use sparingly: when they need you, when a long task is done, or when something is worth knowing. Approval requests are announced for them. After a turn of 30 seconds or more in which the agent said nothing, you hear “Done.” and the first sentence of its reply.

Talk mode

Start talk mode with the headset button (or /talk in the terminal UI) and speak. The voice model knows the project you are in, passes what you ask to malac, says what it asked, and when malac finishes tells you the outcome in a sentence or two. Ask how it is going, and answer approvals out loud (“yes, run it”). Your requests show in the conversation like typed ones, so you can read the details later. Talking over the voice stops it.

While talk mode runs, the voice model speaks for malac, so the separate speech above pauses.

Approvals out loud

An approval you answer out loud is passed on by the voice model. Say no when in doubt, and check the request on screen.

The phone apps and browsers cancel the speaker’s echo. In the terminal, use headphones, or malac pauses the microphone while it speaks.

In each app

• Terminal UI. Ctrl+X starts dictation when it is ready: the prompt shows a red dot and the time, Ctrl+X or Enter types what you said at the cursor, and Esc throws it away. /talk starts talk mode (its state shows in the status line; Esc ends it), /speak on|off decides whether this terminal speaks, /voice shows what is available, and /voice speech true (any key of the [voice] table) changes one. The View menu has the same switches. See The terminal UI.
• Web UI. The microphone and headset buttons sit in the composer. Ctrl+Shift+Space (⌘⇧Space on a Mac) starts and stops dictation; Enter puts the text in the message box and Esc throws the recording away. A speaker button in the header switches Speak on this device, and Settings → Voice has the rest.
• VS Code and Cursor. Editors keep the microphone from their webviews, so the extension runs the malac CLI below for the microphone and speaker. The composer’s microphone button or malac: Dictate (⌥⇧⌘D, Ctrl+Alt+Shift+D elsewhere) records; malac: Talk and the headset button open talk mode in a terminal; malac: Voice Settings has the switches, models and voices; malac.speakOnThisDevice decides whether the editor speaks. Voice needs malac on the same machine as the editor. See VS Code.
• Android and iPhone. The microphone in the composer dictates, the headset button opens talk mode, and Daemons → Voice has the settings and Speak on this device. The apps ask for the microphone the first time you use it. See Mobile.

From the command line

Command	What it does
malac voice [status] [--json] [--refresh]	What is available and in use; --refresh reads the providers’ model lists again
malac voice set <setting> <value>	true or false for dictation, speech and talk; a model, voice or language otherwise; none removes it
malac voice dictate [--json] [--max-seconds N]	Record until Enter or the end of input (at most N seconds, default 120) and print the text; Ctrl+C cancels
malac voice say <text>	Read text aloud with the speech model
malac voice play <file>	Play a WAV or MP3 file
malac voice talk [--tab ID] [--agent ID] [--duplex]	Talk mode without the full-screen UI: Esc ends, m mutes, Space stops the voice; --duplex keeps the microphone open, for headphones

Each takes -C <name> to use a saved connection’s daemon with this machine’s microphone and speaker.

macOS. malac uses Core Audio. The first recording asks whether your terminal (or editor) may use the microphone; allow it in System Settings → Privacy & Security → Microphone.

Linux. malac records with pw-record, parecord or arecord and plays with pw-play, paplay or aplay (MP3 with ffplay, mpg123 or mpv), so install the utilities of PipeWire, PulseAudio or ALSA.

Privacy

Recordings go to the daemon, which sends them to the model you chose and keeps nothing. The daemon holds the provider keys, so the apps never see them.

Browsers allow the microphone only on secure pages: https:// (a tunnel) or localhost. On a plain http:// network address the microphone buttons say so.

The protocol side, for your own clients, is in Protocol.