Drawgent: Coding agent on a live Excalidraw canvas
It shows how a live drawing surface can become a front‑end code generator without leaving the browser.
The project wires Excalidraw’s collaborative API into a language model that watches the canvas as you sketch. Each stroke or shape is emitted as a JSON diff, which the agent captures in real time. Those diffs are fed into a prompt that asks the model to interpret the visual intent and emit the corresponding code fragment. The generated snippet is then injected into a preview pane, so the UI you draw materializes as runnable code instantly.
At the core is a streaming pipeline: Excalidraw → WebSocket → JSON parser → LLM prompt → code formatter → sandbox executor. The prompt template encodes a mapping from primitive shapes, rectangles, circles, arrows, to typical UI primitives such as divs, buttons, or SVG elements. The model’s output is run through a formatter to enforce syntactic correctness before being evaluated in a secure iframe. Because the diff stream is incremental, the agent only re‑generates code for the changed region, keeping latency low enough for interactive use.
For engineers, this creates a rapid prototyping feedback loop that bypasses the usual hand‑coding of mockups. Designers can iterate visually while the system keeps a live code representation, making hand‑off to developers trivial. The approach also demonstrates a concrete way to leverage LLMs for domain‑specific translation tasks, turning informal visual language into structured program artifacts without a separate design‑to‑code export step.
The trade‑offs are clear. Sketches are inherently ambiguous; the model must rely on conventions or explicit annotations to resolve intent, otherwise it may produce generic or incorrect components. Latency, while low, still depends on network round‑trips to the LLM service, which can be a bottleneck for large canvases. Generated code runs in a sandbox, but any security flaw in the model’s output could surface if the sandbox is misconfigured. Finally, the system is tightly coupled to Excalidraw’s API shape, so changes to the canvas format could break the pipeline.
TakeawayA live Excalidraw hook streams canvas JSON into an LLM, turning sketches into runnable UI code in real time.