Imagine an intelligent whiteboard that can understand your freehand sketches and convert them into interactive mathematical constructions and physical simulations. We are tinkering with this idea to create human-in-the-loop, real-time effects via a prototype with interconnected, multimodal representations and AI.