Business problem / context
Desktop tasks often require users to switch repeatedly between applications, folders, browsers and documents. VoicePilot explores how natural-language voice commands can simplify these workflows while maintaining control over what an AI system is allowed to execute. The project also addresses the challenge of answering questions from personal documents without giving the language model unrestricted access to the operating system.
Solution
VoicePilot combines speech recognition, LLM-based intent understanding, secure action validation, document retrieval and a bounded agent workflow in one desktop-assistant experience. Users can speak or type commands to open approved applications and folders, perform web searches, create notes, save or recall memories, summarise clipboard content and ask questions about their own documents.
Outcome
The final prototype demonstrates a secure end-to-end voice-assistant architecture with both single-step and agent-based workflows. It includes document search with source references, local AI fallbacks, conversation memory, desktop automation and automated testing.
