Introduction
What if AI could see what you see and guide you in real time?
That idea led to the creation of OmniGuide AI, a real-time multimodal assistant powered by Gemini Live API and deployed using Google Cloud Run.
Instead of typing questions into a chatbot, users simply:
Point their phone camera at a problem
Ask a question using voice
Receive live spoken guidance and visual overlays
OmniGuide acts like an expert standing beside you, helping with tasks like repairing devices, cooking, learning, or troubleshooting.
This article explains how we built OmniGuide AI using Google AI models and Google Cloud, for the purposes of entering the #GeminiLiveAgentChallenge.
The Idea
Most AI assistants today require typing prompts.
But real-world problems happen in physical environments:
Fixing a leaking pipe
Understanding a device error
Cooking a recipe
Solving homework
OmniGuide AI bridges the gap by combining:
Live camera input
Voice interaction
AI reasoning
Real-time guidance
Tech Stack
OmniGuide uses Google AI and cloud infrastructure to create a low-latency multimodal agent.
AI Model
Gemini 1.5 Flash
Used for:
Vision understanding
Voice conversation
Context reasoning
Real-time instruction generation
Streaming AI Interface
Gemini Live API
Allows the app to process:
Video frames
Audio input
Real-time prompts
Backend Infrastructure
Google Cloud Run
Provides:
Scalable AI inference endpoints
Fast container deployment
Low latency API routing
Frontend
Built using:
WebRTC for camera streaming
WebSockets for real-time AI responses
React for UI
Canvas overlays for visual guidance
Architecture
High-level system flow:
User opens OmniGuide
Camera stream begins
Voice input captured
Frames + audio sent to Gemini Live API
Gemini analyzes the scene
AI generates instructions
Voice response + overlay returned
Result: AI guidance in real time.
Key Features
Real-Time Visual Understanding
Gemini analyzes live camera frames to understand objects and environments.
Voice Interaction
Users can simply ask:
“What is this error?”
“How do I fix this?”
Step-by-Step Guidance
The AI provides instructions such as:
pointing to the correct component
highlighting objects
describing the next step
Visual Overlays
On-screen guides help users follow instructions easily.
Example Use Cases
Home Repair
Point the camera at a leaking pipe and ask:
“How do I fix this?”
Cooking
Show ingredients and ask:
“What can I cook with these?”
Education
Students can show math problems or experiments.
Device Troubleshooting
Scan error messages and get solutions instantly.
Challenges We Faced
Real-Time Latency
Handling live video + AI inference required careful optimization.
We solved this by:
compressing frames
streaming only key frames
using Gemini Flash for faster responses.
Multimodal Context
Ensuring Gemini correctly interprets visual context required structured prompts and scene summaries.
What Makes OmniGuide Unique
OmniGuide transforms AI from a chat interface into a real-time expert assistant.
Instead of searching online tutorials, users simply:
show the problem and ask for help.
What's Next
Future improvements include:
AR overlays
smart object detection
multi-step task memory
collaborative remote assistance
Conclusion
OmniGuide AI demonstrates how Google AI models and Google Cloud can power the next generation of multimodal live agents.
By combining vision, voice, and reasoning, we move beyond chatbots into AI that understands the physical world.
This article was created for the purposes of entering the #GeminiLiveAgentChallenge.
More...
What if AI could see what you see and guide you in real time?
That idea led to the creation of OmniGuide AI, a real-time multimodal assistant powered by Gemini Live API and deployed using Google Cloud Run.
Instead of typing questions into a chatbot, users simply:
Point their phone camera at a problem
Ask a question using voice
Receive live spoken guidance and visual overlays
OmniGuide acts like an expert standing beside you, helping with tasks like repairing devices, cooking, learning, or troubleshooting.
This article explains how we built OmniGuide AI using Google AI models and Google Cloud, for the purposes of entering the #GeminiLiveAgentChallenge.
The Idea
Most AI assistants today require typing prompts.
But real-world problems happen in physical environments:
Fixing a leaking pipe
Understanding a device error
Cooking a recipe
Solving homework
OmniGuide AI bridges the gap by combining:
Live camera input
Voice interaction
AI reasoning
Real-time guidance
Tech Stack
OmniGuide uses Google AI and cloud infrastructure to create a low-latency multimodal agent.
AI Model
Gemini 1.5 Flash
Used for:
Vision understanding
Voice conversation
Context reasoning
Real-time instruction generation
Streaming AI Interface
Gemini Live API
Allows the app to process:
Video frames
Audio input
Real-time prompts
Backend Infrastructure
Google Cloud Run
Provides:
Scalable AI inference endpoints
Fast container deployment
Low latency API routing
Frontend
Built using:
WebRTC for camera streaming
WebSockets for real-time AI responses
React for UI
Canvas overlays for visual guidance
Architecture
High-level system flow:
User opens OmniGuide
Camera stream begins
Voice input captured
Frames + audio sent to Gemini Live API
Gemini analyzes the scene
AI generates instructions
Voice response + overlay returned
Result: AI guidance in real time.
Key Features
Real-Time Visual Understanding
Gemini analyzes live camera frames to understand objects and environments.
Voice Interaction
Users can simply ask:
“What is this error?”
“How do I fix this?”
Step-by-Step Guidance
The AI provides instructions such as:
pointing to the correct component
highlighting objects
describing the next step
Visual Overlays
On-screen guides help users follow instructions easily.
Example Use Cases
Home Repair
Point the camera at a leaking pipe and ask:
“How do I fix this?”
Cooking
Show ingredients and ask:
“What can I cook with these?”
Education
Students can show math problems or experiments.
Device Troubleshooting
Scan error messages and get solutions instantly.
Challenges We Faced
Real-Time Latency
Handling live video + AI inference required careful optimization.
We solved this by:
compressing frames
streaming only key frames
using Gemini Flash for faster responses.
Multimodal Context
Ensuring Gemini correctly interprets visual context required structured prompts and scene summaries.
What Makes OmniGuide Unique
OmniGuide transforms AI from a chat interface into a real-time expert assistant.
Instead of searching online tutorials, users simply:
show the problem and ask for help.
What's Next
Future improvements include:
AR overlays
smart object detection
multi-step task memory
collaborative remote assistance
Conclusion
OmniGuide AI demonstrates how Google AI models and Google Cloud can power the next generation of multimodal live agents.
By combining vision, voice, and reasoning, we move beyond chatbots into AI that understands the physical world.
This article was created for the purposes of entering the #GeminiLiveAgentChallenge.
More...