Multimodal
A model that takes more than text: images, audio, video, PDFs. You paste the screenshot instead of describing it.
See it
What it is
A multimodal model accepts more than text in the same request: images, audio, video frames, PDF pages. Under the hood each one is encoded into tokens that sit in the same context window as your words, which is why you can hand it a screenshot and ask 'why is this button off-centre' instead of writing three paragraphs describing the screenshot.
The jobs it wins outright: screenshot to bug report, invoice or receipt to structured JSON, design mock to markup, chart image to the numbers behind it, photo to alt text, whiteboard to meeting notes. Any workflow where a human would say 'just look at it' is the candidate.
Gotcha one: images are not free. A full-resolution screenshot can cost more than a page of text, and most APIs downscale above a size cap, so a 4K screenshot of dense code arrives blurry and the model quietly misreads it. Crop to the region that matters instead of sending the whole desktop. Gotcha two: multimodal usually means multimodal in. Most models read images and still only write text back, so 'edit this image' needs an actual image generation model, not a vision one.
Ask AI for it
Build a screenshot-to-bug-report endpoint using a multimodal model. Accept an image upload plus an optional note, send them as a single message with an image content block followed by a text block, and downscale the image to at most 1568px on the long edge before sending. Instruct the model to return structured JSON with fields: summary, suspected_component, visual_symptoms (array), and confidence (low / medium / high). Tell it to say so explicitly when text in the image is too small to read rather than guessing, and reject non-image uploads before they reach the model.