feat: multiple modalities from the client #263
Conversation
📝 WalkthroughWalkthroughAdds first-class multimodal support across the stack: new multimodal types, ChatClient.sendMessage accepting multimodal payloads and per-message body, UI/example image attachments, adapter MIME-type/data-URI handling, message conversion/stream changes, tests, and multimodal smoke tests. Changes
Sequence DiagramsequenceDiagram
participant React as React Component
participant ChatClient
participant StreamProc as StreamProcessor
participant Converter as MessageConverter
participant Adapter as LLMAdapter
participant LLM as LLM Provider
React->>ChatClient: sendMessage(MultimodalContent)
ChatClient->>ChatClient: normalizeMessageInput() / store pendingMessageBody
ChatClient->>StreamProc: addUserMessage(parts[], id?)
StreamProc-->>ChatClient: UIMessage(parts[])
ChatClient->>ChatClient: emit messageSent(messageId, parts[])
ChatClient->>Converter: uiMessageToModelMessages(UIMessage)
Converter-->>ChatClient: ModelMessage with ContentPart[]
ChatClient->>Adapter: convertContentParts(ContentPart[])
Adapter->>Adapter: detect mimeType / build data: URIs for base64
Adapter-->>ChatClient: provider-specific payload
ChatClient->>LLM: API request (merged body + conversationId)
LLM-->>ChatClient: response stream
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Fix all issues with AI agents
In `@docs/guides/multimodal-content.md`:
- Around line 432-442: The FileReader promise in handleFileUpload lacks error
handling and can hang if reading fails; update the promise used in
handleFileUpload to attach reader.onerror and reject the promise with the error
(and optionally reader.onabort) so callers receive an error instead of waiting
forever, and ensure any cleanup (e.g., removing handlers) happens on both
success and error paths.
In `@examples/ts-react-chat/src/routes/index.tsx`:
- Around line 362-364: The forEach callback currently uses a concise arrow body
that implicitly returns the value of URL.revokeObjectURL, which static analysis
flags; update the callback to a block body that does not return anything — e.g.,
change the call on attachedImages (attachedImages.forEach((img) =>
URL.revokeObjectURL(img.preview))) to use a statement body like
attachedImages.forEach((img) => { URL.revokeObjectURL(img.preview); }); to
ensure no value is returned from the forEach callback.
- Around line 292-300: The FileReader promise for producing `base64` lacks error
handling and can hang on read failures; change the constructor to new
Promise<string>((resolve, reject) => { ... }) and add `reader.onerror = (e) =>
reject(e)` (and optionally `reader.onabort = () => reject(new Error('File read
aborted'))`) alongside the existing `reader.onload` handler; also consider
removing/clearing handlers after resolution/rejection to avoid leaks and keep
using `reader.readAsDataURL(file)` to start the read.
- Around line 159-174: The image data URL is hardcoded to "image/png" when
part.source.type !== 'url'; update the construction of imageUrl to use the
actual MIME type from the part metadata (e.g., read a mime/type field such as
part.source.mediaType or part.source.mimeType or part.metadata.mimeType) instead
of "image/png" so the prefix becomes
`data:{actualMime};base64,${part.source.value}` when rendering in the branch
that handles part.source.type !== 'url'; keep the existing branch for URL
sources unchanged.
In `@packages/typescript/smoke-tests/adapters/src/tests/index.ts`:
- Around line 129-157: The multimodal tests (MMJ, MMP, MMS, MMT) declare
requires: ['text'] but send image content; update the test metadata and
capability enum by adding a new AdapterCapability value (e.g., 'vision') to the
AdapterCapability enum, then change the four tests (identifiable by id:
'MMJ','MMP','MMS','MMT' in the tests array) to requires: ['text','vision'];
alternatively, if you prefer the existing IMG/TTS/TRN pattern, set
skipByDefault: true on those test objects instead of changing requires — ensure
references to AdapterCapability and the test objects are updated consistently.
🧹 Nitpick comments (10)
packages/typescript/ai-openai/src/adapters/text.ts (1)
816-824: Consider usingdetectImageMimeTypefor consistent MIME type detection.The Anthropic and Gemini adapters use
detectImageMimeTypeto infer the actual image format from base64 magic bytes, but this adapter hardcodesimage/jpeg. While data URIs with incorrect MIME types often still work, using the utility would provide more accurate MIME types.♻️ Proposed fix to use detectImageMimeType
First, add the import at the top of the file:
import { detectImageMimeType } from '@tanstack/ai'Then update the data URI construction:
// For base64 data, construct a data URI if not already one const imageValue = part.source.value + const detectedMimeType = detectImageMimeType(imageValue) ?? 'image/jpeg' const imageUrl = imageValue.startsWith('data:') ? imageValue - : `data:image/jpeg;base64,${imageValue}` + : `data:${detectedMimeType};base64,${imageValue}`packages/typescript/ai-grok/src/adapters/text.ts (1)
505-517: Consider usingdetectImageMimeTypefor consistent MIME type detection.Similar to the OpenAI adapter, this hardcodes
image/jpegfor the data URI MIME type. UsingdetectImageMimeType(as done in Anthropic and Gemini adapters) would provide more accurate MIME types based on the actual image format.♻️ Proposed fix to use detectImageMimeType
First, add the import at the top of the file:
import { detectImageMimeType } from '@tanstack/ai'Then update the data URI construction:
// For base64 data, construct a data URI if not already one const imageValue = part.source.value + const detectedMimeType = detectImageMimeType(imageValue) ?? 'image/jpeg' const imageUrl = part.source.type === 'data' && !imageValue.startsWith('data:') - ? `data:image/jpeg;base64,${imageValue}` + ? `data:${detectedMimeType};base64,${imageValue}` : imageValuepackages/typescript/ai/src/utils.ts (1)
17-41: Consider handling data URI prefixes gracefully.The function assumes raw base64 input, but callers might accidentally pass a full data URI (e.g.,
data:image/png;base64,iVBOR...). This would returnundefinedsince the string starts withdata:rather than the magic bytes.♻️ Proposed enhancement to handle data URI input
export function detectImageMimeType( base64Data: string, ): 'image/jpeg' | 'image/png' | 'image/gif' | 'image/webp' | undefined { + // Strip data URI prefix if present + const data = base64Data.includes(',') + ? base64Data.split(',')[1] ?? base64Data + : base64Data + // Get first few bytes (base64 encoded) - const prefix = base64Data.substring(0, 20) + const prefix = data.substring(0, 20)packages/typescript/ai-anthropic/src/adapters/text.ts (1)
312-337: Minor optimization: detect MIME type only for base64 data sources.
detectImageMimeTypeis called unconditionally, but for URL sources the detection is wasteful since the value is a URL string (not base64) and the result is unused.♻️ Proposed optimization
case 'image': { const metadata = part.metadata as AnthropicImageMetadata | undefined - // Detect mime type from base64 magic bytes if not provided - const detectedMimeType = detectImageMimeType(part.source.value) const imageSource: Base64ImageSource | URLImageSource = part.source.type === 'data' ? { type: 'base64', data: part.source.value, media_type: - metadata?.mediaType ?? detectedMimeType ?? 'image/jpeg', + metadata?.mediaType ?? + detectImageMimeType(part.source.value) ?? + 'image/jpeg', } : { type: 'url', url: part.source.value, }packages/typescript/ai-openrouter/src/adapters/text.ts (1)
593-605: Consider usingdetectImageMimeTypefor consistent MIME type detection.Similar to OpenAI and Grok adapters, this hardcodes
image/jpegfor the data URI MIME type. For consistency with Anthropic and Gemini adapters, consider usingdetectImageMimeType.♻️ Proposed fix to use detectImageMimeType
First, add the import at the top of the file:
import { detectImageMimeType } from '@tanstack/ai'Then update the data URI construction:
case 'image': { const meta = part.metadata as OpenRouterImageMetadata | undefined // For base64 data, construct a data URI if not already one const imageValue = part.source.value + const detectedMimeType = detectImageMimeType(imageValue) ?? 'image/jpeg' const imageUrl = part.source.type === 'data' && !imageValue.startsWith('data:') - ? `data:image/jpeg;base64,${imageValue}` + ? `data:${detectedMimeType};base64,${imageValue}` : imageValueexamples/ts-react-chat/src/routes/index.tsx (2)
24-29: Use the exportedgenerateMessageIdfrom@tanstack/aiinstead of duplicating.This function duplicates
generateMessageIdwhich is already exported from@tanstack/ai(visible in the re-exports atpackages/typescript/ai-client/src/index.tsline 41). Consider importing and using the shared implementation for consistency.♻️ Suggested change
-/** - * Generate a random message ID - */ -function generateMessageId(): string { - return `msg-${Date.now()}-${Math.random().toString(36).substring(2, 9)}` -} +import { generateMessageId } from '@tanstack/ai-react'Remove the local function and add
generateMessageIdto the existing imports from@tanstack/ai-react.
347-354: Redundant metadata fields: bothmediaTypeandmimeTypeare set to the same value.Consider using a single field name for consistency. Based on the type definitions,
metadatais provider-specific, but having both fields with the same value adds no benefit.♻️ Suggested simplification
contentParts.push({ type: 'image', source: { type: 'data', value: img.base64 }, - metadata: { mediaType: img.mimeType, mimeType: img.mimeType }, + metadata: { mimeType: img.mimeType }, })docs/guides/multimodal-content.md (1)
370-377: Model override example doesn't demonstrate the override.The example shows
model: 'gpt-5'in both the base body and the per-message override, which doesn't clearly demonstrate the override behavior.📝 Suggested improvement
const client = new ChatClient({ connection: fetchServerSentEvents('/api/chat'), body: { model: 'gpt-5' }, // Base body params }) // Override model for this specific message await client.sendMessage('Analyze this complex problem', { - model: 'gpt-5', + model: 'gpt-5-turbo', // Overrides base model for this request temperature: 0.2, })packages/typescript/smoke-tests/adapters/src/tests/mms-multimodal-structured.ts (2)
10-25: Extract duplicatedgetMimeTypeto a shared utility.This function is duplicated verbatim in
mmi-multimodal-image.ts. Consider extracting it to a shared module (e.g.,test-utils.ts) to follow DRY principles.♻️ Proposed refactor
Create a new file
packages/typescript/smoke-tests/adapters/src/tests/utils.ts:/** * Detect image mime type from file extension */ export function getMimeType(filename: string): string { const ext = filename.toLowerCase().split('.').pop() switch (ext) { case 'jpg': case 'jpeg': return 'image/jpeg' case 'png': return 'image/png' case 'gif': return 'image/gif' case 'webp': return 'image/webp' default: return 'image/jpeg' } }Then import from both test files:
-function getMimeType(filename: string): string { - // ... implementation -} +import { getMimeType } from './utils'
99-118: Consider extracting shared JSON validation logic.The JSON parsing and validation logic in
runMMS(lines 99-118) andrunMMT(lines 201-220) are nearly identical. For better maintainability, consider extracting a helper function.♻️ Proposed helper extraction
function parseAndValidateImageDescription( response: string ): { parsed: ImageDescription | null error?: string validationMeta: Record<string, boolean> } { let parsed: ImageDescription | null = null try { const jsonMatch = response.match(/```(?:json)?\s*([\s\S]*?)```/) const jsonStr = jsonMatch && jsonMatch[1] ? jsonMatch[1].trim() : response.trim() parsed = JSON.parse(jsonStr) } catch { try { parsed = JSON.parse(response) } catch { return { parsed: null, error: `Failed to parse response as JSON: ${response.substring(0, 200)}`, validationMeta: {} } } } const hasDescription = typeof parsed?.description === 'string' && parsed.description.length > 0 const hasMainSubject = typeof parsed?.mainSubject === 'string' && parsed.mainSubject.length > 0 const hasColors = Array.isArray(parsed?.colors) && parsed.colors.length > 0 const hasTextBoolean = typeof parsed?.hasText === 'boolean' return { parsed, validationMeta: { hasDescription, hasMainSubject, hasColors, hasTextBoolean } } }Also applies to: 201-220
| const handleFileUpload = async (file: File) => { | ||
| // Convert file to base64 | ||
| const base64 = await new Promise<string>((resolve) => { | ||
| const reader = new FileReader() | ||
| reader.onload = () => { | ||
| const result = reader.result as string | ||
| // Remove data URL prefix (e.g., "data:image/png;base64,") | ||
| resolve(result.split(',')[1]) | ||
| } | ||
| reader.readAsDataURL(file) | ||
| }) |
There was a problem hiding this comment.
File reading promise missing error handling.
The FileReader promise doesn't handle the onerror event. If file reading fails, the promise will never resolve, causing the function to hang.
📝 Suggested improvement
// Convert file to base64
- const base64 = await new Promise<string>((resolve) => {
+ const base64 = await new Promise<string>((resolve, reject) => {
const reader = new FileReader()
reader.onload = () => {
const result = reader.result as string
// Remove data URL prefix (e.g., "data:image/png;base64,")
resolve(result.split(',')[1])
}
+ reader.onerror = () => reject(new Error('Failed to read file'))
reader.readAsDataURL(file)
})📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const handleFileUpload = async (file: File) => { | |
| // Convert file to base64 | |
| const base64 = await new Promise<string>((resolve) => { | |
| const reader = new FileReader() | |
| reader.onload = () => { | |
| const result = reader.result as string | |
| // Remove data URL prefix (e.g., "data:image/png;base64,") | |
| resolve(result.split(',')[1]) | |
| } | |
| reader.readAsDataURL(file) | |
| }) | |
| const handleFileUpload = async (file: File) => { | |
| // Convert file to base64 | |
| const base64 = await new Promise<string>((resolve, reject) => { | |
| const reader = new FileReader() | |
| reader.onload = () => { | |
| const result = reader.result as string | |
| // Remove data URL prefix (e.g., "data:image/png;base64,") | |
| resolve(result.split(',')[1]) | |
| } | |
| reader.onerror = () => reject(new Error('Failed to read file')) | |
| reader.readAsDataURL(file) | |
| }) |
🤖 Prompt for AI Agents
In `@docs/guides/multimodal-content.md` around lines 432 - 442, The FileReader
promise in handleFileUpload lacks error handling and can hang if reading fails;
update the promise used in handleFileUpload to attach reader.onerror and reject
the promise with the error (and optionally reader.onabort) so callers receive an
error instead of waiting forever, and ensure any cleanup (e.g., removing
handlers) happens on both success and error paths.
| const base64 = await new Promise<string>((resolve) => { | ||
| const reader = new FileReader() | ||
| reader.onload = () => { | ||
| const result = reader.result as string | ||
| // Remove data URL prefix (e.g., "data:image/png;base64,") | ||
| resolve(result.split(',')[1]) | ||
| } | ||
| reader.readAsDataURL(file) | ||
| }) |
There was a problem hiding this comment.
Add error handling for FileReader.
The FileReader promise doesn't handle the onerror case, which could cause the promise to hang indefinitely if the file read fails.
🛡️ Proposed fix to add error handling
const base64 = await new Promise<string>((resolve, reject) => {
const reader = new FileReader()
reader.onload = () => {
const result = reader.result as string
// Remove data URL prefix (e.g., "data:image/png;base64,")
resolve(result.split(',')[1])
}
+ reader.onerror = () => reject(reader.error)
reader.readAsDataURL(file)
})📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const base64 = await new Promise<string>((resolve) => { | |
| const reader = new FileReader() | |
| reader.onload = () => { | |
| const result = reader.result as string | |
| // Remove data URL prefix (e.g., "data:image/png;base64,") | |
| resolve(result.split(',')[1]) | |
| } | |
| reader.readAsDataURL(file) | |
| }) | |
| const base64 = await new Promise<string>((resolve, reject) => { | |
| const reader = new FileReader() | |
| reader.onload = () => { | |
| const result = reader.result as string | |
| // Remove data URL prefix (e.g., "data:image/png;base64,") | |
| resolve(result.split(',')[1]) | |
| } | |
| reader.onerror = () => reject(reader.error) | |
| reader.readAsDataURL(file) | |
| }) |
🤖 Prompt for AI Agents
In `@examples/ts-react-chat/src/routes/index.tsx` around lines 292 - 300, The
FileReader promise for producing `base64` lacks error handling and can hang on
read failures; change the constructor to new Promise<string>((resolve, reject)
=> { ... }) and add `reader.onerror = (e) => reject(e)` (and optionally
`reader.onabort = () => reject(new Error('File read aborted'))`) alongside the
existing `reader.onload` handler; also consider removing/clearing handlers after
resolution/rejection to avoid leaks and keep using `reader.readAsDataURL(file)`
to start the read.
| // Clean up image previews | ||
| attachedImages.forEach((img) => URL.revokeObjectURL(img.preview)) | ||
| setAttachedImages([]) |
There was a problem hiding this comment.
Fix the forEach callback to not return a value.
The static analysis tool flagged this: the callback passed to forEach() should not return a value. URL.revokeObjectURL returns undefined, but using it in an arrow function expression body implies a return.
🐛 Proposed fix
// Clean up image previews
- attachedImages.forEach((img) => URL.revokeObjectURL(img.preview))
+ for (const img of attachedImages) {
+ URL.revokeObjectURL(img.preview)
+ }
setAttachedImages([])📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Clean up image previews | |
| attachedImages.forEach((img) => URL.revokeObjectURL(img.preview)) | |
| setAttachedImages([]) | |
| // Clean up image previews | |
| for (const img of attachedImages) { | |
| URL.revokeObjectURL(img.preview) | |
| } | |
| setAttachedImages([]) |
🧰 Tools
🪛 Biome (2.3.13)
[error] 363-363: This callback passed to forEach() iterable method should not return a value.
Either remove this return or remove the returned value.
(lint/suspicious/useIterableCallbackReturn)
🤖 Prompt for AI Agents
In `@examples/ts-react-chat/src/routes/index.tsx` around lines 362 - 364, The
forEach callback currently uses a concise arrow body that implicitly returns the
value of URL.revokeObjectURL, which static analysis flags; update the callback
to a block body that does not return anything — e.g., change the call on
attachedImages (attachedImages.forEach((img) =>
URL.revokeObjectURL(img.preview))) to use a statement body like
attachedImages.forEach((img) => { URL.revokeObjectURL(img.preview); }); to
ensure no value is returned from the forEach callback.
| { | ||
| id: 'MMJ', | ||
| name: 'Multimodal JPEG', | ||
| description: | ||
| 'Describe a JPEG image (meme with man, React icon, code/email text)', | ||
| run: runMMJ, | ||
| requires: ['text'], | ||
| }, | ||
| { | ||
| id: 'MMP', | ||
| name: 'Multimodal PNG', | ||
| description: 'Describe a PNG image (beach scene with AG UI text)', | ||
| run: runMMP, | ||
| requires: ['text'], | ||
| }, | ||
| { | ||
| id: 'MMS', | ||
| name: 'Multimodal Structured JPEG', | ||
| description: 'Describe a JPEG image with structured JSON output', | ||
| run: runMMS, | ||
| requires: ['text'], | ||
| }, | ||
| { | ||
| id: 'MMT', | ||
| name: 'Multimodal Structured PNG', | ||
| description: 'Describe a PNG image with structured JSON output', | ||
| run: runMMT, | ||
| requires: ['text'], | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's understand the file structure and locate the test file
git ls-files packages/typescript/smoke-tests/adapters/ | head -20Repository: TanStack/ai
Length of output: 1368
🏁 Script executed:
# Find AdapterCapability definition
rg -n "type AdapterCapability|enum AdapterCapability|AdapterCapability\s*=" --type ts -g '**/*.ts'Repository: TanStack/ai
Length of output: 149
🏁 Script executed:
# Check the full test file to see all test definitions and how capabilities are used
wc -l packages/typescript/smoke-tests/adapters/src/tests/index.tsRepository: TanStack/ai
Length of output: 118
🏁 Script executed:
# Search for vision/multimodal capability or references
rg -n "vision|multimodal" --type ts -g '**/adapters/**'Repository: TanStack/ai
Length of output: 2253
🏁 Script executed:
# Check IMG, TTS, TRN tests mentioned in the review to see skipByDefault pattern
rg -n "'IMG'|'TTS'|'TRN'" --type ts -g '**/tests/**'Repository: TanStack/ai
Length of output: 291
🏁 Script executed:
# Look at how ContentPart is used in adapters
rg -n "ContentPart" --type ts -g '**/adapters/**'Repository: TanStack/ai
Length of output: 4726
🏁 Script executed:
# Read the AdapterCapability definition and context
sed -n '20,50p' packages/typescript/smoke-tests/adapters/src/tests/index.tsRepository: TanStack/ai
Length of output: 785
🏁 Script executed:
# Read the test definitions including IMG, TTS, TRN to see skipByDefault pattern
sed -n '100,160p' packages/typescript/smoke-tests/adapters/src/tests/index.tsRepository: TanStack/ai
Length of output: 1512
🏁 Script executed:
# Check if there's a skipByDefault property mentioned anywhere
rg -n "skipByDefault" --type ts -g '**/adapters/**'Repository: TanStack/ai
Length of output: 883
🏁 Script executed:
# Read the mmi-multimodal-image.ts test implementation to see what happens
cat packages/typescript/smoke-tests/adapters/src/tests/mmi-multimodal-image.tsRepository: TanStack/ai
Length of output: 5918
🏁 Script executed:
# Check how adapters handle image content that they may not support
# Look for error handling in adapter implementations
rg -n "type.*image|image.*support|unsupported.*image" --type ts -g '**/ai-*/src/adapters/**' -A 2Repository: TanStack/ai
Length of output: 2916
🏁 Script executed:
# Check the harness to understand test execution and error handling
cat packages/typescript/smoke-tests/adapters/src/harness.tsRepository: TanStack/ai
Length of output: 11441
Add 'vision' capability to AdapterCapability enum and update multimodal tests accordingly.
Multimodal tests (MMJ, MMP, MMS, MMT) declare requires: ['text'] but actually send image content that requires vision support. The AdapterCapability enum lacks a 'vision' or 'multimodal' option to properly declare this dependency. This mismatch means these tests will attempt to run on all text adapters, failing on those without vision support (e.g., text-only models).
Recommended approach: Add 'vision' to the AdapterCapability enum and update these four tests to requires: ['text', 'vision']. Alternatively, follow the IMG/TTS/TRN pattern by adding skipByDefault: true to reduce noise from unsupported adapters.
🤖 Prompt for AI Agents
In `@packages/typescript/smoke-tests/adapters/src/tests/index.ts` around lines 129
- 157, The multimodal tests (MMJ, MMP, MMS, MMT) declare requires: ['text'] but
send image content; update the test metadata and capability enum by adding a new
AdapterCapability value (e.g., 'vision') to the AdapterCapability enum, then
change the four tests (identifiable by id: 'MMJ','MMP','MMS','MMT' in the tests
array) to requires: ['text','vision']; alternatively, if you prefer the existing
IMG/TTS/TRN pattern, set skipByDefault: true on those test objects instead of
changing requires — ensure references to AdapterCapability and the test objects
are updated consistently.
|
View your CI Pipeline Execution ↗ for commit 4e400f5
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-anthropic
@tanstack/ai-client
@tanstack/ai-devtools-core
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@docs/guides/multimodal-content.md`:
- Around line 464-473: The example passed to sendMessage places mimeType inside
the metadata object incorrectly; update the content payload so that the file
item's mimeType is moved into the source object (i.e., for the array element
with keys type and source, add mimeType under source rather than metadata) to
match the type definitions and other examples; locate the sendMessage call and
adjust the file element structure (the object with fields type, source,
metadata) so source includes mimeType and metadata is either removed or left for
other metadata only.
🧹 Nitpick comments (4)
packages/typescript/ai-react/tests/use-chat.test.ts (1)
1313-1345: Consider adding audio URL test for completeness.The tests cover audio with data source but not with URL source. For consistency with image, video, and document tests (which cover both URL and data sources), consider adding a test for audio URLs.
📝 Suggested test case
it('should send a multimodal message with audio URL', async () => { const chunks = createTextChunks('The audio says hello') const adapter = createMockConnectionAdapter({ chunks }) const { result } = renderUseChat({ connection: adapter }) await result.current.sendMessage({ content: [ { type: 'text', content: 'Transcribe this audio' }, { type: 'audio', source: { type: 'url', value: 'https://example.com/audio.mp3' }, }, ], }) await waitFor(() => { expect(result.current.messages.length).toBeGreaterThan(0) }) const userMessage = result.current.messages.find((m) => m.role === 'user') expect(userMessage?.parts[1]).toEqual({ type: 'audio', source: { type: 'url', value: 'https://example.com/audio.mp3' }, }) })examples/ts-react-chat/src/routes/index.tsx (1)
24-29: Consider using the exportedgenerateMessageIdfrom@tanstack/aiinstead of duplicating it.This local implementation duplicates the utility already exported from
packages/typescript/ai/src/activities/chat/messages.ts. Using the exported version would reduce code duplication and ensure consistency across the codebase.Note: There's a subtle difference - the exported version uses
substring(7)while this usessubstring(2, 9). If the 7-character output is intentional for consistency, import from@tanstack/ai.♻️ Suggested change
-/** - * Generate a random message ID - */ -function generateMessageId(): string { - return `msg-${Date.now()}-${Math.random().toString(36).substring(2, 9)}` -} +import { generateMessageId } from '@tanstack/ai'packages/typescript/smoke-tests/adapters/src/tests/mms-multimodal-structured.ts (2)
30-45: RenameSTRUCTURED_PROMPTto camelCase for consistency.
This keeps variable naming aligned with the codebase convention.♻️ Proposed rename
-const STRUCTURED_PROMPT = `Analyze this image and provide a structured description. Return ONLY valid JSON (no markdown code blocks) matching this schema: +const structuredPrompt = `Analyze this image and provide a structured description. Return ONLY valid JSON (no markdown code blocks) matching this schema: { "description": "A brief description of what the image shows", "hasText": true/false, "textContent": "The text content visible in the image, if any", "mainSubject": "The main subject or focal point of the image", "colors": ["array", "of", "primary", "colors"] }` @@ - content: STRUCTURED_PROMPT, + content: structuredPrompt, @@ - content: STRUCTURED_PROMPT, + content: structuredPrompt,As per coding guidelines:
**/*.{ts,tsx,js,jsx}: Use camelCase for function and variable names throughout the codebase.
55-147: Consider extracting shared validation/payload logic to reduce duplication withrunMMT.
Both runners repeat the same fixture loading, content construction, and JSON validation; a shared helper would make future changes safer and smaller.
| await sendMessage({ | ||
| content: [ | ||
| { type: 'text', content: `Please analyze this ${type}` }, | ||
| { | ||
| type, | ||
| source: { type: 'data', value: base64 }, | ||
| metadata: { mimeType: file.type } | ||
| } | ||
| ] | ||
| }) |
There was a problem hiding this comment.
Incorrect mimeType placement in file upload example.
The example places mimeType in metadata, but according to the type definitions and all other examples in this document, mimeType should be in the source object for data sources.
📝 Proposed fix
await sendMessage({
content: [
{ type: 'text', content: `Please analyze this ${type}` },
{
type,
- source: { type: 'data', value: base64 },
- metadata: { mimeType: file.type }
+ source: { type: 'data', value: base64, mimeType: file.type }
}
]
})🤖 Prompt for AI Agents
In `@docs/guides/multimodal-content.md` around lines 464 - 473, The example passed
to sendMessage places mimeType inside the metadata object incorrectly; update
the content payload so that the file item's mimeType is moved into the source
object (i.e., for the array element with keys type and source, add mimeType
under source rather than metadata) to match the type definitions and other
examples; locate the sendMessage call and adjust the file element structure (the
object with fields type, source, metadata) so source includes mimeType and
metadata is either removed or left for other metadata only.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@packages/typescript/ai-client/src/chat-client.ts`:
- Around line 423-429: The mergedBody construction in sendMessage (where
mergedBody = { ...this.body, ...this.pendingMessageBody, conversationId:
this.uniqueId }) silently overrides any caller-supplied conversationId from the
per-message body; update sendMessage's JSDoc near the function (and/or add a
short warning log in sendMessage) to explicitly state that conversationId is
forced to this.uniqueId and will override per-message values, or alternatively
change the merge order to preserve a caller-supplied conversationId if that
behavior is desired—refer to mergedBody, this.body, this.pendingMessageBody, and
this.uniqueId when making the change.
- Around line 334-337: The empty-content guard currently only checks string
inputs via the emptyMessage variable and lets MultimodalContent values with
empty payloads through; update the check to detect MultimodalContent (type or
interface name MultimodalContent) and treat it as empty when its inner content
is an empty string or an empty array. Concretely, inside the method that defines
emptyMessage (referencing emptyMessage and this.isLoading) add a branch that
detects MultimodalContent (or use a small helper isMultimodalContent) and
returns true when its .content is "" after trim or when .content is an array
with length === 0, then keep the existing early return if (emptyMessage ||
this.isLoading).
🧹 Nitpick comments (1)
packages/typescript/ai-client/src/chat-client.ts (1)
29-29: Consider passingbodyas a parameter tostreamResponseinstead of using a class field.
pendingMessageBodyintroduces temporal coupling —streamResponsedepends on this field being set by its caller. Since onlysendMessageneeds per-message body, an optional parameter onstreamResponsewould be more explicit and eliminate the need for set-then-clear bookkeeping:private async streamResponse(perMessageBody?: Record<string, any>): Promise<void> {This is a minor structural improvement and entirely optional.
Also applies to: 402-437
| const emptyMessage = typeof content === 'string' && !content.trim() | ||
| if (emptyMessage || this.isLoading) { | ||
| return | ||
| } |
There was a problem hiding this comment.
Empty-content guard doesn't cover MultimodalContent inputs.
A MultimodalContent with content: '' or content: [] bypasses the guard and sends a contentless user message. Consider extending the check:
Proposed fix
- const emptyMessage = typeof content === 'string' && !content.trim()
- if (emptyMessage || this.isLoading) {
+ const emptyMessage =
+ typeof content === 'string'
+ ? !content.trim()
+ : typeof content.content === 'string'
+ ? !content.content.trim()
+ : content.content.length === 0
+ if (emptyMessage || this.isLoading) {
return
}🤖 Prompt for AI Agents
In `@packages/typescript/ai-client/src/chat-client.ts` around lines 334 - 337, The
empty-content guard currently only checks string inputs via the emptyMessage
variable and lets MultimodalContent values with empty payloads through; update
the check to detect MultimodalContent (type or interface name MultimodalContent)
and treat it as empty when its inner content is an empty string or an empty
array. Concretely, inside the method that defines emptyMessage (referencing
emptyMessage and this.isLoading) add a branch that detects MultimodalContent (or
use a small helper isMultimodalContent) and returns true when its .content is ""
after trim or when .content is an array with length === 0, then keep the
existing early return if (emptyMessage || this.isLoading).
| // Merge body: base body + per-message body (per-message takes priority) | ||
| // Include conversationId for server-side event correlation | ||
| const mergedBody = { | ||
| ...this.body, | ||
| ...this.pendingMessageBody, | ||
| conversationId: this.uniqueId, | ||
| } |
There was a problem hiding this comment.
conversationId silently overrides per-message body value.
The conversationId property is assigned after spreading pendingMessageBody, so a caller-supplied conversationId in the body parameter will be silently discarded. This is likely intentional, but the JSDoc on sendMessage (line 299) says "per-message body taking priority" without noting this exception. A brief doc note or a warning log would avoid confusion.
🤖 Prompt for AI Agents
In `@packages/typescript/ai-client/src/chat-client.ts` around lines 423 - 429, The
mergedBody construction in sendMessage (where mergedBody = { ...this.body,
...this.pendingMessageBody, conversationId: this.uniqueId }) silently overrides
any caller-supplied conversationId from the per-message body; update
sendMessage's JSDoc near the function (and/or add a short warning log in
sendMessage) to explicitly state that conversationId is forced to this.uniqueId
and will override per-message values, or alternatively change the merge order to
preserve a caller-supplied conversationId if that behavior is desired—refer to
mergedBody, this.body, this.pendingMessageBody, and this.uniqueId when making
the change.
2 participants
🎯 Changes
Added the ability to send multi-modal messages from the client
Added the ability to send extra body data with sendMessage api
added ability to add messageIds from the client
Added new e2e tests to the smoke tests harness
✅ Checklist
pnpm run test:pr.🚀 Release Impact
Summary by CodeRabbit
New Features
Documentation
Tests