<aside>

Multilingual AI Systems

<aside>

Definition:

AI systems that can process and generate multiple human languages — useful in translation, localization, multilingual chatbots, global search engines, etc.

</aside>

<aside>

Capabilities:

• Text understanding across languages (e.g., sentiment analysis in Hindi, summarization in German).
• Zero-shot or few-shot translation (e.g., translating between languages it wasn’t explicitly trained on).
• Code-switching (handling mixed languages in a single input). </aside>

<aside>

Techniques:

• Shared Embedding Spaces

  Words from different languages are mapped into a common vector space (e.g., multilingual BERT, XLM-R).

  ---

• Language Tags in Prompts

  Used in multilingual LLMs to set context.

  Example: Translate to Spanish: [input]

  ---

• Transfer Learning Across Languages

  Models fine-tuned on high-resource languages generalize to low-resource ones (via shared syntax/semantics).

  ---

• Mixture of Experts (MoE)

  Activates specific "experts" per language to scale efficiently.

  ---

• Tokenizer Innovations

  Unicode-based or sentence piece tokenizers allow support for multiple scripts and writing systems.

</aside>

<aside>

Examples of Multilingual Models:

• mBERT (Multilingual BERT)
• XLM-R (Facebook AI)
• M2M-100 (Facebook)
• NLLB-200 (No Language Left Behind)
• Gemini & GPT-4 (supports 20–50+ languages well) </aside>

</aside>

<aside>

Multimodal AI Systems

<aside>

Definition:

AI systems that understand and process more than one data modality — for example, combining text + image + audio + video + code.

</aside>

<aside>

Capabilities:

• Visual question answering (VQA)
• Image captioning / text-to-image
• Video understanding / summarization
• Audio transcription / generation
• Cross-modal search (e.g., find images that match a description)
• Multi-modal reasoning (e.g., charts + narrative explanation) </aside>

<aside>

Techniques:

1. Cross-Attention Fusion

Aligns and fuses information across modalities (e.g., CLIP aligns image and text embeddings).

2. Vision-Language Models (VLMs)

Models trained on image-text pairs.

Examples:

• CLIP (OpenAI): learns joint vision-language representations.
• BLIP-2: image-to-text generation.
• GPT-4V (Vision): accepts images as input, supports complex reasoning.

3. Multimodal Transformers

Unified architecture that handles various inputs with modality-specific embeddings.

4. Contrastive Learning

Used in CLIP, Flamingo, and similar models to align image and text representations.

5. Multimodal Prompting

Combine text + image or other data into prompts.

Example:

"Here’s a photo of a traffic light. What does the sign say?" → with image and text input.

</aside>

</aside>

<aside>

Examples of Multimodal Models:

<aside>

Multilingual + Multimodal Systems

These systems:

• Handle visual questions in multiple languages.
• Perform multilingual OCR and audio transcription.
• Enable multilingual image captioning or text-to-image in non-English prompts.

Example Use Case:

Upload a Japanese poster → AI summarizes it in English, identifies images, reads text, suggests cultural context.

Examples:

• GPT-4V with multilingual support.
• Gemini 1.5 with language + video input.
• NLLB + CLIP combo for multilingual multimodal search. </aside>

<aside>

Challenges

• Alignment across modalities and languages
• Bias and underperformance in low-resource languages
• Tokenization for multilingual multimodal inputs
• Latency and memory usage </aside>

<aside>

</aside>