Extracción de metadatos de documentos con Llama

Copyright (c) Meta Platforms, Inc. y afiliados. Este software puede usarse y distribuirse según los términos del Acuerdo de licencia de la comunidad de Llama.

Este tutorial te muestra cómo construir un sistema de procesamiento de facturas que extrae automáticamente datos estructurados de imágenes de facturas. Usando un pipeline de dos etapas con los modelos multimodales y de texto de Llama, transformarás diversos formatos de factura en datos JSON limpios y validados, listos para una importación fluida a los sistemas de contabilidad.

Mientras que las herramientas de OCR tradicionales tienen dificultades con diseños diversos y requieren una configuración extensa de plantillas, el enfoque aquí utiliza las capacidades de visión de Llama para comprender cualquier formato de factura, enriquecer los datos con servicios externos y marcar excepciones para revisión humana, lo que ofrece la precisión requerida para la automatización financiera.

Qué aprenderás

• Construir un pipeline de procesamiento de dos etapas que separa la extracción visual del refinamiento inteligente para una precisión y eficiencia de costos óptimas.
• Aprovechar las capacidades multimodales de Llama para extraer datos de imágenes de facturas con diversos diseños y formatos.
• Usar la llamada a herramientas de la API de Llama para enriquecer datos con servicios externos como las API de conversión de moneda.
• Implementar la salida estructurada JSON para garantizar una extracción de datos consistente y confiable en todo momento.

Nota sobre los proveedores de inferencia: Este tutorial utiliza la API de Llama con fines de demostración. Sin embargo, puedes ejecutar modelos de Llama con cualquier proveedor de inferencia preferido. Ejemplos comunes incluyen Amazon Bedrock y Together AI. La lógica central de este tutorial se puede adaptar a cualquiera de estos proveedores.

Instantánea del escenario

• Corpus: El conjunto de datos SROIE (Scanned Receipts OCR and Information Extraction), disponible públicamente en Kaggle. El conjunto de datos contiene recibos y facturas del mundo real con diversos diseños, fuentes y formatos que simulan de manera realista los desafíos que enfrenta un departamento de Cuentas por Pagar. Los documentos van desde recibos simples hasta facturas complejas con calidad variable, desde exportaciones digitales impecables hasta escaneos descoloridos.

Ejemplo de factura del conjunto de datos SROIE:

Ejemplo de factura de Malasia que muestra desafíos de extracción típicos: moneda extranjera (RM), formatos de fecha mixtos (29/01/2018), información del proveedor y calidad de texto variable.

• Usuarios: Los usuarios objetivo son especialistas en Cuentas por Pagar (AP) y empleados de contabilidad cuya responsabilidad principal es procesar las facturas entrantes para el pago de manera precisa y eficiente. Por lo general, dedican de 3 a 5 minutos por factura a la entrada manual de datos, lo que provoca cuellos de botella en el procesamiento y errores humanos.

• Solicitudes típicas: La tarea principal es extraer datos estructurados de cada documento de factura. Una tarea de extracción típica sería:

  • Extraer lo siguiente de la factura: vendor_name, invoice_date, vendor_address, total_amount y currency_symbol
  • Convertir cualquier monto en moneda extranjera a USD utilizando las tasas de cambio actuales
  • Marcar cualquier factura con campos obligatorios faltantes o errores de validación para revisión manual

Solución: Pipeline de procesamiento inteligente de dos etapas

La solución es un pipeline de dos etapas que optimiza tanto la precisión como el costo utilizando diferentes modelos de Llama especializados para la tarea de cada etapa:

flowchart TD
    A[Invoice Image] --> B[Stage 1: Accurate Transcription]
    B -- Raw JSON with Ambiguities --> C[Stage 2: Intelligent Refinement]
    C -- Enriched & Validated Data --> E[Final Structured JSON]
    
    subgraph "Multimodal Llama Model"
        B
    end
    
    subgraph "Text Refinement"
        C
    end
    
    D[External Tools, e.g., Currency API] <--> C

Etapa 1: Transcripción precisa (visión a datos brutos)

La primera etapa actúa como los "ojos" del sistema, centrándose por completo en convertir la información visual en texto estructurado con la máxima precisión.

Estrategia clave: Se le indica al modelo que extraiga, no que invente. Captura cualquier ambigüedad visual en el campo extraction_notes (por ejemplo, "El símbolo de la moneda parece ser 'RM' pero podría ser 'SR' debido a la calidad de impresión"), creando una representación digital bruta pero fiel con incertidumbres documentadas.

Etapa 2: Refinamiento inteligente (datos a información)

La segunda etapa actúa como el "cerebro" del sistema, aplicando lógica de negocio, conocimiento externo y reglas de validación para producir datos limpios y confiables.

La separación de preocupaciones te permite usar modelos multimodales costosos solo para tareas visuales, mientras aprovechas modelos de texto más rápidos y económicos para el refinamiento de datos, lo que reduce los costos hasta en un 60 % en comparación con el uso de modelos multimodales en todo el proceso.

Requisitos previos

Antes de comenzar, asegúrate de tener una clave de API de Llama de Llama API.

Instalar dependencias

Necesitarás algunas bibliotecas para este proyecto: llama-api-client para el acceso a la API y pillow para el manejo de imágenes.

# Install dependencies
!pip install --quiet llama-api-client pillow

Importaciones y configuración del cliente de la API de Llama

Importa los módulos necesarios e inicializa el LlamaAPIClient usando tu clave de API como una variable de entorno.

Nota: Este tutorial utiliza la API de Llama, pero puedes adaptarlo a otros proveedores de inferencia como Amazon Bedrock y Together AI.

import os, sys, json
import base64
import textwrap
from pathlib import Path
from typing import Dict, List, Optional, Any
from datetime import datetime
from io import BytesIO
from PIL import Image
from pydantic import BaseModel, Field
from llama_api_client import LlamaAPIClient

# --- Llama client ---
API_KEY = os.getenv("LLAMA_API_KEY")
if not API_KEY:
    sys.exit("❌ Please set the LLAMA_API_KEY environment variable.")

client = LlamaAPIClient(api_key=API_KEY)

Configuración del modelo

El tutorial utiliza dos modelos Llama especializados, optimizados para diferentes tareas: Llama 4 Maverick para el procesamiento de entrada multimodal (texto + imágenes), mientras que Llama 4 Scout para un rendimiento rápido de texto con capacidades de llamada a herramientas.

# --- Constants & Configuration ---
STAGE1_MODEL = "Llama-4-Maverick-17B-128E-Instruct-FP8"
STAGE2_MODEL = "Llama-4-Scout-17B-16E-Instruct-FP8"
MAX_COMPLETION_TOKENS = 2000  # Max tokens for model completion

# Setting a token limit is a best practice for controlling costs and ensuring 
# predictable performance. This value acts as a safeguard, preventing runaway 
# requests while being high enough to handle complex invoices.

Cargar datos de facturas de ejemplo

El tutorial utiliza un pequeño subconjunto del conjunto de datos SROIE (Scanned Receipts OCR and Information Extraction), que contiene recibos y facturas del mundo real con diversos diseños.

El conjunto de datos SROIE completo contiene miles de facturas con calidad variable, desde exportaciones digitales impecables hasta escaneos descoloridos.

# Path to SROIE dataset samples
DATA_DIR = Path("data")
INVOICE_IMG_DIR = DATA_DIR / "invoice_img"
INVOICE_JSON_DIR = DATA_DIR / "invoice_json"

def load_sroie_data():
    """Load SROIE dataset invoice images and ground truth."""
    invoice_images = sorted(INVOICE_IMG_DIR.glob("*.jpg"))
    
    invoices = []
    for img_path in invoice_images:  # Load all invoice images
        json_path = INVOICE_JSON_DIR / f"{img_path.stem}.txt"
        
        ground_truth = {}
        if json_path.exists():
            with open(json_path, 'r') as f:
                ground_truth = json.loads(f.read())
        
        invoices.append({
            "image_path": str(img_path),
            "filename": img_path.name,
            "ground_truth": ground_truth
        })
    
    print(f"✅ {len(invoices)} invoices loaded")
    return invoices

def load_invoice_image(image_path: str) -> str:
    """Load and encode invoice image for API calls."""
    with open(image_path, "rb") as img_file:
        return base64.b64encode(img_file.read()).decode('utf-8')

# Load SROIE dataset
sroie_invoices = load_sroie_data()

✅ 10 invoices loaded

Etapa 1 - Extracción visual con modelo multimodal

La etapa 1 utiliza las capacidades multimodales de Llama 4 Maverick para una extracción de texto precisa de diseños de facturas complejos. La clave es ser fiel a la fuente, capturando exactamente lo que es visible, incluidas las ambigüedades.

Definir el esquema de la factura

Usaremos modelos Pydantic para definir nuestra estructura de salida esperada, asegurando una extracción consistente en todas las facturas.

# --- Pydantic Models for Structured Output ---
class RawInvoiceData(BaseModel):
    """Model for Stage 1 raw extraction output - aligned with SROIE dataset."""
    vendor_name: str = Field(description="Company that issued the invoice")
    invoice_date: str = Field(
        description="Date invoice was issued (preserve original format)")
    vendor_address: str = Field(description="Vendor address as shown on invoice")
    total_amount: str = Field(
        description="Total amount as numeric string only (e.g., '123.45', no currency symbols)")
    currency_symbol: str = Field(
        description="Currency symbol or code found (e.g., 'RM', '$', 'USD')")
    extraction_notes: str = Field(
        description="Any visual ambiguities, unclear text, or alternative interpretations observed")

Implementar la extracción visual

Estrategia de prompt: Instruye al modelo para que actúe como un transcriptor de alta fidelidad, separando los montos numéricos de los símbolos de moneda y documentando las incertidumbres visuales en extraction_notes para la resolución de la Etapa 2.

def stage1_visual_extraction(image_path: str) -> Dict:
    """
    Stage 1: Extract raw data from invoice image using multimodal model.
    Focus: accurate transcription without interpretation.
    """
    
    system_prompt = """Extract invoice information from the image with perfect accuracy.

EXTRACTION RULES:
1. Extract ONLY what you clearly see - use "UNCLEAR" or "NOT_FOUND" if uncertain
2. Preserve original date formats (e.g., "15/01/2019")
3. total_amount: numeric value only (e.g., "123.45") - NO symbols or formatting
4. currency_symbol: symbol/code only (e.g., "RM", "$", "USD")
5. extraction_notes: document visual uncertainties and alternative interpretations

EXAMPLES of extraction_notes:
- "Currency symbol appears to be 'RM' but could be 'SR' due to print quality"
- "Date format ambiguous - could be DD/MM or MM/DD"
- "Vendor name partially obscured"

Output using the provided JSON schema."""

    # Load and encode image
    base64_image = load_invoice_image(image_path)
    
    user_content = [
        {
            "type": "text",
            "text": "Extract all information from this invoice image."
        },
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
        }
    ]
    
    response_format = {
        "type": "json_schema",
        "json_schema": {
            "name": RawInvoiceData.__name__,
            "schema": RawInvoiceData.model_json_schema(),
        },
    }
    
    try:
        response = client.chat.completions.create(
            model=STAGE1_MODEL,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_content}
            ],
            temperature=0.1,
            max_completion_tokens=MAX_COMPLETION_TOKENS,
            response_format=response_format
        )
        result = json.loads(response.completion_message.content.text)
        
        return {'success': True, 'raw_data': result}
        
    except Exception as e:
        return {'success': False, 'error': str(e)}

Ahora procesemos todas las facturas SROIE a través de la Etapa 1:

from typing import Any, Dict, Optional

def normalize_company_name(name: Optional[str]) -> str:
    """Normalize company name for consistent comparison."""
    return (name or "").upper().strip()

def normalize_amount(amount: Any) -> str:
    """Normalize amount for consistent comparison by removing symbols and whitespace."""
    amount_str = str(amount or "")
    return amount_str.replace('$', '').replace('RM', '').strip()

def evaluate_extraction(data: Dict, gt: Dict, amount_field: str = 'total_amount') -> tuple[bool, bool]:
    """Shared evaluation logic for both stages using normalization helpers."""
    company_norm = normalize_company_name(data.get('vendor_name'))
    gt_company_norm = normalize_company_name(gt.get('company'))
    
    amount_str = normalize_amount(data.get(amount_field))
    gt_amount_str = normalize_amount(gt.get('total'))
    
    return company_norm == gt_company_norm, amount_str == gt_amount_str

def print_comparison(i, filename, data, gt, company_match, total_match, 
                     stage_name, amount_field='total_amount', extra_info=None):
    """Shared printing logic for both stages."""
    print(f"[{i}] {filename}")
    print(f"{stage_name}: {data.get('vendor_name')} | {data.get(amount_field)}")
    print(f"Target: {gt.get('company')} | {gt.get('total')}")
    overall_match = company_match and total_match
    print(f"{'✅' if overall_match else '❌'} {'✓' if overall_match else '✗'} "
          f"(Company:{'✓' if company_match else '✗'} | Amount:{'✓' if total_match else '✗'})")
    if extra_info: print(extra_info)
    print()

print("🔍 Processing invoices through Stage 1...")

stage1_results = []
for i, invoice in enumerate(sroie_invoices, 1):
    result = stage1_visual_extraction(invoice['image_path'])
    
    if result['success']:
        raw_data, gt = result['raw_data'], invoice['ground_truth']
        
        # Normalize for comparison
        company_match, total_match = evaluate_extraction(raw_data, gt)
        
        print(f"[{i}] {invoice['filename']}")
        print(f"Extracted: {raw_data.get('vendor_name')} | {raw_data.get('total_amount')}")
        print(f"Target {gt.get('company')} | {gt.get('total')}")
        if raw_data.get('extraction_notes') and raw_data['extraction_notes'].strip():
            print(f"📝 Notes: {raw_data['extraction_notes']}")
        overall_match = company_match and total_match
        print(f"{'✅' if overall_match else '❌'} {'✓' if overall_match else '✗'} "
              f"(Company:{'✓' if company_match else '✗'} | "
            f"Amount:{'✓' if total_match else '✗'})\n")
        
        stage1_results.append({
            'invoice': invoice, 'result': result,
            'company_match': company_match, 'total_match': total_match
        })
    else:
        print(f"[{i}] {invoice['filename']} ❌ FAILED\n")
        stage1_results.append({
            'invoice': invoice, 'result': result,
            'company_match': False, 'total_match': False
        })

successful = [r for r in stage1_results if r['result']['success']]
company_accuracy = (sum(1 for r in successful if r['company_match']) / 
                    len(successful) if successful else 0)
total_accuracy = (sum(1 for r in successful if r['total_match']) / 
                  len(successful) if successful else 0)

print(f"✅ Stage 1: {len(successful)}/{len(sroie_invoices)} processed | "
      f"Accuracy: {company_accuracy:.1%} company, {total_accuracy:.1%} amount")

🔍 Processing invoices through Stage 1...
[1] X00016469670.jpg
Extracted: OJC MARKETING SDN BHD | 193.00
Target OJC MARKETING SDN BHD | 193.00
📝 Notes: The currency symbol 'SR' is used, which typically represents Saudi Riyal. The invoice is clearly marked as a 'TAX INVOICE' and includes details such as invoice number, date, cashier, sales person, and bill to information. The product details and total amount are also clearly listed.
✅ ✓ (Company:✓ | Amount:✓)

[2] X00016469671.jpg
Extracted: OJC MARKETING SDN BHD | 170.00
Target OJC MARKETING SDN BHD | 170.00
📝 Notes: The currency symbol is not explicitly shown on the invoice, but the amounts are listed with two decimal places, suggesting a currency that uses this format, such as MYR (Malaysian Ringgit). The vendor is based in Malaysia, supporting this interpretation.
✅ ✓ (Company:✓ | Amount:✓)

[3] X51005200931.jpg
Extracted: PERNIAGAAN ZHENG HUI | 436.20
Target PERNIAGAAN ZHENG HUI | 436.20
📝 Notes: The invoice is clear and legible, with all necessary information visible. The date format is DD/MM/YYYY.
✅ ✓ (Company:✓ | Amount:✓)

[4] X51005230605.jpg
Extracted: PETRON BKT LANJAN SB | : 4.90
Target PETRON BKT LANJAN SB | 4.90
📝 Notes: The receipt appears to be from a Petron gas station, and it includes a purchase of food items and GST. The total amount is clearly stated as RM 4.90. The date is in the format DD/MM/YYYY.
❌ ✗ (Company:✓ | Amount:✗)

[5] X51005230616.jpg
Extracted: Gerbang Alaf Restaurants Sdn Bhd (formerly known as Golden Arches Restaurants Sdn Bhd) | 38.90
Target GERBANG ALAF RESTAURANTS SDN BHD | 38.90
📝 Notes: The currency symbol is assumed to be 'RM' as it is the local currency in Malaysia where the invoice is from, but it is not explicitly shown on the invoice.
❌ ✗ (Company:✗ | Amount:✓)

[6] X51005230621.jpg
Extracted: SIN LIANHAP SDN BHD | .$30
Target SIN LIANHAP SDN BHD | 7.30
📝 Notes: The total amount is listed as '7.30' under 'Payment', and the currency symbol is 'RM' as indicated next to the item prices.
❌ ✗ (Company:✓ | Amount:✗)

[7] X51005230648.jpg
Extracted: CROSS CHANNEL NETWORK SDN. BHD. | 6.35
Target CROSS CHANNEL NETWORK SDN. BHD. | 6.35
📝 Notes: The invoice number is BTG-052332. The product purchased is 'SCHNEIDER E15R 13A SWITCH SOCKET OUTLET' with a quantity of 1. The total amount includes GST at 6%. The paid amount was RM 10.00, and the change given was RM 3.65. The GST summary shows SR @ A with an amount of RM 6.00 and tax of RM 0.36.
✅ ✓ (Company:✓ | Amount:✓)

[8] X51005230657.jpg
Extracted: CROSS CHANNEL NETWORK SDN. BHD. | 10.00
Target CROSS CHANNEL NETWORK SDN. BHD. | 7.95
📝 Notes: The invoice is clear and legible. The date is in the format DD/MM/YYYY and includes a timestamp. The total amount is clearly stated as 'Total Amt Payable: 10.00'. The currency symbol 'RM' is used consistently throughout the invoice.
❌ ✗ (Company:✓ | Amount:✗)

[9] X51005230659.jpg
Extracted: SWC ENTERPRISE SDN BHD | $patchy image obscuring total amount
Target SWC ENTERPRISE SDN BHD | 8.00
📝 Notes: The total amount is partially obscured by a patchy image, making it difficult to determine the exact value. The visible amount is '8.00', but it's unclear if this is the total or a subtotal. The currency symbol is not explicitly shown on the invoice.
❌ ✗ (Company:✓ | Amount:✗)

[10] X51005268275.jpg
Extracted: LIGHTROOM GALLERY SDN BHD | 278.80
Target LIGHTROOM GALLERY SDN BHD | 278.80
📝 Notes: The image is a clear receipt from Lightroom Gallery Sdn Bhd, dated 20/11/2017. The total amount is RM 278.80. The receipt includes details of items purchased, GST, and payment information.
✅ ✓ (Company:✓ | Amount:✓)

✅ Stage 1: 10/10 processed | Accuracy: 90.0% company, 60.0% amount

Etapa 2 - Refinamiento inteligente con llamada a herramientas

La etapa 2 utiliza Llama 4 Scout para un rendimiento rápido y llamada a herramientas, aplicando lógica de negocio para resolver ambigüedades y enriquecer datos con servicios externos. Aquí es donde el sistema se vuelve verdaderamente inteligente al resolver ambigüedades y enriquecer datos con información externa.

Definir herramientas para servicios externos

Crearemos herramientas que el modelo puede usar para enriquecer los datos extraídos. Estas herramientas permiten que el sistema realice la conversión de moneda y el enriquecimiento de datos.

Aprende más sobre la llamada a herramientas: Para obtener una guía completa sobre la implementación de la llamada a herramientas con modelos de Llama, consulta la Guía de llamada a herramientas de Meta.

Estrategia de herramientas: En este tutorial usamos la conversión de moneda para demostrar la llamada a herramientas con un propósito claro y un manejo de datos estructurado. Este mismo patrón se extiende a otras herramientas como la validación de proveedores, el cálculo de impuestos, las verificaciones de cumplimiento y otras integraciones de lógica de negocio.

Nota de implementación: La implementación de conversión de moneda a continuación utiliza tasas de cambio estáticas para simplificar el tutorial. En producción, te integrarías con API de moneda en vivo como ExchangeRate-API, Fixer.io o el servicio de moneda de tu sistema financiero.

# --- Tool Definitions ---
def get_currency_conversion_tool():
    """Define the currency conversion tool for Stage 2."""
    return {
        "type": "function",
        "function": {
            "name": "convert_currency",
            "description": "Convert amount from one currency to USD using static exchange rates",
            "parameters": {
                "type": "object",
                "properties": {
                    "amount": {
                        "type": "number",
                        "description": "The amount to convert"
                    },
                    "from_currency": {
                        "type": "string",
                        "description": "Source currency code (MYR, EUR, GBP, SGD)"
                    },
                    "to_currency": {
                        "type": "string",
                        "description": "Target currency code (USD only)"
                    }
                },
                "required": ["amount", "from_currency", "to_currency"]
            }
        }
    }

def convert_currency(amount: float, from_currency: str, to_currency: str = "USD") -> Dict:
    """Convert currency amounts using static exchange rates (tutorial implementation)."""
    
    # Static rates for tutorial (August 2024)
    rates = {"MYR": 0.21, "EUR": 1.09, "GBP": 1.27, "SGD": 0.74}
    
    # Validate inputs
    if not isinstance(amount, (int, float)) or amount < 0:
        return {"error": "Amount must be a non-negative number"}
    
    if from_currency == to_currency:
        return {
            "converted_amount": float(amount), 
            "exchange_rate": 1.0, 
            "note": "No conversion needed"
        }
    
    if from_currency in rates and to_currency == "USD":
        converted = amount * rates[from_currency]
        return {
            "converted_amount": round(converted, 2),
            "exchange_rate": rates[from_currency],
            "source_currency": from_currency,
            "target_currency": to_currency
        }
    
    return {"error": f"Currency conversion from {from_currency} to {to_currency} not supported"}

def execute_tool(tool_name: str, arguments: Dict) -> Dict:
    """Execute the requested tool and return results."""
    
    if tool_name == "convert_currency":
        try:
            return convert_currency(
                amount=arguments["amount"],
                from_currency=arguments["from_currency"], 
                to_currency=arguments["to_currency"]
            )
        except KeyError as e:
            return {"error": f"Missing required argument: {e}"}
        except (TypeError, ValueError) as e:
            return {"error": f"Invalid argument type: {e}"}
    
    return {"error": f"Unknown tool: {tool_name}"}

Definir el esquema de salida enriquecido

El esquema enriquecido estructura la extracción bruta de la Etapa 1 en datos limpios y listos para el negocio con conversión de moneda y transparencia de procesamiento.

class EnrichedInvoiceData(BaseModel):
    """Model for Stage 2 enriched output with currency conversion."""
    vendor_name: str = Field(description="Vendor company name")
    vendor_address: str = Field(description="Vendor address")
    invoice_date: str = Field(
        description="Invoice date in ISO format (YYYY-MM-DD)")
    original_amount: str = Field(
        description="Original amount as numeric string (e.g., '123.45', no symbols)")
    original_currency: str = Field(
        description="Original currency symbol (e.g., 'MYR', 'USD')")
    converted_amount_usd: str = Field(
        description="USD amount as numeric string (e.g., '25.89', no symbols)")
    exchange_rate: str = Field(
        description="Exchange rate as numeric string (e.g., '0.21')")
    reasoning_notes: str = Field(description="AI reasoning summary")

Implementar el refinamiento inteligente

Estrategia de procesamiento: Usamos Llama 4 Scout con salida JSON estructurada y llamada a herramientas para enriquecer los datos brutos de la Etapa 1, resolver ambigüedades de moneda, estandarizar fechas y convertir montos a USD. El modelo analiza extraction_notes para resolver ambigüedades documentadas usando el contexto de negocio, estandariza fechas, sigue reglas estrictas de formato numérico y convierte montos a USD.

def stage2_intelligent_refinement(raw_data: Dict) -> Dict:
    """Stage 2: Refine and enrich raw extracted data using tool calling."""
    
    # Validate input
    if not isinstance(raw_data, dict):
        return {'success': False, 'error': 'Invalid input: raw_data must be a dictionary', 'stage': 2}
    
    required_fields = ['vendor_name', 'total_amount', 'currency_symbol']
    missing_fields = [field for field in required_fields if field not in raw_data]
    if missing_fields:
        return {'success': False, 'error': f'Missing required fields: {missing_fields}', 'stage': 2}
    
    try:
        tools = [get_currency_conversion_tool()]
        response_format = {
            "type": "json_schema",
            "json_schema": {
                "name": EnrichedInvoiceData.__name__,
                "schema": EnrichedInvoiceData.model_json_schema(),
            },
        }
        
        refinement_prompt = f"""Analyze the raw invoice data below and enrich it with currency conversion.

Raw data from Stage 1:
{json.dumps(raw_data, indent=2)}

TASKS:
1. Extract original_amount from total_amount (numeric string only)
2. Resolve currency ambiguities using extraction_notes + context clues
3. Standardize date to YYYY-MM-DD format
4. Use convert_currency tool for non-USD amounts
5. Document reasoning in reasoning_notes

AMBIGUITY RESOLUTION:
- Use extraction_notes to identify uncertainties from Stage 1
- Attempt to resolve uncertainties based on the information you have
- Apply context clues: Malaysian addresses suggest MYR currency
- Reference extraction_notes findings in your reasoning_notes

OUTPUT FORMATTING (numeric strings only):
- original_amount: "123.45" (no symbols)
- converted_amount_usd: "25.89" (no symbols)
- exchange_rate: "0.21" (no symbols)
- original_currency: "MYR" (code only, not "RM")
"""

        messages = [{"role": "user", "content": refinement_prompt}]
        
        response = client.chat.completions.create(
            model=STAGE2_MODEL,
            messages=messages,
            tools=tools,
            temperature=0.3,
            max_completion_tokens=MAX_COMPLETION_TOKENS,
            response_format=response_format
        )
        
        # Handle tool calls if the model wants to use them
        if response.completion_message.tool_calls:
            # Add assistant's response to conversation
            messages.append({
                "role": "assistant",
                "tool_calls": [{
                    "id": call.id,
                    "function": {
                        "name": call.function.name,
                        "arguments": call.function.arguments
                    }
                } for call in response.completion_message.tool_calls]
            })
            
            # Execute tools and add results
            for tool_call in response.completion_message.tool_calls:
                try:
                    arguments = json.loads(tool_call.function.arguments)
                    tool_result = execute_tool(tool_call.function.name, arguments)
                    
                    messages.append({
                        "role": "tool",
                        "tool_call_id": tool_call.id,
                        "content": json.dumps(tool_result)
                    })
                except json.JSONDecodeError:
                    return {'success': False, 'error': 'Invalid tool arguments JSON', 'stage': 2}
            
            # Get final structured response
            messages.append({"role": "user", "content": 
                "Provide the complete enriched invoice data following the required schema. "
                "Include conversion details in converted_amount_usd and exchange_rate fields."})
            
            final_response = client.chat.completions.create(
                model=STAGE2_MODEL,
                messages=messages,
                temperature=0.1,
                max_completion_tokens=MAX_COMPLETION_TOKENS,
                response_format=response_format
            )
            
            try:
                enriched_data = json.loads(final_response.completion_message.content.text)
            except json.JSONDecodeError:
                return {'success': False, 'error': 'Invalid JSON response from final call', 'stage': 2}
        else:
            # No tools needed, parse direct response
            try:
                enriched_data = json.loads(response.completion_message.content.text)
            except json.JSONDecodeError:
                return {'success': False, 'error': 'Invalid JSON response from initial call', 'stage': 2}
        
        # Validate required fields in enriched data
        required_enriched_fields = ['vendor_name', 'original_amount']
        missing_enriched = [field for field in required_enriched_fields if field not in enriched_data]
        if missing_enriched:
            return {'success': False, 'error': f'Missing enriched fields: {missing_enriched}', 'stage': 2}
        
        return {
            'success': True,
            'stage': 2,
            'enriched_data': enriched_data
        }
        
    except Exception as e:
        return {
            'success': False,
            'error': str(e),
            'stage': 2
        }

Ahora procesemos los resultados exitosos de la Etapa 1 a través de la Etapa 2:

print("🧠 Processing invoices through Stage 2 (intelligent refinement)...")

stage2_results = []
successful_stage1 = [r for r in stage1_results if r['result']['success']]

for i, stage1_result in enumerate(successful_stage1, 1):
    invoice = stage1_result['invoice']
    result = stage2_intelligent_refinement(stage1_result['result']['raw_data'])
    
    if result['success']:
        enriched, gt = result['enriched_data'], invoice['ground_truth']
        company_match, total_match = evaluate_extraction(enriched, gt, 'original_amount')
        
        # Currency conversion info
        extra_info = None
        if enriched.get('converted_amount_usd') and enriched.get('original_currency') not in ['USD', '$']:
            curr = enriched.get('original_currency', 'Unknown')
            usd = enriched.get('converted_amount_usd', 'N/A')
            extra_info = f"💱 Converted: {curr} → USD ${usd}"
        
        print_comparison(i, invoice['filename'], enriched, gt, company_match, total_match, "Enriched", 'original_amount', extra_info)
        
        stage2_results.append({
            'invoice': invoice, 'stage1_result': stage1_result, 'stage2_result': result,
            'company_match': company_match, 'total_match': total_match
        })
    else:
        print(f"[{i}] {invoice['filename']} ❌ FAILED: {result.get('error', 'Unknown error')}\n")
        stage2_results.append({
            'invoice': invoice, 'stage1_result': stage1_result, 'stage2_result': result,
            'company_match': False, 'total_match': False
        })

successful_stage2 = [r for r in stage2_results if r['stage2_result']['success']]
company_accuracy = sum(1 for r in successful_stage2 if r['company_match']) / len(successful_stage2) if successful_stage2 else 0
total_accuracy = sum(1 for r in successful_stage2 if r['total_match']) / len(successful_stage2) if successful_stage2 else 0
conversions = [r for r in successful_stage2 if r['stage2_result']['enriched_data'].get('converted_amount_usd')]

print(f"✅ Stage 2: {len(successful_stage2)}/{len(successful_stage1)} enriched | "
      f"Accuracy: {company_accuracy:.1%} company, {total_accuracy:.1%} amount")
print(f"   Currency conversions: {len(conversions)} invoices")

🧠 Processing invoices through Stage 2 (intelligent refinement)...
[1] X00016469670.jpg
Enriched: OJC MARKETING SDN BHD | 193.00
Target: OJC MARKETING SDN BHD | 193.00
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $46.09

[2] X00016469671.jpg
Enriched: OJC MARKETING SDN BHD | 170.00
Target: OJC MARKETING SDN BHD | 170.00
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $40.09

[3] X51005200931.jpg
Enriched: PERNIAGAAN ZHENG HUI | 436.20
Target: PERNIAGAAN ZHENG HUI | 436.20
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $97.53

[4] X51005230605.jpg
Enriched: PETRON BKT LANJAN SB | 4.90
Target: PETRON BKT LANJAN SB | 4.90
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $1.16

[5] X51005230616.jpg
Enriched: Gerbang Alaf Restaurants Sdn Bhd (formerly known as Golden Arches Restaurants Sdn Bhd) | 38.90
Target: GERBANG ALAF RESTAURANTS SDN BHD | 38.90
❌ ✗ (Company:✗ | Amount:✓)
💱 Converted: MYR → USD $,{

[6] X51005230621.jpg
Enriched: SIN LIANHAP SDN BHD | 7.30
Target: SIN LIANHAP SDN BHD | 7.30
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $1.75

[7] X51005230648.jpg
Enriched: CROSS CHANNEL NETWORK SDN. BHD. | 6.35
Target: CROSS CHANNEL NETWORK SDN. BHD. | 6.35
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $[convert_currency(amount=6.35, from_currency='MYR', to_currency='USD')]

[8] X51005230657.jpg
Enriched: CROSS CHANNEL NETWORK SDN. BHD. | 10.00
Target: CROSS CHANNEL NETWORK SDN. BHD. | 7.95
❌ ✗ (Company:✓ | Amount:✗)
💱 Converted: MYR → USD $2.40

[9] X51005230659.jpg
Enriched: SWC ENTERPRISE SDN BHD | 8.00
Target: SWC ENTERPRISE SDN BHD | 8.00
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $1.92

[10] X51005268275.jpg
Enriched: LIGHTROOM GALLERY SDN BHD | 278.80
Target: LIGHTROOM GALLERY SDN BHD | 278.80
✅ ✓ (Company:✓ | Amount:✓)
💱 Converted: MYR → USD $62.49

✅ Stage 2: 10/10 enriched | Accuracy: 90.0% company, 90.0% amount
   Currency conversions: 10 invoices

Examinemos las salidas estructuradas finales de nuestro pipeline de dos etapas:

print("📋 Final Structured Outputs from Two-Stage Pipeline:\n")

for i, result in enumerate(successful_stage2, 1):
    invoice = result['invoice']
    enriched_data = result['stage2_result']['enriched_data']
    
    print(f"[{i}] {invoice['filename']}:")
    print(json.dumps(enriched_data, indent=2))
    print("-" * 50)

📋 Final Structured Outputs from Two-Stage Pipeline:

[1] X00016469670.jpg:
{
  "vendor_name": "OJC MARKETING SDN BHD",
  "vendor_address": "NO 2 & 4, JALAN BAYU 4, BANDAR SERI ALAM, 81750 MASAI, JOHOR",
  "invoice_date": "2019-01-15",
  "original_amount": "193.00",
  "original_currency": "MYR",
  "converted_amount_usd": "46.09",
  "exchange_rate": "0.2387",
  "reasoning_notes": "The currency symbol 'SR' was initially provided, but based on the vendor address in Malaysia and the extraction notes, it seems there was a confusion. The address suggests the currency is likely MYR. The amount 193.00 was converted from MYR to USD using the exchange rate 0.2387, resulting in 46.09 USD."
}
--------------------------------------------------
[2] X00016469671.jpg:
{
  "vendor_name": "OJC MARKETING SDN BHD",
  "vendor_address": "NO 2 & 4, JALAN BAYU 4, BANDAR SERI ALAM, 81750 MASAI, JOHOR",
  "invoice_date": "2019-02-01",
  "original_amount": "170.00",
  "original_currency": "MYR",
  "converted_amount_usd": "40.09",
  "exchange_rate": "0.2357",
  "reasoning_notes": "The currency symbol was not explicitly shown, but the vendor is based in Malaysia, and the amounts have two decimal places, suggesting MYR. The exchange rate used for conversion is based on static rates."
}
--------------------------------------------------
[3] X51005200931.jpg:
{
  "vendor_name": "PERNIAGAAN ZHENG HUI",
  "vendor_address": "NO.59 JALAN PERMAS 9/5 BANDAR BARU PERMAS JAYA 81750 JOHOR BAHRU",
  "invoice_date": "2018-02-09",
  "original_amount": "436.20",
  "original_currency": "MYR",
  "converted_amount_usd": "97.53",
  "exchange_rate": "0.2236",
  "reasoning_notes": "The invoice contains a Malaysian address, suggesting the currency is MYR. The extraction notes mention that the invoice is clear and legible. The currency symbol 'RM' is commonly used in Malaysia to represent MYR. Therefore, it is reasonable to assume that the original currency is MYR. The convert_currency tool was used to convert the amount from MYR to USD."
}
--------------------------------------------------
[4] X51005230605.jpg:
{
  "vendor_name": "PETRON BKT LANJAN SB",
  "vendor_address": "KM 458.4 BKT LANJAN UTARA, L/RAYA UTARA SELATAN,SG BULOH 47000 SUNGAI BULOH",
  "invoice_date": "2018-02-01",
  "original_amount": "4.90",
  "original_currency": "MYR",
  "converted_amount_usd": "1.16",
  "exchange_rate": "0.237",
  "reasoning_notes": "The extraction_notes mention that the receipt appears to be from a Petron gas station in Malaysia, and the total amount is clearly stated as RM 4.90. The vendor_address also suggests a Malaysian location, which implies the currency is MYR. The convert_currency tool was used to convert MYR 4.90 to USD."
}
--------------------------------------------------
[5] X51005230616.jpg:
{
  "vendor_name": "Gerbang Alaf Restaurants Sdn Bhd (formerly known as Golden Arches Restaurants Sdn Bhd)",
  "vendor_address": "Level 6, Bangunan TH, Damansara Uptown3 No.3, Jalan SS21/39, 47400 Petaling Jaya Selangor",
  "invoice_date": "2018-01-18",
  "original_amount": "38.90",
  "original_currency": "MYR",
  "converted_amount_usd": ",{",
  "exchange_rate": "",
  "reasoning_notes": ""
}
--------------------------------------------------
[6] X51005230621.jpg:
{
  "vendor_name": "SIN LIANHAP SDN BHD",
  "vendor_address": "LOT 13, JALAN IPOH, KG BATU 30, ULU YAM LAMA 44300 BTG KALI, SELANGOR",
  "invoice_date": "2018-05-02",
  "original_amount": "7.30",
  "original_currency": "MYR",
  "converted_amount_usd": "1.75",
  "exchange_rate": "0.24",
  "reasoning_notes": "The vendor address is in Malaysia, suggesting MYR currency. The extraction notes mention 'RM' next to item prices, which is the currency symbol for Malaysian Ringgit. The total amount is listed as '7.30' under 'Payment'. Using convert_currency tool to convert MYR to USD."
}
--------------------------------------------------
[7] X51005230648.jpg:
{
  "vendor_name": "CROSS CHANNEL NETWORK SDN. BHD.",
  "vendor_address": "47, JALAN MERANTI 1, SEK. 3, BANDAR UTAMA BATANG KALI, 44300 BATANG KALI, SELANGOR",
  "invoice_date": "2018-01-29",
  "original_amount": "6.35",
  "original_currency": "MYR",
  "converted_amount_usd": "[convert_currency(amount=6.35, from_currency='MYR', to_currency='USD')]",
  "exchange_rate": "[convert_currency(amount=1, from_currency='MYR', to_currency='USD')]",
  "reasoning_notes": "The vendor address is in Malaysia, suggesting MYR currency. The currency symbol 'RM' is consistent with MYR. The extraction notes confirm the total amount includes GST at 6%, and the paid amount and change given are in RM, further supporting MYR as the original currency."
}
--------------------------------------------------
[8] X51005230657.jpg:
{
  "vendor_name": "CROSS CHANNEL NETWORK SDN. BHD.",
  "vendor_address": "47, JALAN MERANTI 1, SEK. 3, BANDAR UTAMA BATANG KALI, 44300 BATANG KALI, SELANGOR",
  "invoice_date": "2017-12-31",
  "original_amount": "10.00",
  "original_currency": "MYR",
  "converted_amount_usd": "2.40",
  "exchange_rate": "0.24",
  "reasoning_notes": "The vendor address suggests a Malaysian origin, and the currency symbol 'RM' is commonly used in Malaysia, which corresponds to MYR. The extraction notes confirm the currency symbol 'RM' is used consistently throughout the invoice. Therefore, the original currency is MYR. The amount '10.00' is converted to USD using the convert_currency tool."
}
--------------------------------------------------
[9] X51005230659.jpg:
{
  "vendor_name": "SWC ENTERPRISE SDN BHD",
  "vendor_address": "NO. 5-7, Jalan Mahagoni 7/1, Sekysen 4, Bandar Utama, 44300 Batang Kali, Selangor.",
  "invoice_date": "2018-01-08",
  "original_amount": "8.00",
  "original_currency": "MYR",
  "converted_amount_usd": "1.92",
  "exchange_rate": "0.24",
  "reasoning_notes": "The vendor address suggests a Malaysian origin, which implies the currency might be MYR. Given the partial obscuration of the total amount and the visible '8.00', it is reasonable to assume this is the total in MYR. The exchange rate used for conversion is based on static rates."
}
--------------------------------------------------
[10] X51005268275.jpg:
{
  "vendor_name": "LIGHTROOM GALLERY SDN BHD",
  "vendor_address": "No: 28, JALAN ASTANA 1C, BANDAR BUKIT RAJA, 41050 KLANG SELANGOR D.E, MALAYSIA",
  "invoice_date": "2017-11-20",
  "original_amount": "278.80",
  "original_currency": "MYR",
  "converted_amount_usd": "62.49",
  "exchange_rate": "0.224",
  "reasoning_notes": "The extraction_notes indicate that the receipt is from Lightroom Gallery Sdn Bhd, dated 20/11/2017, and the total amount is RM 278.80. The vendor_address suggests a Malaysian location, which implies the currency is MYR. The currency_symbol 'RM' is commonly used for Malaysian Ringgit. Therefore, the original_currency is MYR. Using the convert_currency tool, we can convert the amount to USD."
}
--------------------------------------------------

Análisis de patrones de falla comunes

Incluso con alta precisión, la extracción basada en LLM puede producir errores que revelan desafíos comunes:

• Sobre-extracción: El modelo extrae información técnicamente correcta pero contextualmente excesiva (por ejemplo, incluyendo el nombre anterior de una empresa). Este patrón a menudo requiere reglas de formato de salida más estrictas o un postprocesamiento más sofisticado.

• Diseño ambiguo: El modelo identifica incorrectamente un campo porque el diseño del documento contiene múltiples candidatos plausibles (por ejemplo, extrayendo un monto de "pago debido" en lugar del "total de la factura"). Esta clase de errores a menudo se maneja mejor implementando puntuaciones de confianza para marcar casos ambiguos para revisión humana.

Para abordar estos patrones de falla, puedes implementar la puntuación de confianza para marcar automáticamente los diseños ambiguos para revisión humana, un paso crítico para transacciones de alto valor. Para problemas como la sobre-extracción, puedes refinar tus prompts con instrucciones de formato más específicas o proporcionar ejemplos de pocas tomas para guiar al modelo hacia la estructura de salida deseada.

Estos patrones de falla subrayan que el objetivo no es eliminar la participación humana, sino aumentarla. Un sistema exitoso maneja de manera confiable la mayoría de las facturas, mientras marca inteligentemente las excepciones complejas para que los especialistas de Cuentas por Pagar las revisen, permitiéndoles concentrarse en decisiones de alto valor.

Próximos pasos y rutas de actualización

Has construido un sistema de procesamiento de facturas que combina las capacidades multimodales de Llama para manejar la complejidad de documentos del mundo real. La arquitectura de dos etapas proporciona una base flexible que se puede adaptar a diversas industrias y requisitos de escala. Aquí te mostramos cómo extender este sistema para necesidades comerciales específicas y requisitos de escala.

Expansión con herramientas de producción

Aunque este tutorial utiliza la conversión de moneda para demostrar la llamada a herramientas, los sistemas de producción suelen integrar herramientas comerciales de alto impacto:

Validación de proveedores: validate_vendor verifica a los proveedores con bases de datos de proveedores aprobados, lo que reduce el riesgo de fraude y garantiza el cumplimiento de las políticas de adquisición.

Detección de duplicados: duplicate_detection evita pagos dobles al comparar montos de facturas, fechas y detalles del proveedor con el historial de pagos recientes.

Aprobación de presupuesto: check_budget_approval verifica las compras con los presupuestos aprobados y los límites de gasto, lo que permite flujos de trabajo de aprobación automatizados para transacciones conformes.

Cada herramienta adicional sigue el mismo patrón: define el esquema de la herramienta, implementa la función y deja que el modelo de Llama decida cuándo usarla según los datos de la factura y las reglas comerciales.