Classify and extract in one-shot using tools

I recently resolved a misconception around tool calling that I felt was worth sharing.

I had been under the impression that given various function signatures and descriptions, the LLM would decide which one would be appropriate and then send back text that could basically be eval()’ed back in your python runtime to allow the continuation of you workflow.

Below I demonstrate an intriguing workaround that happens to work because the follow-on work is actually just data extraction according to the “tool” that the LLM selects.

Using OpenAI’s function calling with Pydantic models, we define multiple data models that define the fields we’d like to extract. We supply instructions and the content from which the data should be extracted. This approach:

Combines classification and extraction into a single API call
Provides structured, validated output
Reduces complexity and latency
Ensures type safety through Pydantic validation

Use Case: Medical Record Processing

We’ll demonstrate this technique using medical record processing as an example, where we need to extract different types of information:

Patient demographic information
Medication records
Diagnosis records

Setup

from typing import List, Optional

import openai
from pydantic import BaseModel, Field
from rich import print

from op_secrets import get_key

client = openai.OpenAI(api_key=await get_key("openai"))

Data models

We define our data models using pydantic, affording not only type validation but natural hints to the LLM about what data should be extracted.

class PatientInfo(BaseModel):
    patient_name: str = Field(..., description="Full name of the patient")
    age: int
    gender: Optional[str]

class MedicationRecord(BaseModel):
    medication_name: str
    dose_mg: float = Field(..., description="Dose in milligrams")
    frequency_per_day: int

class DiagnosisRecord(BaseModel):
    diagnosis_code: str = Field(..., description="ICD-10 code")
    description: str

Tools

This helper function openai.pydantic_function_tool JSONifies our pydantic model with some extra info needed for OpenAI’s API contract. It’s helpful to see exactly which parts are being rendered as text to understand what details might inform the LLM of our desired output.

def get_tools(models: List[BaseModel]):
    return [openai.pydantic_function_tool(model) for model in models]


tools = get_tools([PatientInfo, MedicationRecord, DiagnosisRecord])
print(tools)

[
    {
        'type': 'function',
        'function': {
            'name': 'PatientInfo',
            'strict': True,
            'parameters': {
                'properties': {
                    'patient_name': {
                        'description': 'Full name of the patient',
                        'title': 'Patient Name',
                        'type': 'string'
                    },
                    'age': {'title': 'Age', 'type': 'integer'},
                    'gender': {'anyOf': [{'type': 'string'}, {'type': 'null'}], 'title': 'Gender'}
                },
                'required': ['patient_name', 'age', 'gender'],
                'title': 'PatientInfo',
                'type': 'object',
                'additionalProperties': False
            }
        }
    },
    {
        'type': 'function',
        'function': {
            'name': 'MedicationRecord',
            'strict': True,
            'parameters': {
                'properties': {
                    'medication_name': {'title': 'Medication Name', 'type': 'string'},
                    'dose_mg': {'description': 'Dose in milligrams', 'title': 'Dose Mg', 'type': 'number'},
                    'frequency_per_day': {'title': 'Frequency Per Day', 'type': 'integer'}
                },
                'required': ['medication_name', 'dose_mg', 'frequency_per_day'],
                'title': 'MedicationRecord',
                'type': 'object',
                'additionalProperties': False
            }
        }
    },
    {
        'type': 'function',
        'function': {
            'name': 'DiagnosisRecord',
            'strict': True,
            'parameters': {
                'properties': {
                    'diagnosis_code': {'description': 'ICD-10 code', 'title': 'Diagnosis Code', 'type': 'string'},
                    'description': {'title': 'Description', 'type': 'string'}
                },
                'required': ['diagnosis_code', 'description'],
                'title': 'DiagnosisRecord',
                'type': 'object',
                'additionalProperties': False
            }
        }
    }
]

Patient Info

For this first pass, I print the objects to help the reader internalize the context of what’s being passed around.

Request Object

The primary content of the request payload to OpenAI is the messages object.

messages is a list of dictionaries where role and content are defined.

role can take on one of ["system", "user", "assistant"].

content is itself a list of dictionaries. The relevant ones for this topic could include text, image_url, and base64-encoding an image.

An important note on image_url is adding the level of detail required. The default of auto often reduces the image to 512x512 pixels, which is generally too small for the OCR use case when trying to extract content from a JPG representation of a document. More details can be found at: Specify image input detail level.

import base64


def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

image_path = "path_to_your_image.jpg"
base64_image = encode_image(image_path)

[
    {
        "type": "text",
        "text": "This is the content being consumed by the LLM."
    },
    {
        "type": "image_url",
        "image_url": {"url": "https://picsum.photos/200", "detail": "high"}
    },
    {
        "type": "image_url",
        "image_url": {"url": f"data:image/jpeg;base64,{base64_image}", "detail": "high"}
    },
]

text = "Patient John Doe is a 45-year-old male diagnosed with E11.9, type 2 diabetes mellitus without complications."

messages = [
    {
        "role": "user",
        "content": (
            "Given the following paragraph, pick which extractor tool to call (patient info, "
            "medication record, diagnosis record) and extract the relevant fields strictly "
            "as JSON arguments:\n\n" + text
        ),
    }
]
print(messages)

[
    {
        'role': 'user',
        'content': 'Given the following paragraph, pick which extractor tool to call (patient info, medication 
record, diagnosis record) and extract the relevant fields strictly as JSON arguments:\n\nPatient John Doe is a 
45-year-old male diagnosed with E11.9, type 2 diabetes mellitus without complications.'
    }
]

Completion Object

The ParsedChatCompletion object takes a bit of getting used to. Note that parse is only available through the client.beta as of this writing.

Typically we’re interested in completion.choices[0].message.parsed, but with the use of tool calls, we want to pay attention to completion.choices[0].message.tool_calls[0].function.parsed_arguments.

completion = client.beta.chat.completions.parse(
    model="gpt-4.1-mini",
    messages=messages,
    tools=tools,
)

print(completion)

ParsedChatCompletion[NoneType](
    id='chatcmpl-Bm2RFaNOkXyeoK8zHRKMgEnn7Fko4',
    choices=[
        ParsedChoice[NoneType](
            finish_reason='tool_calls',
            index=0,
            logprobs=None,
            message=ParsedChatCompletionMessage[NoneType](
                content=None,
                refusal=None,
                role='assistant',
                annotations=[],
                audio=None,
                function_call=None,
                tool_calls=[
                    ParsedFunctionToolCall(
                        id='call_RyDAB2tXpMnTqsjtihBjyQEn',
                        function=ParsedFunction(
                            arguments='{"patient_name": "John Doe", "age": 45, "gender": "male"}',
                            name='PatientInfo',
                            parsed_arguments=PatientInfo(patient_name='John Doe', age=45, gender='male')
                        ),
                        type='function'
                    ),
                    ParsedFunctionToolCall(
                        id='call_f80gpTgfK2ShJBvFRyAE1ol1',
                        function=ParsedFunction(
                            arguments='{"diagnosis_code": "E11.9", "description": "type 2 diabetes mellitus without
complications"}',
                            name='DiagnosisRecord',
                            parsed_arguments=DiagnosisRecord(
                                diagnosis_code='E11.9',
                                description='type 2 diabetes mellitus without complications'
                            )
                        ),
                        type='function'
                    )
                ],
                parsed=None
            )
        )
    ],
    created=1750788817,
    model='gpt-4.1-mini-2025-04-14',
    object='chat.completion',
    service_tier='default',
    system_fingerprint='fp_6f2eabb9a5',
    usage=CompletionUsage(
        completion_tokens=71,
        prompt_tokens=208,
        total_tokens=279,
        completion_tokens_details=CompletionTokensDetails(
            accepted_prediction_tokens=0,
            audio_tokens=0,
            reasoning_tokens=0,
            rejected_prediction_tokens=0
        ),
        prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)
    )
)

Pydantic Return

The parsed_arguments is a hydrated pydantic model. The class and data can be extracted as follows.

extracted_data = completion.choices[0].message.tool_calls[0].function.parsed_arguments

assert isinstance(extracted_data, PatientInfo)
assert issubclass(extracted_data.__class__, BaseModel)

print(f"Classification: {extracted_data.__class__.__name__}")
print(f"Extraction: {extracted_data.model_dump_json()}")

Classification: PatientInfo

Extraction: {"patient_name":"John Doe","age":45,"gender":"male"}

MedicationRecord

Verifying that the data extracted for MedicationRecord works as expcted.

text = "The patient is prescribed Metformin at a dose of 500 mg, to be taken twice daily after meals."

messages = [
    {
        "role": "user",
        "content": (
            "Given the following paragraph, pick which extractor tool to call (patient info, "
            "medication record, diagnosis record) and extract the relevant fields strictly "
            "as JSON arguments:\n\n" + text
        ),
    }
]

completion = client.beta.chat.completions.parse(
    model="gpt-4.1-mini",
    messages=messages,
    tools=tools,
)
extracted_data = completion.choices[0].message.tool_calls[0].function.parsed_arguments

print(f"Classification: {extracted_data.__class__.__name__}")
print(f"Extraction: {extracted_data.model_dump_json()}")

Classification: MedicationRecord

Extraction: {"medication_name":"Metformin","dose_mg":500.0,"frequency_per_day":2}

DiagnosisRecord

Finally, validating that DiagnosisRecord works as expected as well.

text = "The patient has been diagnosed with J45.909, unspecified asthma, uncomplicated."

messages = [
    {
        "role": "user",
        "content": (
            "Given the following paragraph, pick which extractor tool to call (patient info, "
            "medication record, diagnosis record) and extract the relevant fields strictly "
            "as JSON arguments:\n\n" + text
        ),
    }
]

completion = client.beta.chat.completions.parse(
    model="gpt-4.1-mini",
    messages=messages,
    tools=tools,
)
extracted_data = completion.choices[0].message.tool_calls[0].function.parsed_arguments

print(f"Classification: {extracted_data.__class__.__name__}")
print(f"Extraction: {extracted_data.model_dump_json()}")

Classification: DiagnosisRecord

Extraction: {"diagnosis_code":"J45.909","description":"unspecified asthma, uncomplicated"}