About

Data cleaning experiment with OpenAI chat completion API and the Instructor library.
The objective was to see how pydantic models can be leveraged to clean data and ensure a structured output consistently.
See notebook here.

The material used is a csv file with unstructured fake customer data. The file consists of a single column where each cell refers to a unique customer.

In each cell, the following information can be found, in a random order:

• first name
• last name
• current age
• city
• account type (premium or non-premium)

The data structure is intentionnally diverse for a more challenging test case.

By using models and passing them in the response_model parameter of the function, the following output can be obtained

Model used:

class Premium(Enum):
    """Defines the type of account of the customer, either premium or non-premium"""
    PREMIUM = 1
    NON_PREMIUM = 0

class Customer(BaseModel):
    """Represents a customer, including their first name, last name, age, city, and type of account"""
    first_name: str = Field(..., description="The first name of the customer")
    last_name: str = Field(..., description="The last name of the customer")
    age: int = Field(..., description="The age of the customer")
    city: str = Field(... , description="The city where the customer lives")
    premium: Premium = Field(..., description="Type of account, either premium or non-premium")

class CustomerList(BaseModel):
    """A list of customers"""
    customers: List[Customer] = Field(..., description="A list of customers")