From Training to Endpoint: How We Deploy Custom ML Models

Deploying a custom ML model means packaging its preprocessing, defining a prediction API, validating inputs and monitoring the service. Training is only part of the work. This guide uses a numeric classifier to explain the delivery steps and the decisions that still need project-specific testing.

What "production" means for an ML model

A production ML model needs to:

• Accept requests over HTTP (or another transport)
• Return predictions in a consistent format in predictable time
• Handle errors gracefully without crashing the service
• Be monitorable: you need to know if accuracy is drifting or the endpoint is slow
• Have a documented deployment and rollback procedure

FastAPI is one framework we use for serving custom models. It provides request validation and OpenAPI documentation. Serving capacity still depends on the model, worker configuration and infrastructure.

Step 1: Data preparation

Before training, check whether the data represents the task and whether the target is reliable. A different algorithm will not correct mislabeled outcomes or information leaked from the test set.

Checks to make:

• Schema validation early. Define columns, types, units and acceptable ranges. Reject malformed inputs and review out-of-range values rather than assuming the model can handle them.
• Split before any preprocessing. Fit scalers, encoders, and imputers on training data only. Apply the fit transformers to validation and test sets. Applying fit on the full dataset before splitting is a common data leakage mistake.
• Document the target variable. If it's a classification target, document every class label and its distribution. Imbalanced classes need handling.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import pandas as pd

df = pd.read_csv("training_data.csv")

# Split BEFORE fitting anything
X = df.drop("target", axis=1)
y = df["target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Fit scaler on training data only
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)  # transform only, not fit_transform

Step 2: Training and evaluation

For tabular tasks, compare a simple baseline with gradient boosting such as XGBoost or LightGBM before adding more complexity. Measure quality, training time and inference cost on your own data.

For text, embeddings with a lightweight classifier can be a useful baseline. Image tasks need their own comparison of pretrained models, available labels and deployment constraints.

Define what "good enough" means before training. If the business needs 95% precision on the positive class, record that as a proposed acceptance criterion and check whether the available data can support it. That number is an example, not a promise for every model.

from xgboost import XGBClassifier
from sklearn.metrics import classification_report

model = XGBClassifier(n_estimators=300, learning_rate=0.05, max_depth=6)
model.fit(X_train_scaled, y_train)

y_pred = model.predict(X_test_scaled)
print(classification_report(y_test, y_pred))

If metrics do not meet the agreed bar, inspect errors and labels before choosing more data, different features or another model. For time-dependent or grouped records, replace the random split above with a split that respects time or entity boundaries.

Step 3: Packaging the model and its transformers

The serving process needs the same preprocessing used during training. For this example, that includes the fitted scaler and the expected feature order. Omitting either can cause incorrect predictions or a runtime error.

We use joblib for sklearn-compatible transformers and the model. We package them together into a single artifact:

import joblib

# Save as a bundle, not separately
artifact = {
    "model": model,
    "scaler": scaler,
    "feature_columns": list(X_train.columns),
    "model_version": "1.0.0",
    "trained_at": "2026-06-01",
}

joblib.dump(artifact, "model_artifact.joblib")

Include the artifact version and pin the training and serving dependencies. Record provenance and verify artifact integrity through your trusted release process. Never load a user-uploaded or untrusted joblib file: loading it can execute code. See the scikit-learn model persistence guidance.

Step 4: The FastAPI endpoint

This illustrative endpoint uses Pydantic 2 and a numeric classifier with integer class labels and predict_proba. It rejects missing, unexpected and non-finite features. Its request and failure paths were checked with FastAPI 0.141.1 and Pydantic 2.13.5 using model test doubles, not a trained-model performance benchmark. Add domain-specific range checks, authentication, request limits and load testing before exposing a real service.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, ConfigDict, FiniteFloat
import joblib
import pandas as pd
import numpy as np

app = FastAPI(title="ML Model API", version="1.0.0")

# Load once at startup, not on every request
artifact = joblib.load("model_artifact.joblib")
model = artifact["model"]
scaler = artifact["scaler"]
feature_columns = artifact["feature_columns"]

class PredictRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")
    features: dict[str, FiniteFloat]

class PredictResponse(BaseModel):
    prediction: int
    probability: float
    model_version: str

@app.post("/predict", response_model=PredictResponse)
def predict(request: PredictRequest):
    if set(request.features) != set(feature_columns):
        raise HTTPException(status_code=422, detail="Feature names do not match the model schema")
    try:
        # Preserve the training feature order; never silently invent a value.
        df = pd.DataFrame([request.features], columns=feature_columns)
        
        scaled = scaler.transform(df)
        prediction = int(model.predict(scaled)[0])
        probability = float(np.max(model.predict_proba(scaled)[0]))
        
        return PredictResponse(
            prediction=prediction,
            probability=probability,
            model_version=artifact["model_version"],
        )
    except Exception:
        # Use sanitized operational telemetry, not request data or raw errors.
        raise HTTPException(status_code=500, detail="Prediction unavailable") from None

@app.get("/health")
async def health():
    return {"status": "ok", "model_version": artifact["model_version"]}

Key decisions in this endpoint:

• Load the trusted artifact once per worker. Avoid repeated loading; benchmark latency and memory with your actual model and worker count.
• Validate inputs. Missing features are errors unless the training pipeline includes an explicitly designed imputation strategy. Pydantic may coerce numeric strings; use strict field types if your contract must reject them.
• Keep blocking inference out of an async handler. This synchronous handler uses FastAPI's thread-pool behavior. CPU-heavy work still needs capacity testing or a separate worker service.
• Treat probability carefully. The largest class probability is not a guarantee of correctness. Evaluate calibration before using it to automate consequential decisions.
• Return the model version. Every prediction response should include which model generated it.
• Health endpoint. Required for load balancers and uptime monitoring.

Step 5: Docker packaging

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model_artifact.joblib .
COPY main.py .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

One decision to make: include the model artifact in the Docker image (simpler, larger image) or load it at startup from S3/GCS (more complex, enables model updates without rebuilding the image). For most clients, we start with artifact-in-image and move to remote loading when model iteration frequency justifies it.

Step 6: Monitoring in production

Monitor service behavior and model quality:

• Latency. Measure P95 and P99 as well as the average, then compare them with the workflow's response-time requirements.
• Prediction distribution. Track the class distribution over time. A large shift is a reason to investigate inputs, model versions and pipeline behavior, not proof of a particular cause.

Prometheus and Grafana can track service metrics. Aggregate prediction distributions without recording unnecessary personal data. Agree access and retention rules for any samples retained for evaluation.

import time
from fastapi import Request

@app.middleware("http")
async def add_timing_header(request: Request, call_next):
    start = time.perf_counter()
    response = await call_next(request)
    duration = (time.perf_counter() - start) * 1000
    response.headers["X-Response-Time-Ms"] = str(round(duration, 2))
    return response

Input distribution drift means the values reaching the model have changed. That can signal a broken upstream feed, a new audience or a seasonal pattern. It does not by itself prove that accuracy has fallen.

Evidently's data and ML checks provide one way to compare reference and recent datasets. Its current documentation uses Dataset, DataDefinition, Report and DataDriftPreset; older ColumnMapping examples use a different API. Pin a tested version and follow its matching documentation. This article does not include an untested vendor-specific monitoring script.

Choose the drift method and threshold for each feature, sample size and business context. PSI values such as 0.1 or 0.25 are sometimes used as heuristics, not universal rules for retraining. Investigate a change, check label quality and measure performance on recent reviewed outcomes before retraining. Do not use the model's own predictions as unquestioned training labels.

Compare a replacement model against a versioned holdout and relevant recent labeled data. Keep a rollback path. A weekly review may suit a low-volume batch workflow; a high-volume or higher-risk service may need more frequent checks.

What this looks like end-to-end

Agree these steps before committing to delivery:

1. Data review, access permissions and feature definitions.
2. Training pipeline and experiment records.
3. Evaluation against the agreed acceptance criteria.
4. Prediction endpoint, schema validation and failure tests.
5. Container deployment, capacity checks and rollback.
6. Monitoring, documentation and named support ownership.

A 7–10-business-day schedule can be a planning estimate for a small classifier with clean labeled data, an agreed target and ready infrastructure. It is not a fixed delivery promise. Labeling, security review, integration work and additional evaluation can change the scope and timing.

If you need a custom ML model built, deployed, and monitored in production, tell us what you're trying to predict.

For a related document-processing implementation, see the DataVersion Document Intelligence case study. If the model is one step in a wider business process, see Operations Automation.

Related: Fine-Tuning vs RAG: How to Choose for Your Use Case

AI & Machine Learning services →