cannot load deepset-mxbai-embed-de-large-v1 without optimization

[arnej27959]

Describe the issue

trying to load the model from
https://huggingface.co/mixedbread-ai/deepset-mxbai-embed-de-large-v1/resolve/main/onnx/model_fp16.onnx
with GraphOptimizationLevel.ORT_DISABLE_ALL fails like this:
[ONNXRuntimeError] : 1 : FAIL : Load model from model_fp16.onnx failed:Type Error: Type (tensor(float16)) of output arg (/0/auto_model/Cast_output_0) of node (/0/auto_model/Cast) does not match expected type (tensor(float)).
After running various python scripts to analyze the model and how onnxruntime loads it, this was the analysis from Claude,
in case that's helpful:

What's wrong: ORT 1.27.0 has a bug in its InsertedPrecisionFreeCast mechanism — triggered by ORT_DISABLE_ALL mode —
that fires on the 24 Cast_1 nodes in the model (one per encoder layer, each doing FP16→FP16).

Why those casts are there: The model computes a dynamic attention scale from the tensor shape — INT64 head_dim →
Cast(→FP16) → Sqrt → Div(1.0/result). ORT's CPU kernels for ops like Sqrt and Div can execute internally in float32
even when the declared types are float16. The model exporter inserted explicit FP16 casts after that chain to enforce
the output type.

The ORT bug: Without optimization, when ORT encounters a Cast whose declared input and output types are both FLOAT16
but whose upstream op actually ran in float32 internally, it renames the Cast output with InsertedPrecisionFreeCast_
and then computes the output type as tensor(float) rather than tensor(float16). The type checker then sees this as a
mismatch against the declared tensor(float16) and fails.

Confirmation: ORT_ENABLE_BASIC eliminates those 24 identity casts (and the 1 BOOL→BOOL one) before the type check
runs, bringing the graph from 1991 to 1916 nodes — and loading succeeds. The graph-preparation step responsible for
InsertedPrecisionFreeCast only triggers at ORT_DISABLE_ALL.

Bottom line: The model itself is ONNX-valid (passes onnx.checker). You need at least ORT_ENABLE_BASIC to load it with
ORT 1.27.0. ORT_DISABLE_ALL hits a type-inference bug in ORT's handling of these casts, not a model-correctness
problem.

To reproduce

wget -L 'https://huggingface.co/mixedbread-ai/deepset-mxbai-embed-de-large-v1/resolve/main/onnx/model_fp16.onnx'

python3 -m venv ~/onnxruntime-env
source ~/onnxruntime-env/bin/activate
pip install --upgrade pip
pip install onnx onnxruntime==1.27.0

python3 << 'EOF'

import onnxruntime as ort
print("onnxruntime", ort.version)

print("OPT LOAD")
options = ort.SessionOptions()
options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
session = ort.InferenceSession("model_fp16.onnx", sess_options=options, providers=["CPUExecutionProvider"])
print("OPT LOAD OK")

print("NO-OPT LOAD")
options = ort.SessionOptions()
options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
session = ort.InferenceSession("model_fp16.onnx", sess_options=options, providers=["CPUExecutionProvider"])
print("NO-OPT LOAD OK")

EOF

Urgency

No response

Platform

Linux

OS Version

AlmaLinux 10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.27.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response