LLM Stigma Language

For the prompts, together with the models’ outputs and LLM reasoning traces of the tasks that are allowed to be open-sourced, please visit the HuggingFace dataset.

This repository contains the data and Python plotting scripts used to generate the figures for Large Language Models Generate Stigmatizing Language During Reasoning Over Real-World Clinical Data. It reproduces figures from the tabular files in Data/ and figure3/figure3a/figure3a.xlsx; it does not regenerate those source tables from the original model outputs.

1. System requirements

Software dependencies

The scripts require Python and the following third-party packages. The exact versions below are the versions on which the repository was validated:

Software	Tested version
Python	3.11.9
NumPy	2.2.6
pandas	3.0.2
Matplotlib	3.10.9
seaborn	0.13.2
SciPy	1.13.1
openpyxl	3.1.5
adjustText	1.4.0

openpyxl is needed for the .xlsx inputs and outputs. All other imported modules are part of the Python standard library.

Tested operating system

The scripts were tested on Ubuntu 22.04.5 LTS (Linux 5.15, x86-64). Other operating systems have not been tested in this repository.

Hardware

No non-standard hardware is required. The scripts perform tabular analysis and plotting on a CPU; no GPU is used. A normal desktop or laptop with enough free space for the Python environment and generated PDF/SVG files is sufficient.

2. Installation guide

From the repository root, create an isolated environment and install the tested dependency versions:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install \
  numpy==2.2.6 \
  pandas==3.0.2 \
  matplotlib==3.10.9 \
  seaborn==0.13.2 \
  scipy==1.13.1 \
  openpyxl==3.1.5 \
  adjustText==1.4.0

On Windows, activate the environment with .venv\Scripts\activate instead.

Installation typically takes approximately 2–5 minutes on a normal desktop with a broadband connection. The exact time depends mainly on network speed and whether prebuilt packages are available for the operating system.

3. Demo

Figure 2A is a short end-to-end demo. It reads the included Data/stigma_info_107models.xlsx, computes summary statistics, and renders the plot:

source .venv/bin/activate
python figure2/figure2a/figure2a.py

Expected terminal output consists of the paths to these three generated files:

figure2/figure2a/figure2a_stats.txt
figure2/figure2a/figure2a.svg
figure2/figure2a/figure2a.pdf

The statistics file contains entries such as models_n, total_pairs, nonzero_pairs, the mean and median non-zero stigma rates, and the observed range. A successful run exits without an exception and creates all three files in figure2/figure2a/.

The validated demo run took approximately 3 seconds. On a normal desktop, allow about 5–15 seconds; the first Matplotlib run can take longer while it creates a font cache.

4. Instructions for use

Run commands from the repository root unless a command explicitly changes directory. Each script reads the included table shown below and writes its outputs beside the script.

Result	Command	Included input(s)	Expected output(s)
Figure 1	python figure1/figure1.py	Data/stigma_info_107models.xlsx; Data/Clinical Benchmark and LLM 107.xlsx	figure1.pdf, figure1.svg
Figure 2A	python figure2/figure2a/figure2a.py	Data/stigma_info_107models.xlsx	figure2a.pdf, figure2a.svg, figure2a_stats.txt
Figure 2B	python figure2/figure2b/figure2b.py	Data/stigma_info_107models.xlsx; Data/Clinical Benchmark and LLM 107.xlsx	figure2b.pdf, figure2b.svg, figure2b_stats.txt
Figure 2C–D	python figure2/figure2cd/figure2cd.py	Data/stigma_info_107models.xlsx; Data/Clinical Benchmark and LLM 107.xlsx	figure2cd.pdf, figure2cd.svg, figure2cd_lollipop_data.xlsx
Figure 3A	python figure3/figure3a/figure3a.py	figure3/figure3a/figure3a.xlsx	figure3a.pdf, figure3a.svg
Figure 3B	python figure3/figure3b/figure3b.py	Data/stigma_term_distribution_by_domain.csv	figure3b.svg
Figure 4A	python figure4/figure4a/figure4a.py	Data/output_stigma_rate.csv	figure4a.svg
Figure 4B	python figure4/figure4b/figure4b.py	Data/stigma_info_107models.xlsx	figure4b.pdf, figure4b.svg, figure4b_stats.txt
Figure 4C	python figure4/figure4c/figure4c.py	Data/performance_107_stigma_models_only.xlsx; Data/stigma_info_107models.xlsx	figure4c.svg, accuracy_stigma_long.csv, accuracy_stigma_long.xlsx
Figure 5A	python figure5/figure5a/figure5a.py	Data/output_stigma_rate.csv; Data/output_stigma_rate_destigmatized.csv	figure5a.svg
Figure 5B	python figure5/figure5b/figure5b.py	Data/CLF_CoT.csv; Data/CLF_CoT_destigmatized.csv; Data/EXT_CoT.csv; Data/EXT_CoT_destigmatized.csv	figure5b.svg
Supplementary Figure S1	python supplementary/appendix_1_task_taxonomy.py	Data/stigma_info_107models.xlsx; Data/Clinical Benchmark and LLM 107.xlsx	figure_s1_task_taxonomy.pdf, figure_s1_task_taxonomy.svg

Running the software on your own data

All scripts use fixed input and output paths declared near the top of each script. They expect the same filenames, worksheet names, columns, model identifiers, and table layouts as the bundled inputs. To use your own data:

1. Make a backup of the included input file.
2. Either replace it with your table while preserving its schema and filename, or update the corresponding path constant near the top of the script.
3. Run the corresponding command from the table above.
4. Check that the expected output files were created and inspect any generated *_stats.txt, CSV, or Excel summary before using the figure.

Reproducing all figures

After installing the dependencies, the following commands reproduce every directly runnable figure and supplementary figure from the included data:

python figure1/figure1.py
python figure2/figure2a/figure2a.py
python figure2/figure2b/figure2b.py
python figure2/figure2cd/figure2cd.py
python figure3/figure3a/figure3a.py
python figure3/figure3b/figure3b.py
python figure4/figure4a/figure4a.py
python figure4/figure4b/figure4b.py
python figure4/figure4c/figure4c.py
python figure5/figure5a/figure5a.py
python figure5/figure5b/figure5b.py
python supplementary/appendix_1_task_taxonomy.py

These plotting scripts reproduce the figures and the summary artifacts listed above. They do not reproduce the upstream model inference or the creation of the bundled source tables because that code is not present in this repository.