diff --git a/docs/blog/posts/2026_09_impactia_integration.md b/docs/blog/posts/2026_09_impactia_integration.md new file mode 100644 index 00000000..8b34d173 --- /dev/null +++ b/docs/blog/posts/2026_09_impactia_integration.md @@ -0,0 +1,52 @@ +--- +title: Integrating Impact'IA into EcoLogits +date: 2026-09-20 +authors: + - samuelrince +slug: impactia-integration +description: > + EcoLogits integrates the work of the Impact'IA calculator from SNCF, Wavestone and Resilio: input tokens, data center network and building impacts, and an experimental estimation of training impacts. +categories: + - Methodology +--- + +# Integrating Impact'IA into EcoLogits + +[Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) is an LLM environmental footprint calculator and a detailed methodology guide developed by **SNCF**, with **Wavestone** and **Resilio**, to help companies assess the environmental impacts of their generative AI projects. It builds on the EcoLogits methodology for the inference phase and extends it to a broader scope of the model life cycle. Its authors explicitly chose not to industrialize it and suggested integrating their work into open source tools such as EcoLogits. This is what this release does. + + + +## What changes for the inference phase + +Three improvements from Impact'IA are now part of the core [LLM inference methodology](../../methodology/llm_inference.md) and apply to every request: + +* **Input tokens are taken into account.** The pre-fill latency (time-to-first-token) is now modeled as a linear function of the number of input tokens, scaled to each model with its median time-to-first-token measured on OpenRouter. Long prompts (large contexts, retrieval-augmented generation, agents) now increase the estimated latency, thus the server and embodied impacts of the request. All the tracers pass the number of input tokens reported by the providers. +* **Data center network equipment is included.** The firewalls, routers and switches attached to each server are accounted for both in the electricity consumption and in the embodied impacts (GWP) of the request. +* **Data center building embodied impacts are included.** The construction of the building and of the technical environment (electrical and cooling equipment) is allocated per kWh of IT electricity consumption (GWP). + +These additions are modest in magnitude (a few percent of the impacts of a request) but they widen the scope of what EcoLogits accounts for, in line with life cycle assessment practices. + +## An experimental training phase + +The most discussed addition of Impact'IA is the **allocation of the training impacts** of a model to each request. Training impacts (final run, research and development experiments, dataset storage) are estimated top-down from the model size, its release date and the compute capacity of the provider, then shared between all the tokens the model is expected to generate over its lifetime. + +EcoLogits now reports this estimation in a new `training` phase of the impacts, see the [LLM training methodology](../../methodology/llm_training.md). + +```python +response = client.chat.completions.create(model="gpt-4.1-mini", messages=[...]) + +print(response.impacts.gwp.value) # inference impacts (usage + embodied) +print(response.impacts.training.gwp.value) # training impacts allocated to the request +``` + +We chose to report the training phase **separately** and to **not include it in the total impacts** of a request for two reasons. First, the estimation of the number of tokens a model generates during its lifetime is very uncertain (at least one order of magnitude), as providers do not publish these figures. Second, the training phase can only be estimated for models with a known release date and for providers with public compute capacity announcements, so including it in the totals would make models and providers harder to compare. + +While reproducing the calculator, we also found a unit inconsistency in the Impact'IA spreadsheet that divides the estimated number of generated tokens by 1000, which explains most of the training share reported by the calculator (13% of the inference impacts in median). With consistent units, the training share estimated by EcoLogits is typically below 1% for the large providers. We reported it to the Impact'IA team and the methodology will keep evolving with better data on token volumes, such as the ones some providers started publishing. + +## What we did not integrate + +Impact'IA also models the impacts of the **retrieval-augmented generation embeddings** and of the **application platform** (the servers hosting the application around the model) of a project. The former is on our roadmap as an embeddings methodology. The latter depends on each application architecture and is out of the scope of an API-level library, it is better assessed with tools such as [e-footprint](https://e-footprint.boavizta.org/). Finally, Impact'IA reports an AWARE-weighted water use indicator (m³ eq) that differs from the water consumption footprint (L) reported by EcoLogits, so the embodied water values were not merged. + +## Thanks + +Many thanks to Xavier Verne and Thibaud Auzière (SNCF), Tatiana Fouquet, Benoit Durand and Feriel Sraieb (Wavestone), Rosendo Manas Faura and You Li (Resilio) for this work and for sharing it with the community. The Impact'IA methodology guide and calculator are published under the CC BY-NC-SA license. diff --git a/docs/methodology/index.md b/docs/methodology/index.md index 6cd0dd47..9848b890 100644 --- a/docs/methodology/index.md +++ b/docs/methodology/index.md @@ -5,6 +5,7 @@ The following methodologies are **currently available and implemented in EcoLogits**: - [x] [LLM Inference](llm_inference.md) +- [x] [LLM Training](llm_training.md) (experimental, adapted from [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA)) - [x] [Video Generation](video_generation.md) Upcoming methodologies ([join us](https://discord.gg/CAecQ2zM4n) to help speed up our progress): @@ -32,7 +33,7 @@ The methodology is grounded in **transparency** and **reproducibility**, utilizi ## Scope of the methodology -Our methodology focuses on **assessing the environmental impacts of GenAI inference tasks**. That is why we exclude impacts from training, networking and end-used devices, we thoroughly evaluate the impacts associated with hosting and running the model inferences. +Our methodology focuses on **assessing the environmental impacts of GenAI inference tasks**. We thoroughly evaluate the impacts associated with hosting and running the model inferences, including the network equipment and the building of the data center. We exclude the network outside of the data center and the end-user devices. The impacts of the model training are estimated separately with an [experimental methodology](llm_training.md) and are not included in the total impacts of a request.
![Figure: Energy consumption per output token vs. number of active parameters ](../assets/methodology/llm/figure_boundaries.png) diff --git a/docs/methodology/llm_inference.md b/docs/methodology/llm_inference.md index da0ec6c5..b3d3dcd0 100644 --- a/docs/methodology/llm_inference.md +++ b/docs/methodology/llm_inference.md @@ -105,6 +105,22 @@ For a typical high-end GPU-accelerated cloud instance, we use $W_{\text{server} }' | jq .verbose.avg_power ``` +#### Modeling network equipment energy consumption + +Servers are connected through network equipment inside the data center (firewalls, routers and switches) that consume electricity as well. Following the [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) methodology, we consider that a fixed power, $W_{\text{network}}$, of network equipment is attached to each server, and allocate it to the request the same way as the server power: + +$$ +E_{\text{network}}(\Delta T) = \Delta T \times W_{\text{network}} \times \frac{\text{GPU}}{\#\text{GPU}_{\text{installed}}} \times \frac{1}{B}. +$$ + +The network power attached to one server is the sum of the power of each equipment weighted by its effective usage ratio (share of the equipment capacity used by one server), $W_{\text{network}} = \sum_i W_i \times r_i \approx 161$ W, with the following values from [ADEME Base Empreinte®](https://base-empreinte.ademe.fr/) and the [ADEME-ARCEP-ARCOM report](https://www.arcep.fr/la-regulation/grands-dossiers-thematiques-transverses/lempreinte-environnementale-du-numerique.html): + +| Equipment | Power $W_i$ | Usage ratio $r_i$ | +|-----------|-------------|-------------------| +| Firewall | 90 W | 0.0358 | +| Router | 90 W | 0.286 | +| Switch | 90 W | 1.468 | + #### Estimating the generation latency The generation latency, $\Delta T$, is the duration of the inference measure on the server and is independent of networking latency. It is estimated based on real-world statistical metrics from [OpenRouter](https://openrouter.ai) when available and falls back to an alternative approach using the [ML.ENERGY Leaderboard](https://ml.energy/leaderboard/?__theme=light) benchmark, previously mentioned. @@ -117,6 +133,16 @@ $$ \Delta T(\#T_{\text{out}}) = \min \left\{ \text{TTFT} + \frac{\#T_{\text{out}}}{\text{TPS}}, \Delta T_{\text{request}}\right\}. $$ +##### Pre-fill latency and input tokens + +The pre-fill phase consists in processing the input tokens (the prompt) before generating the first output token, its duration thus depends on the number of input tokens, $\#T_{\text{in}}$. When the number of input tokens is known, we follow the improvement proposed by [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) and model the pre-fill latency as a linear function of the number of input tokens, based on the measurements of [Agrawal et al. (2024)](https://www.usenix.org/conference/osdi24/presentation/agrawal) on Mistral 7B ($\text{TTFT} = 0.062 \text{ ms} \times \#T_{\text{in}} + 17.88 \text{ ms}$). The per-token coefficient is scaled to the target model with the ratio of its median time-to-first-token on OpenRouter to the one of Mistral 7B (0.44 s): + +$$ +\text{TTFT}(\#T_{\text{in}}) = 0.062 \text{ ms} \times \frac{\text{TTFT}_{\text{model}}}{0.44 \text{ s}} \times \#T_{\text{in}} + 17.88 \text{ ms}. +$$ + +When the time-to-first-token of the model is not available on OpenRouter, the coefficient is scaled with the number of active parameters instead ($P_{\text{active}} / 7.3$). When the number of input tokens is unknown, the median time-to-first-token is used as is. + ??? info "Alternative approach using ML.ENERGY benchmark data (will be deprecated)" @@ -169,10 +195,10 @@ To stay consistent with previous assumptions based on [ML.ENERGY Leaderboard](ht #### Complete server energy consumption -The total server energy consumption for the request, $E_{\text{server}}$, is computed as follows: +The total server energy consumption for the request, $E_{\text{server}}$, including the network equipment, is computed as follows: $$ -E_{\text{server}} = E_{\text{server} \backslash \text{GPU}} + \text{GPU} \times E_{\text{GPU}}. +E_{\text{server}} = E_{\text{server} \backslash \text{GPU}} + \text{GPU} \times E_{\text{GPU}} + E_{\text{network}}. $$ ### Modeling request energy consumption @@ -254,17 +280,44 @@ I^{\text{e}}_{\text{server}}=\frac{\text{GPU}}{\#\text{GPU}_{\text{installed}}} $$ +### Modeling network equipment embodied impacts + +Following the [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) methodology, we also account for the embodied impacts of the network equipment attached to each server. The embodied impacts of each equipment are weighted by its usage ratio (see [above](#modeling-network-equipment-energy-consumption)), $I^{\text{e}}_{\text{network}} = \sum_i I^{\text{e}}_i \times r_i \approx 660$ kgCO2eq, using the GWP values from the [Resilio DB](https://db.resilio.tech/) reported by Impact'IA: + +| Equipment | GWP (kgCO2eq) | +|------------------------------------|---------------| +| Firewall (Fortinet FortiGate 100E) | $334$ | +| Router (Juniper PTX10002-36QDD) | $403$ | +| Switch (Arista 7050TX3-48C8) | $363$ | + +The lifetime of the network equipment, $\Delta L_{\text{network}}$, is fixed at 5 years. + +!!! warning "Network equipment embodied impacts are only modeled for GWP due to a lack of data for ADPe and PE." + +### Modeling data center building embodied impacts + +The construction of the data center building and of its technical environment (electrical and cooling equipment) also has embodied impacts. We allocate them in proportion to the IT electricity consumption of the request, with a factor $F_{\text{building}} = 0.01$ kgCO2eq per kWh of IT electricity, derived by [Hubblo](https://hubblo.org/) from the ADEME product category rules for cloud services and reported by Impact'IA. + +!!! warning "Data center building embodied impacts are only modeled for GWP due to a lack of data for ADPe and PE." + ### Modeling request embodied environmental impacts -To allocate the server embodied impacts to the request, we use an allocation based on the hardware utilization factor, $\frac{\Delta T}{B \times \Delta L}$. In this case, $\Delta L$ represents the lifetime of the server and GPU, which we fix at 3 years ([Ostrouchov et al. (2020)](https://www.osti.gov/servlets/purl/1771896), [tomshardware.com](https://www.tomshardware.com/pc-components/gpus/datacenter-gpu-service-life-can-be-surprisingly-short-only-one-to-three-years-is-expected-according-to-unnamed-google-architect)), and $B$ is the batch size such as above: +To allocate the server embodied impacts to the request, we use an allocation based on the hardware utilization factor, $\frac{\Delta T}{B \times \Delta L}$. In this case, $\Delta L$ represents the lifetime of the server and GPU, which we fix at 3 years ([Ostrouchov et al. (2020)](https://www.osti.gov/servlets/purl/1771896), [tomshardware.com](https://www.tomshardware.com/pc-components/gpus/datacenter-gpu-service-life-can-be-surprisingly-short-only-one-to-three-years-is-expected-according-to-unnamed-google-architect)), and $B$ is the batch size such as above. The network equipment embodied impacts are allocated the same way, with the share of the server used by the model, and the data center building embodied impacts are allocated with the IT electricity consumption of the request: $$ -I^{\text{e}}_{\text{request}}=\frac{\Delta T}{B \times \Delta L} \times I^{\text{e}}_{\text{server}}. +I^{\text{e}}_{\text{request}}=\frac{\Delta T}{B \times \Delta L} \times I^{\text{e}}_{\text{server}} + \frac{\Delta T}{B \times \Delta L_{\text{network}}} \times \frac{\text{GPU}}{\#\text{GPU}_{\text{installed}}} \times I^{\text{e}}_{\text{network}} + E_{\text{server}} \times F_{\text{building}}. $$ +The last two terms only apply to the GWP criteria. + !!! warning "Water consumption (WCF) impact is not modeled for the embodied phase due to a lack of data." +## Training impacts + +The impacts of the training of the model allocated to the request are estimated separately with an experimental methodology adapted from Impact'IA, see the [LLM training methodology](llm_training.md). They are reported in a dedicated `training` phase and are **not included in the total impacts** of the request. + + ## Supplemental material ### Data center configuration @@ -302,6 +355,7 @@ Assuming the **energy consumption** for AI models is done through benchmarking o **Limitations:** * We do not account for the multiple modalities of a model (only text-to-text generation). +* The energy consumption of the GPU during the pre-fill phase is not modeled explicitly, the number of input tokens only affects the generation latency (thus the server, network and embodied impacts). ### On hardware @@ -315,8 +369,9 @@ We estimate the **required infrastructure** to run the service in terms of hardw **Limitations:** * We do not account for TPUs or other type of accelerators. -* We do not account for networking or storage primitives. +* We do not account for storage primitives, nor for the network outside of the data center. * We do not account for infrastructure overheads or utilization factors. +* Network equipment and data center building embodied impacts are only accounted for GWP. ### On data centers @@ -354,6 +409,8 @@ We aim at covering the largest scope possible when assessing the environmental i - [ML.ENERGY Leaderboard](https://ml.energy/leaderboard/?__theme=light) to estimate GPU energy consumption and latency based on the model architecture and number of output tokens. - [Lees-Perasso et al. (2026)](https://librairie.ademe.fr/economie-circulaire-et-dechets/9103-analyse-de-cycle-de-vie-de-gpu-cartes-graphiques-pour-l-intelligence-artificielle.html) and [BoaviztAPI](https://github.com/Boavizta/boaviztapi) to estimate GPU and server embodied impacts and base energy consumption. - [Our World in Data](https://ourworldindata.org/), [ADEME Base Empreinte®](https://base-empreinte.ademe.fr/) and [World Resource Institute](https://www.wri.org/) for electricity mix impacts per country. +- [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) (SNCF, Wavestone, Resilio, 2026) for the pre-fill latency model based on input tokens, the network equipment and the data center building impacts. +- [Agrawal et al. (2024)](https://www.usenix.org/conference/osdi24/presentation/agrawal) for the relation between pre-fill latency and number of input tokens. ## :material-bookshelf: Citation diff --git a/docs/methodology/llm_training.md b/docs/methodology/llm_training.md new file mode 100644 index 00000000..e35fe574 --- /dev/null +++ b/docs/methodology/llm_training.md @@ -0,0 +1,130 @@ +# Environmental Impacts of LLM Training + +!!! warning "Experimental methodology" + + The training phase is reported **separately** from the usage and embodied phases of the inference and is **not included in the total impacts** of a request. The estimation relies on a top-down approach with very few public data points and should be considered as an **order of magnitude** with an uncertainty of at least one order of magnitude. + +## Introduction + +Training a model (data collection, experiments, final training run) happens once, before the model is used, but it is a prerequisite to every inference. Following the [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) methodology developed by SNCF, Wavestone and Resilio, EcoLogits estimates the impacts of the training of a model and allocates a share of them to each request, in proportion to the number of generated tokens. + +The training impacts allocated to a request, $I_{\text{request}}^{\text{t}}$, are computed as: + +$$ +I_{\text{request}}^{\text{t}} = \frac{\#T_{\text{out}}}{\#T_{\text{model}}} \times \left( I_{\text{train}} + I_{\text{R\&D}} + I_{\text{storage}} \right), +$$ + +where $\#T_{\text{out}}$ is the number of output tokens of the request, $\#T_{\text{model}}$ is the total number of tokens the model is expected to generate during its lifetime and $I_{\text{train}}$, $I_{\text{R\&D}}$ and $I_{\text{storage}}$ are the impacts of the final training run, of the research and development experiments and of the training data storage respectively. + +## Scope + +The following stages of the model life cycle are modeled, based on the [macro-estimations of Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA/tree/main/doc): + +| Stage | Modeled | Comment | +|--------------|---------|--------------------------------------------------------------------------------| +| Final training run | :material-check: | Compute estimated from the model size and release date. | +| Experiments and test runs (R&D) | :material-check: | Estimated as a multiple of the final training run. | +| Training data storage | :material-check: | Electricity consumption of the hard disk drives during the training run. | +| Data annotation | :material-close: | Negligible (below 0.1% of the inference impacts). | +| Workstations of the model developers | :material-close: | Negligible (below 0.1% of the inference impacts). | + +## Final training run + +### Compute of the training run + +The compute of the final training run, $C_{\text{train}}$ (in FLOPs), is estimated with a regression made by Impact'IA on the [Epoch AI](https://epoch.ai/data/notable-ai-models) notable models dataset. It depends on the total number of parameters of the model, $P_{\text{total}}$, and on the release date of the model, expressed as a number of days since January 1st, 2020, $D_{\text{release}}$. At equal size, recent models are trained with more compute: + +$$ +C_{\text{train}} = 10^{\alpha D_{\text{release}} + \beta} \times P_{\text{total}}^{\gamma}, +$$ + +with $\alpha = 6 \times 10^{-4}$, $\beta = 17.151$ and $\gamma = 0.541$. + +??? example "Example" + + For a 76B parameters model released on April 14, 2025 ($D_{\text{release}} = 1930$), the estimated compute is $C_{\text{train}} \approx 1.6 \times 10^{24}$ FLOPs. + +### Energy consumption of the training run + +The compute is converted into an electricity consumption with the compute efficiency of the training hardware, $\eta$ (in FLOPS per watt). We use $\eta = 1.4 \times 10^{12}$ FLOPS/W, the peak dense BF16 efficiency of an NVIDIA H100 GPU, to stay consistent with the hardware assumptions of the [inference methodology](llm_inference.md). To cover the same technical scope as the inference phase (GPUs, servers and network equipment), the GPU energy is multiplied by an infrastructure overhead ratio, $R_{\text{infra}}$, and by the PUE of the data center: + +$$ +E_{\text{train}} = \frac{C_{\text{train}}}{\eta \times 3.6 \times 10^{6}} \times R_{\text{infra}} \times \text{PUE}, \quad R_{\text{infra}} = \frac{\#\text{GPU}_{\text{installed}} \times W_{\text{GPU}} + W_{\text{server} \backslash \text{GPU}} + W_{\text{network}}}{\#\text{GPU}_{\text{installed}} \times W_{\text{GPU}}}. +$$ + +With the [server configuration of the inference methodology](llm_inference.md#modeling-server-energy-consumption) (8 GPUs of 700 W, 1.2 kW server and 161 W of network equipment), $R_{\text{infra}} \approx 1.24$. + +## Research and development experiments + +The experiments, test runs and ablations that precede the final training run are poorly documented. Impact'IA reviewed the available data (OpenAI 2024 compute spending, MiniMax and Z.ai technical reports) which suggest that the R&D compute is between 1 and 19 times the final training run. We use the conservative median value of 4: + +$$ +E_{\text{R\&D}} = 4 \times E_{\text{train}}. +$$ + +## Training data storage + +The number of training tokens is derived from the compute with the approximation of [Kaplan et al. (2020)](https://arxiv.org/abs/2001.08361), $C_{\text{train}} = 6 \times P_{\text{total}} \times \#T_{\text{train}}$. Each token is assumed to be stored as 4 characters of 32 bits, on 30 TB hard disk drives of 9.5 W used at 20%, during a 100-day training run. The resulting electricity consumption, $E_{\text{storage}}$, is negligible compared to the training run (below 0.01%). The embodied impacts of the hard disk drives are not modeled for the same reason. + +## Allocation to the request + +### Lifetime number of generated tokens + +The training impacts are shared between all the tokens the model generates during its lifetime. Since providers do not disclose these volumes, Impact'IA proposes a top-down estimation from the **AI compute capacity of the provider**, $W_{\text{provider}}$, gathered from public announcements: + +$$ +\#T_{\text{model}} = \frac{W_{\text{provider}} \times r_{\text{inference}}}{N_{\text{active}}} \times \frac{\eta \times \epsilon}{\text{PUE}} \times \frac{\Delta L_{\text{model}}}{2 \times P_{\text{active}}}, +$$ + +where: + +* $r_{\text{inference}} = 0.8$ is the share of the compute capacity dedicated to inference, +* $N_{\text{active}}$ is the number of active models of the provider, i.e. models released less than 2 years ago (dated snapshots and `latest` versions of the same model are counted once), +* $\epsilon = 0.5 \times 0.5 \times 0.7 = 0.175$ is an efficiency factor accounting for the GPU utilization, the memory-bound limitation of the decoding phase and the sharing of the GPUs, +* $\Delta L_{\text{model}} = 2$ years is the lifetime of the model, +* $2 \times P_{\text{active}}$ is the number of FLOPs required to generate one token. + +The AI compute capacities currently used are: + +| AI Provider | Compute capacity | Source | +|-------------|------------------|-------------------------------------------------------------------------------------------------------------------------------------| +| Anthropic | 1.4 GW | [wccftech.com](https://wccftech.com/openai-sets-goal-to-scale-up-ai-compute-capacity-to-a-whopping-30gw-by-2030/) | +| Google | 9 GW | [cnbc.com](https://www.cnbc.com/2025/11/21/google-must-double-ai-serving-capacity-every-6-months-to-meet-demand.html) | +| Mistral AI | 200 MW | [cnbc.com](https://www.cnbc.com/2026/03/30/mistral-ai-paris-data-center-cluster-debt-financing.html) | +| OpenAI | 2 GW | [finance.yahoo.com](https://finance.yahoo.com/sectors/technology/articles/coreweave-signs-multi-deal-anthropic-162247263.html) | + +The training phase is not reported for the other providers (Cohere, Hugging Face Hub) nor for models without a known release date, a [`training-not-modeled`](../tutorial/warnings_and_errors.md#training-not-modeled) warning is reported instead. + +### Environmental impacts + +The training location of the models being unknown, the usage impacts are computed with the **world average electricity mix**, $F_{\text{em}}^{\text{WOR}}$, and the PUE and WUE of the provider: + +$$ +I_{\text{request}}^{\text{t,u}} = \frac{\#T_{\text{out}}}{\#T_{\text{model}}} \times \left( E_{\text{train}} + E_{\text{R\&D}} + E_{\text{storage}} \right) \times F_{\text{em}}^{\text{WOR}}. +$$ + +The embodied impacts of the training infrastructure are estimated by applying the **embodied impact intensity of the inference phase** (embodied impacts per kWh of IT electricity, for each criteria) to the IT electricity consumption of the training allocated to the request. This assumes that the training and inference infrastructures have the same hardware and lifetime. + +The water consumption footprint is computed as for the [inference phase](llm_inference.md#modeling-request-water-consumption-footprint-for-usage-phase) with the world average off-site WUE. + +## Assumptions and limitations + +* The compute regression is fitted on notable models and extrapolated to all models, including small ones for which it can overestimate the training compute. +* The compute efficiency $\eta$ is a peak theoretical value. Real-world training runs reach 30% to 50% of the peak (model FLOPs utilization), which underestimates the training energy by a factor of 2 to 3. +* The lifetime number of generated tokens is the most uncertain parameter: the compute capacities are announced targets rather than actual serving capacities, the share dedicated to inference and the utilization factors are assumptions, and models are not equally used. We estimate that the resulting per-token allocation is uncertain by **at least one order of magnitude**. +* **Difference with the Impact'IA calculator**: the Impact'IA spreadsheet expresses compute capacities in kW but uses them as W when converting them into FLOPS, which divides the lifetime number of tokens by 1000 and thus multiplies the training impacts per token by 1000. EcoLogits uses consistent units. As a consequence, the training share reported by EcoLogits (typically below 1% of the inference impacts for large providers) is much lower than the one reported by the Impact'IA calculator (13% in median). This has been reported to the Impact'IA team and the parameters of the methodology may be recalibrated in the future. +* The embodied impacts of the training infrastructure are approximated with the intensity of the inference infrastructure. +* Mixture of experts models use the mean number of active parameters to compute the number of generated tokens. + +All the parameters of the training phase can be overridden when calling [`compute_llm_impacts`][impacts.llm.compute_llm_impacts], including the lifetime number of tokens (`model_lifetime_output_token_count`) if you have better data for your use case. + +## References + +- [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) methodology guide (SNCF, Wavestone, Resilio, 2026), from which the training, R&D, storage and allocation models are adapted. +- [Epoch AI](https://epoch.ai/data/notable-ai-models) notable AI models dataset used to fit the training compute regression. +- [Kaplan et al. (2020)](https://arxiv.org/abs/2001.08361) for the relation between compute, parameters and training tokens. +- [Mistral AI (2025)](https://mistral.ai/news/our-contribution-to-a-global-environmental-standard-for-ai), [Epoch AI (2025)](https://epoch.ai/data-insights/openai-compute-spend) and [Aubakirova et al. (2025)](https://openrouter.ai/state-of-ai) for the orders of magnitude of training versus inference impacts. + +## :material-scale-balance: License and credits + +The Impact'IA methodology guide and calculator are published by SNCF, Wavestone and Resilio under the [CC BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. This page is an original description of the adaptation of their methodology within EcoLogits and is licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) as the rest of the EcoLogits methodology. Please cite both Impact'IA and EcoLogits when using the training phase estimations. diff --git a/docs/tutorial/impacts.md b/docs/tutorial/impacts.md index b57951aa..69113594 100644 --- a/docs/tutorial/impacts.md +++ b/docs/tutorial/impacts.md @@ -1,6 +1,6 @@ # Environmental Impacts -Environmental impacts are reported for each request in the **[`ImpactsOutput`][tracers.utils.ImpactsOutput]** that features multiple [impact criteria](#impact-criteria) such as [energy consumption](#energy) or the [global warming potential](#global-warming-potential-gwp) per phase ([usage](#usage) or [embodied](#embodied)) as well as the total impacts. It also contains potential [warnings and errors](warnings_and_errors.md) that can occur during the calculation. +Environmental impacts are reported for each request in the **[`ImpactsOutput`][tracers.utils.ImpactsOutput]** that features multiple [impact criteria](#impact-criteria) such as [energy consumption](#energy) or the [global warming potential](#global-warming-potential-gwp) per phase ([usage](#usage), [embodied](#embodied) or [training](#training)) as well as the total impacts. It also contains potential [warnings and errors](warnings_and_errors.md) that can occur during the calculation. !!! note "To learn more on how we estimate the environmental impacts and what are our hypotheses go to the [methodology](../methodology/index.md) section." @@ -11,7 +11,7 @@ The [`ImpactsOutput`][tracers.utils.ImpactsOutput] is structured the following w ```python from ecologits.tracers.utils import ImpactsOutput -from ecologits.impacts.modeling import ADPe, Embodied, Energy, GWP, PE, Usage +from ecologits.impacts.modeling import ADPe, Embodied, Energy, GWP, PE, Training, Usage, WCF ImpactsOutput( @@ -19,25 +19,35 @@ ImpactsOutput( gwp=GWP(), # Total global warming potential (or GHG emissions) adpe=ADPe(), # Total abiotic resource depletion pe=PE(), # Total energy consumed from primary sources + wcf=WCF(), # Total water consumption (usage phase only) usage=Usage( # (1)! energy=Energy(), gwp=GWP(), adpe=ADPe(), pe=PE(), + wcf=WCF(), ), embodied=Embodied( # (2)! gwp=GWP(), adpe=ADPe(), pe=PE(), ), - warnings=None, # (3)! + training=Training( # (3)! + energy=Energy(), + gwp=GWP(), + adpe=ADPe(), + pe=PE(), + wcf=WCF(), + ), + warnings=None, # (4)! errors=None ) ``` 1. Usage impacts for the electricity consumption impacts. Note that the energy is equal to the "total" energy impact. 2. Embodied impacts for resource extract, manufacturing and transportation of hardware components allocated to the request. -3. List of [`WarningMessage`][status_messages.WarningMessage] and [`ErrorMessage`][status_messages.ErrorMessage]. +3. Training impacts allocated to the request (experimental). They are **not included in the total impacts** and are `None` when they cannot be estimated. +4. List of [`WarningMessage`][status_messages.WarningMessage] and [`ErrorMessage`][status_messages.ErrorMessage]. ### Example of an Impact Value @@ -187,6 +197,30 @@ The [Embodied][impacts.modeling.Embodied] phase accounts for the upstream enviro docstring_section_style: list +### Training + +The [Training][impacts.modeling.Training] phase accounts for the impacts of the training of the model (final training run, research and development experiments and training data storage) allocated to the request in proportion to the number of generated tokens. This phase is **experimental** and relies on a top-down estimation adapted from the [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) methodology, see the [LLM training methodology](../methodology/llm_training.md) for the assumptions and limitations. + +The training impacts are **reported separately and are not included in the total impacts** (`energy`, `gwp`, `adpe`, `pe` and `wcf`) which only cover the usage and embodied phases of the inference. When the training impacts cannot be estimated (unknown model release date or provider compute capacity) the `training` field is `None` and a [`training-not-modeled`](warnings_and_errors.md#training-not-modeled) warning is reported. + +```python +>>> response.impacts.training.gwp.value +0.0004 # GHG emissions of the training allocated to the request in kgCO2eq. + +>>> total_with_training = response.impacts.gwp + response.impacts.training.gwp # (1)! +``` + +1. Impacts can be added together when they have the same type (criteria), including `RangeValue` intervals. + +??? note "Training model attributes" + ::: impacts.modeling.Training + options: + show_root_toc_entry: false + show_bases: false + show_docstring_description: false + docstring_section_style: list + + ## Impact Factors We use impact factors to quantify environmental harm from human activities, measuring the ratio of greenhouse gases, resource consumption, and other criteria resulting from activities like energy consumption, industrial processes, transportation, waster management and more. diff --git a/docs/tutorial/warnings_and_errors.md b/docs/tutorial/warnings_and_errors.md index ba9a99db..aa0b5a5a 100644 --- a/docs/tutorial/warnings_and_errors.md +++ b/docs/tutorial/warnings_and_errors.md @@ -51,6 +51,10 @@ This warning is reported when the selected electricity mix uses the world defaul This warning is reported when the selected electricity mix uses the world default value for the WUE impact factor. This can happen for countries where GWP data is available, but country-specific WUE data is not. +### `training-not-modeled` + +This warning is reported when the [training impacts](../methodology/llm_training.md) cannot be estimated for the request. This happens when the release date of the model is unknown or when the AI compute capacity of the provider is unknown (Cohere, Hugging Face Hub). The `training` field of the [`ImpactsOutput`][tracers.utils.ImpactsOutput] is then `None`, the other impacts are not affected. + ## Errors diff --git a/ecologits/data/models.json b/ecologits/data/models.json index c415ee31..595903a2 100644 --- a/ecologits/data/models.json +++ b/ecologits/data/models.json @@ -3349,7 +3349,8 @@ "deployment": { "tps": 57.2, "ttft": 0.21 - } + }, + "release_date": "2024-05-01" }, { "type": "model", @@ -3367,7 +3368,8 @@ "deployment": { "tps": 57.2, "ttft": 0.21 - } + }, + "release_date": "2024-11-01" }, { "type": "model", @@ -3385,7 +3387,8 @@ "deployment": { "tps": 39.4, "ttft": 1.04 - } + }, + "release_date": "2024-12-01" }, { "type": "model", @@ -3403,7 +3406,8 @@ "deployment": { "tps": 57.2, "ttft": 0.21 - } + }, + "release_date": "2025-01-01" }, { "type": "model", @@ -3421,7 +3425,8 @@ "deployment": { "tps": 57.2, "ttft": 0.21 - } + }, + "release_date": "2025-08-01" }, { "type": "model", @@ -3439,7 +3444,8 @@ "deployment": { "tps": 57.2, "ttft": 0.21 - } + }, + "release_date": "2025-08-01" }, { "type": "model", @@ -3456,7 +3462,8 @@ "deployment": { "tps": 39.4, "ttft": 1.04 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3473,7 +3480,8 @@ "deployment": { "tps": 39.4, "ttft": 1.04 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3495,7 +3503,8 @@ "deployment": { "tps": 39.4, "ttft": 1.04 - } + }, + "release_date": "2025-07-10" }, { "type": "model", @@ -3517,7 +3526,8 @@ "deployment": { "tps": 48.1, "ttft": 1.83 - } + }, + "release_date": "2025-09-01" }, { "type": "model", @@ -3539,7 +3549,8 @@ "deployment": { "tps": 34.2, "ttft": 0.47 - } + }, + "release_date": "2025-09-01" }, { "type": "model", @@ -3556,7 +3567,8 @@ "deployment": { "tps": 98.8, "ttft": 0.29 - } + }, + "release_date": "2025-09-01" }, { "type": "model", @@ -3573,7 +3585,8 @@ "deployment": { "tps": 55.9, "ttft": 0.36 - } + }, + "release_date": "2025-09-01" }, { "type": "model", @@ -3590,7 +3603,8 @@ "deployment": { "tps": 48.1, "ttft": 0.27 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3607,7 +3621,8 @@ "deployment": { "tps": 48.1, "ttft": 0.27 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3625,7 +3640,8 @@ "deployment": { "tps": 35.0, "ttft": 0.2 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3643,7 +3659,8 @@ "deployment": { "tps": 35.0, "ttft": 0.2 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3661,7 +3678,8 @@ "deployment": { "tps": 42.4, "ttft": 0.23 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3679,7 +3697,8 @@ "deployment": { "tps": 42.4, "ttft": 0.23 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3697,7 +3716,8 @@ "deployment": { "tps": 41.5, "ttft": 0.68 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3715,7 +3735,8 @@ "deployment": { "tps": 35.6, "ttft": 0.54 - } + }, + "release_date": "2025-12-01" }, { "type": "model", @@ -3737,7 +3758,8 @@ "deployment": { "tps": 48.1, "ttft": 1.83 - } + }, + "release_date": "2023-12-11" }, { "type": "model", @@ -3759,7 +3781,8 @@ "deployment": { "tps": 34.5, "ttft": 0.47 - } + }, + "release_date": "2025-05-01" }, { "type": "model", @@ -3781,7 +3804,8 @@ "deployment": { "tps": 34.5, "ttft": 0.47 - } + }, + "release_date": "2025-08-01" }, { "type": "model", @@ -3801,7 +3825,8 @@ "deployment": { "tps": 34.5, "ttft": 0.47 - } + }, + "release_date": "2026-04-01" }, { "type": "model", @@ -3821,7 +3846,8 @@ "deployment": { "tps": 34.5, "ttft": 0.47 - } + }, + "release_date": "2025-05-07" }, { "type": "model", @@ -3841,7 +3867,8 @@ "deployment": { "tps": 48.1, "ttft": 1.83 - } + }, + "release_date": "2026-04-01" }, { "type": "model", @@ -3861,7 +3888,8 @@ "deployment": { "tps": 48.1, "ttft": 1.83 - } + }, + "release_date": "2026-04-01" }, { "type": "model", @@ -3883,7 +3911,8 @@ "deployment": { "tps": 34.5, "ttft": 0.47 - } + }, + "release_date": "2026-04-01" }, { "type": "model", @@ -3901,7 +3930,8 @@ "deployment": { "tps": 116.9, "ttft": 0.46 - } + }, + "release_date": "2025-06-01" }, { "type": "model", @@ -3922,7 +3952,8 @@ "deployment": { "tps": 116.9, "ttft": 0.46 - } + }, + "release_date": "2026-03-01" }, { "type": "model", @@ -3943,7 +3974,8 @@ "deployment": { "tps": 98.8, "ttft": 0.29 - } + }, + "release_date": "2026-03-01" }, { "type": "model", @@ -3962,7 +3994,8 @@ "deployment": { "tps": 98.8, "ttft": 0.29 - } + }, + "release_date": "2024-07-01" }, { "type": "model", @@ -3981,7 +4014,8 @@ "deployment": { "tps": 98.8, "ttft": 0.29 - } + }, + "release_date": "2024-07-01" }, { "type": "model", @@ -3999,7 +4033,8 @@ "deployment": { "tps": 98.8, "ttft": 0.29 - } + }, + "release_date": "2023-09-27" }, { "type": "model", @@ -4017,7 +4052,8 @@ "deployment": { "tps": 81.0, "ttft": 0.3 - } + }, + "release_date": "2024-07-18" }, { "type": "model", @@ -4035,7 +4071,8 @@ "deployment": { "tps": 81.0, "ttft": 0.3 - } + }, + "release_date": "2024-07-18" }, { "type": "model", @@ -4055,7 +4092,8 @@ "deployment": { "tps": 98.5, "ttft": 0.39 - } + }, + "release_date": "2024-09-17" }, { "type": "model", @@ -4075,7 +4113,8 @@ "deployment": { "tps": 48.1, "ttft": 0.27 - } + }, + "release_date": "2024-09-17" }, { "type": "model", @@ -4095,7 +4134,8 @@ "deployment": { "tps": 41.5, "ttft": 0.68 - } + }, + "release_date": "2024-09-17" }, { "type": "model", @@ -4114,7 +4154,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2025-07-01" }, { "type": "model", @@ -4133,7 +4174,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2025-09-01" }, { "type": "model", @@ -4152,7 +4194,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2026-02-01" }, { "type": "model", @@ -4171,7 +4214,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2026-02-01" }, { "type": "model", @@ -4190,7 +4234,8 @@ "deployment": { "tps": 79.9, "ttft": 0.31 - } + }, + "release_date": "2025-07-01" }, { "type": "model", @@ -4209,7 +4254,8 @@ "deployment": { "tps": 98.8, "ttft": 0.29 - } + }, + "release_date": "2025-07-15" }, { "type": "model", @@ -4234,7 +4280,8 @@ "deployment": { "tps": 12.1, "ttft": 1.74 - } + }, + "release_date": "2025-08-05" }, { "type": "model", @@ -4259,7 +4306,8 @@ "deployment": { "tps": 15.9, "ttft": 1.99 - } + }, + "release_date": "2025-05-22" }, { "type": "model", @@ -4284,7 +4332,8 @@ "deployment": { "tps": 40.4, "ttft": 0.94 - } + }, + "release_date": "2025-05-22" }, { "type": "model", @@ -4309,7 +4358,8 @@ "deployment": { "tps": 32.2, "ttft": 1.33 - } + }, + "release_date": "2025-09-29" }, { "type": "model", @@ -4334,7 +4384,8 @@ "deployment": { "tps": 41.4, "ttft": 1.56 - } + }, + "release_date": "2026-02-17" }, { "type": "model", @@ -4356,7 +4407,8 @@ "deployment": { "tps": 71.1, "ttft": 0.67 - } + }, + "release_date": "2025-10-15" }, { "type": "model", @@ -4381,7 +4433,8 @@ "deployment": { "tps": 39.6, "ttft": 1.56 - } + }, + "release_date": "2025-11-24" }, { "type": "model", @@ -4406,7 +4459,8 @@ "deployment": { "tps": 37.0, "ttft": 2.19 - } + }, + "release_date": "2026-02-05" }, { "type": "model", @@ -4456,7 +4510,8 @@ "deployment": { "tps": 58.0, "ttft": 1.63 - } + }, + "release_date": "2026-05-28" }, { "type": "model", @@ -4482,7 +4537,8 @@ "deployment": { "tps": 64.6, "ttft": 1.1 - } + }, + "release_date": "2025-02-05" }, { "type": "model", @@ -4508,7 +4564,8 @@ "deployment": { "tps": 64.6, "ttft": 1.1 - } + }, + "release_date": "2025-02-05" }, { "type": "model", @@ -4531,7 +4588,8 @@ "deployment": { "tps": 101.0, "ttft": 0.5 - } + }, + "release_date": "2025-02-25" }, { "type": "model", @@ -4554,7 +4612,8 @@ "deployment": { "tps": 101.0, "ttft": 0.5 - } + }, + "release_date": "2025-02-25" }, { "type": "model", @@ -4580,7 +4639,8 @@ "deployment": { "tps": 64.6, "ttft": 1.1 - } + }, + "release_date": "2025-06-17" }, { "type": "model", @@ -4606,7 +4666,8 @@ "deployment": { "tps": 158.5, "ttft": 7.2 - } + }, + "release_date": "2025-08-26" }, { "type": "model", @@ -4629,7 +4690,8 @@ "deployment": { "tps": 101.0, "ttft": 0.5 - } + }, + "release_date": "2025-07-22" }, { "type": "model", @@ -4655,7 +4717,8 @@ "deployment": { "tps": 89.6, "ttft": 2.28 - } + }, + "release_date": "2025-06-17" }, { "type": "model", @@ -4681,7 +4744,8 @@ "deployment": { "tps": 73.9, "ttft": 1.24 - } + }, + "release_date": "2025-12-17" }, { "type": "model", @@ -4707,7 +4771,8 @@ "deployment": { "tps": 55.2, "ttft": 22.28 - } + }, + "release_date": "2025-11-20" }, { "type": "model", @@ -4733,7 +4798,8 @@ "deployment": { "tps": 73.0, "ttft": 4.32 - } + }, + "release_date": "2025-11-18" }, { "type": "model", @@ -4831,7 +4897,8 @@ "deployment": { "tps": 73.0, "ttft": 4.32 - } + }, + "release_date": "2026-02-19" }, { "type": "model", @@ -4857,7 +4924,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2026-02-19" }, { "type": "model", @@ -4885,7 +4953,8 @@ "deployment": { "tps": 130.6, "ttft": 2.72 - } + }, + "release_date": "2026-05-19" }, { "type": "model", @@ -5055,7 +5124,8 @@ "deployment": { "tps": 44.3, "ttft": 0.43 - } + }, + "release_date": "2023-03-01" }, { "type": "model", @@ -5077,7 +5147,8 @@ "deployment": { "tps": 44.3, "ttft": 0.43 - } + }, + "release_date": "2024-01-25" }, { "type": "model", @@ -5099,7 +5170,8 @@ "deployment": { "tps": 44.3, "ttft": 0.43 - } + }, + "release_date": "2023-11-06" }, { "type": "model", @@ -5121,7 +5193,8 @@ "deployment": { "tps": 37.2, "ttft": 0.44 - } + }, + "release_date": "2023-06-13" }, { "type": "model", @@ -5143,7 +5216,8 @@ "deployment": { "tps": 79.4, "ttft": 0.73 - } + }, + "release_date": "2023-09-14" }, { "type": "model", @@ -5165,7 +5239,8 @@ "deployment": { "tps": 79.4, "ttft": 0.73 - } + }, + "release_date": "2023-09-14" }, { "type": "model", @@ -5191,7 +5266,8 @@ "deployment": { "tps": 22.4, "ttft": 0.85 - } + }, + "release_date": "2023-03-14" }, { "type": "model", @@ -5217,7 +5293,8 @@ "deployment": { "tps": 22.4, "ttft": 0.85 - } + }, + "release_date": "2023-06-13" }, { "type": "model", @@ -5243,7 +5320,8 @@ "deployment": { "tps": 14.9, "ttft": 1.17 - } + }, + "release_date": "2024-04-09" }, { "type": "model", @@ -5269,7 +5347,8 @@ "deployment": { "tps": 14.9, "ttft": 1.17 - } + }, + "release_date": "2024-04-09" }, { "type": "model", @@ -5295,7 +5374,8 @@ "deployment": { "tps": 42.5, "ttft": 0.63 - } + }, + "release_date": "2025-04-14" }, { "type": "model", @@ -5321,7 +5401,8 @@ "deployment": { "tps": 42.5, "ttft": 0.63 - } + }, + "release_date": "2025-04-14" }, { "type": "model", @@ -5344,7 +5425,8 @@ "deployment": { "tps": 34.4, "ttft": 0.69 - } + }, + "release_date": "2025-04-14" }, { "type": "model", @@ -5367,7 +5449,8 @@ "deployment": { "tps": 42.5, "ttft": 0.63 - } + }, + "release_date": "2025-04-14" }, { "type": "model", @@ -5390,7 +5473,8 @@ "deployment": { "tps": 26.1, "ttft": 0.48 - } + }, + "release_date": "2025-04-14" }, { "type": "model", @@ -5413,7 +5497,8 @@ "deployment": { "tps": 42.5, "ttft": 0.63 - } + }, + "release_date": "2025-04-14" }, { "type": "model", @@ -5439,7 +5524,8 @@ "deployment": { "tps": 46.2, "ttft": 0.53 - } + }, + "release_date": "2024-05-13" }, { "type": "model", @@ -5465,7 +5551,8 @@ "deployment": { "tps": 47.8, "ttft": 0.35 - } + }, + "release_date": "2024-05-13" }, { "type": "model", @@ -5491,7 +5578,8 @@ "deployment": { "tps": 29.1, "ttft": 1.23 - } + }, + "release_date": "2024-08-06" }, { "type": "model", @@ -5517,7 +5605,8 @@ "deployment": { "tps": 77.8, "ttft": 0.43 - } + }, + "release_date": "2024-11-20" }, { "type": "model", @@ -5540,7 +5629,8 @@ "deployment": { "tps": 33.0, "ttft": 0.6 - } + }, + "release_date": "2024-07-18" }, { "type": "model", @@ -5563,7 +5653,8 @@ "deployment": { "tps": 45.5, "ttft": 0.58 - } + }, + "release_date": "2024-07-18" }, { "type": "model", @@ -5586,7 +5677,8 @@ "deployment": { "tps": 47.1, "ttft": 1.82 - } + }, + "release_date": "2025-03-11" }, { "type": "model", @@ -5609,7 +5701,8 @@ "deployment": { "tps": 47.1, "ttft": 1.82 - } + }, + "release_date": "2025-03-11" }, { "type": "model", @@ -5632,7 +5725,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2025-03-20" }, { "type": "model", @@ -5655,7 +5749,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2025-03-20" }, { "type": "model", @@ -5678,7 +5773,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2025-12-15" }, { "type": "model", @@ -5701,7 +5797,8 @@ "deployment": { "tps": 32.9, "ttft": 0.6 - } + }, + "release_date": "2025-03-20" }, { "type": "model", @@ -5724,7 +5821,8 @@ "deployment": { "tps": 32.9, "ttft": 0.6 - } + }, + "release_date": "2025-03-20" }, { "type": "model", @@ -5747,7 +5845,8 @@ "deployment": { "tps": null, "ttft": null - } + }, + "release_date": "2025-12-15" }, { "type": "model", @@ -5773,7 +5872,8 @@ "deployment": { "tps": 46.0, "ttft": 2.17 - } + }, + "release_date": "2025-03-11" }, { "type": "model", @@ -5799,7 +5899,8 @@ "deployment": { "tps": 46.0, "ttft": 2.17 - } + }, + "release_date": "2025-03-11" }, { "type": "model", @@ -5825,7 +5926,8 @@ "deployment": { "tps": 53.6, "ttft": 1.57 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -5851,7 +5953,8 @@ "deployment": { "tps": 43.8, "ttft": 2.29 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -5877,7 +5980,8 @@ "deployment": { "tps": 73.0, "ttft": 1.01 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -5903,7 +6007,8 @@ "deployment": { "tps": 53.1, "ttft": 2.32 - } + }, + "release_date": "2025-09-15" }, { "type": "model", @@ -5926,7 +6031,8 @@ "deployment": { "tps": 71.0, "ttft": 4.28 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -5949,7 +6055,8 @@ "deployment": { "tps": 53.6, "ttft": 1.57 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -5972,7 +6079,8 @@ "deployment": { "tps": 96.1, "ttft": 2.29 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -5995,7 +6103,8 @@ "deployment": { "tps": 53.6, "ttft": 1.57 - } + }, + "release_date": "2025-08-07" }, { "type": "model", @@ -6021,7 +6130,8 @@ "deployment": { "tps": 12.2, "ttft": 59.54 - } + }, + "release_date": "2025-10-06" }, { "type": "model", @@ -6047,7 +6157,8 @@ "deployment": { "tps": 12.2, "ttft": 59.54 - } + }, + "release_date": "2025-10-06" }, { "type": "model", @@ -6073,7 +6184,8 @@ "deployment": { "tps": 53.6, "ttft": 1.57 - } + }, + "release_date": "2025-10-14" }, { "type": "model", @@ -6099,7 +6211,8 @@ "deployment": { "tps": 53.6, "ttft": 1.57 - } + }, + "release_date": "2025-10-14" }, { "type": "model", @@ -6125,7 +6238,8 @@ "deployment": { "tps": 60.6, "ttft": 1.85 - } + }, + "release_date": "2025-11-13" }, { "type": "model", @@ -6151,7 +6265,8 @@ "deployment": { "tps": 60.6, "ttft": 1.85 - } + }, + "release_date": "2025-11-13" }, { "type": "model", @@ -6177,7 +6292,8 @@ "deployment": { "tps": 73.0, "ttft": 1.01 - } + }, + "release_date": "2025-11-13" }, { "type": "model", @@ -6203,7 +6319,8 @@ "deployment": { "tps": 59.1, "ttft": 2.15 - } + }, + "release_date": "2025-11-13" }, { "type": "model", @@ -6229,7 +6346,8 @@ "deployment": { "tps": 83.9, "ttft": 2.68 - } + }, + "release_date": "2025-11-19" }, { "type": "model", @@ -6252,7 +6370,8 @@ "deployment": { "tps": 71.6, "ttft": 1.7 - } + }, + "release_date": "2025-11-13" }, { "type": "model", @@ -6278,7 +6397,8 @@ "deployment": { "tps": 43.8, "ttft": 2.29 - } + }, + "release_date": "2025-12-11" }, { "type": "model", @@ -6304,7 +6424,8 @@ "deployment": { "tps": 43.8, "ttft": 2.29 - } + }, + "release_date": "2025-12-11" }, { "type": "model", @@ -6330,7 +6451,8 @@ "deployment": { "tps": 73.0, "ttft": 1.01 - } + }, + "release_date": "2025-12-11" }, { "type": "model", @@ -6356,7 +6478,8 @@ "deployment": { "tps": 58.8, "ttft": 2.04 - } + }, + "release_date": "2025-12-18" }, { "type": "model", @@ -6382,7 +6505,8 @@ "deployment": { "tps": 8.1, "ttft": 15.82 - } + }, + "release_date": "2025-12-11" }, { "type": "model", @@ -6408,7 +6532,8 @@ "deployment": { "tps": 8.1, "ttft": 15.82 - } + }, + "release_date": "2025-12-11" }, { "type": "model", @@ -6460,7 +6585,8 @@ "deployment": { "tps": 48.8, "ttft": 3.01 - } + }, + "release_date": "2026-02-05" }, { "type": "model", @@ -6486,7 +6612,8 @@ "deployment": { "tps": 45.5, "ttft": 1.01 - } + }, + "release_date": "2026-03-05" }, { "type": "model", @@ -6512,7 +6639,8 @@ "deployment": { "tps": 45.5, "ttft": 1.01 - } + }, + "release_date": "2026-03-05" }, { "type": "model", @@ -6535,7 +6663,8 @@ "deployment": { "tps": 73.0, "ttft": 0.67 - } + }, + "release_date": "2026-03-17" }, { "type": "model", @@ -6558,7 +6687,8 @@ "deployment": { "tps": 73.0, "ttft": 0.67 - } + }, + "release_date": "2026-03-17" }, { "type": "model", @@ -6581,7 +6711,8 @@ "deployment": { "tps": 70.8, "ttft": 0.54 - } + }, + "release_date": "2026-03-17" }, { "type": "model", @@ -6604,7 +6735,8 @@ "deployment": { "tps": 70.8, "ttft": 0.54 - } + }, + "release_date": "2026-03-17" }, { "type": "model", @@ -6630,7 +6762,8 @@ "deployment": { "tps": 4.0, "ttft": 26.59 - } + }, + "release_date": "2026-03-05" }, { "type": "model", @@ -6656,7 +6789,8 @@ "deployment": { "tps": 4.0, "ttft": 26.59 - } + }, + "release_date": "2026-03-05" }, { "type": "model", @@ -6684,7 +6818,8 @@ "deployment": { "tps": 40.4, "ttft": 3.38 - } + }, + "release_date": "2026-04-23" }, { "type": "model", @@ -6712,7 +6847,8 @@ "deployment": { "tps": 40.4, "ttft": 3.38 - } + }, + "release_date": "2026-04-23" }, { "type": "model", @@ -6740,7 +6876,8 @@ "deployment": { "tps": 14.4, "ttft": 46.04 - } + }, + "release_date": "2026-04-23" }, { "type": "model", @@ -6768,7 +6905,8 @@ "deployment": { "tps": 14.4, "ttft": 46.04 - } + }, + "release_date": "2026-04-23" }, { "type": "model", @@ -6794,7 +6932,8 @@ "deployment": { "tps": 136.6, "ttft": 3.2 - } + }, + "release_date": "2024-12-17" }, { "type": "model", @@ -6820,7 +6959,8 @@ "deployment": { "tps": 136.6, "ttft": 3.2 - } + }, + "release_date": "2024-12-17" }, { "type": "model", @@ -6843,7 +6983,8 @@ "deployment": { "tps": 170.6, "ttft": 4.13 - } + }, + "release_date": "2025-01-31" }, { "type": "model", @@ -6866,7 +7007,8 @@ "deployment": { "tps": 170.6, "ttft": 4.13 - } + }, + "release_date": "2025-01-31" }, { "type": "model", @@ -6889,7 +7031,8 @@ "deployment": { "tps": 95.6, "ttft": 3.93 - } + }, + "release_date": "2025-04-16" }, { "type": "model", @@ -6912,7 +7055,8 @@ "deployment": { "tps": 95.6, "ttft": 3.93 - } + }, + "release_date": "2025-04-16" }, { "type": "model", @@ -6935,7 +7079,8 @@ "deployment": { "tps": 91.9, "ttft": 17.34 - } + }, + "release_date": "2025-06-26" }, { "type": "model", @@ -6958,7 +7103,8 @@ "deployment": { "tps": 91.9, "ttft": 17.34 - } + }, + "release_date": "2025-06-26" } ] -} \ No newline at end of file +} diff --git a/ecologits/impacts/__init__.py b/ecologits/impacts/__init__.py index 6cd40d76..61daa865 100644 --- a/ecologits/impacts/__init__.py +++ b/ecologits/impacts/__init__.py @@ -1,7 +1,9 @@ from .llm import compute_llm_impacts from .modeling import Impacts +from .training import compute_llm_training_impacts_dag __all__ = [ "Impacts", - "compute_llm_impacts" + "compute_llm_impacts", + "compute_llm_training_impacts_dag" ] diff --git a/ecologits/impacts/llm.py b/ecologits/impacts/llm.py index aef8473e..4ed533ec 100644 --- a/ecologits/impacts/llm.py +++ b/ecologits/impacts/llm.py @@ -1,8 +1,10 @@ import math +from datetime import date from typing import Any, Optional, Union, cast from ecologits.impacts.dag import DAG -from ecologits.impacts.modeling import GWP, PE, WCF, ADPe, Embodied, Energy, Impacts, Usage +from ecologits.impacts.modeling import GWP, PE, WCF, ADPe, Embodied, Energy, Impacts, Training, Usage +from ecologits.impacts.training import compute_llm_training_impacts_dag from ecologits.utils.range_value import RangeValue, ValueOrRange MODEL_QUANTIZATION_BITS = 16 @@ -15,7 +17,15 @@ LATENCY_BETA = 0.0003119310311688259 LATENCY_GAMMA = 0.019473717579473387 +# Pre-fill latency modeled from the number of input tokens (adapted from Impact'IA, based on Agrawal et al. (2024) +# measurements of Mistral 7B on a single A100 GPU: TTFT[ms] = 0.062 * input_tokens + 17.88). +PREFILL_LATENCY_REFERENCE_COEFFICIENT = 0.062 / 1000 # s per input token (Mistral 7B, A100) +PREFILL_LATENCY_REFERENCE_TTFT = 0.44 # s, median TTFT of Mistral 7B on OpenRouter +PREFILL_LATENCY_REFERENCE_PARAMETER_COUNT = 7.3 # B parameters (Mistral 7B) +PREFILL_LATENCY_INTERCEPT = 17.88 / 1000 # s + GPU_MEMORY = 80 # GB +GPU_POWER = 0.7 # kW (NVIDIA H100 80GB SXM5) GPU_EMBODIED_IMPACT_GWP = 273 GPU_EMBODIED_IMPACT_ADPE = 0.00895 GPU_EMBODIED_IMPACT_PE = 3721 @@ -26,6 +36,35 @@ SERVER_EMBODIED_IMPACT_ADPE = 0.37 SERVER_EMBODIED_IMPACT_PE = 70000 +# Data center network equipment attached to one server (firewall, router and switch), adapted from Impact'IA. +# Power and effective usage ratios from ADEME Base Empreinte / ADEME-ARCEP-ARCOM, embodied GWP from Resilio DB +# (Fortinet FortiGate 100E firewall, Juniper PTX10002-36QDD router, Arista 7050TX3-48C8 switch). +NETWORK_FIREWALL_POWER = 0.09 # kW +NETWORK_FIREWALL_USAGE_RATIO = 0.0358 +NETWORK_FIREWALL_EMBODIED_GWP = 333.8 # kgCO2eq +NETWORK_ROUTER_POWER = 0.09 # kW +NETWORK_ROUTER_USAGE_RATIO = 0.286 +NETWORK_ROUTER_EMBODIED_GWP = 403 # kgCO2eq +NETWORK_SWITCH_POWER = 0.09 # kW +NETWORK_SWITCH_USAGE_RATIO = 1.468 +NETWORK_SWITCH_EMBODIED_GWP = 363 # kgCO2eq + +NETWORK_POWER = ( + NETWORK_FIREWALL_POWER * NETWORK_FIREWALL_USAGE_RATIO + + NETWORK_ROUTER_POWER * NETWORK_ROUTER_USAGE_RATIO + + NETWORK_SWITCH_POWER * NETWORK_SWITCH_USAGE_RATIO +) # kW, network power allocated to one server +NETWORK_EMBODIED_IMPACT_GWP = ( + NETWORK_FIREWALL_EMBODIED_GWP * NETWORK_FIREWALL_USAGE_RATIO + + NETWORK_ROUTER_EMBODIED_GWP * NETWORK_ROUTER_USAGE_RATIO + + NETWORK_SWITCH_EMBODIED_GWP * NETWORK_SWITCH_USAGE_RATIO +) # kgCO2eq, network embodied impacts allocated to one server +NETWORK_LIFESPAN = 5 * 365 * 24 * 60 * 60 # seconds + +# Embodied impacts of the data center building and technical environment (electrical and cooling equipment) +# allocated per kWh of IT electricity consumption (Hubblo, ADEME cloud PCR), adapted from Impact'IA. +DATACENTER_BUILDING_EMBODIED_GWP_FACTOR = 0.01 # kgCO2eq / kWh + HARDWARE_LIFESPAN = 3 * 365 * 24 * 60 * 60 BATCH_SIZE = 64 @@ -62,6 +101,48 @@ def gpu_energy( return output_token_count * gpu_energy_per_token +@dag.asset +def prefill_latency( + model_active_parameter_count: float, + prefill_latency_reference_coefficient: float, + prefill_latency_reference_ttft: float, + prefill_latency_reference_parameter_count: float, + prefill_latency_intercept: float, + input_token_count: Optional[float] = None, + ttft: Optional[float] = None, +) -> float: + """ + Compute the pre-fill latency (time-to-first-token) in seconds. + + When the number of input tokens is known, the pre-fill latency is modeled as a linear function of the number of + input tokens (Agrawal et al., 2024). The per-token coefficient measured for Mistral 7B is scaled by the ratio + between the median time-to-first-token of the model and the one of Mistral 7B (both measured on OpenRouter), as + proposed by Impact'IA. When the time-to-first-token of the model is unknown, the coefficient is scaled by the + number of active parameters instead. When the number of input tokens is unknown, the measured time-to-first-token + is used as is. + + Args: + model_active_parameter_count: Number of active parameters of the model (in billion). + prefill_latency_reference_coefficient: Pre-fill latency per input token of the reference model in seconds. + prefill_latency_reference_ttft: Median time-to-first-token of the reference model in seconds. + prefill_latency_reference_parameter_count: Number of active parameters of the reference model (in billion). + prefill_latency_intercept: Incompressible pre-fill latency in seconds. + input_token_count: Number of input tokens (optional). + ttft: Time-to-first-token latency of the model in seconds (optional). + + Returns: + The pre-fill latency in seconds. + """ + if input_token_count is None: + return ttft or 0 + if ttft is not None: + coefficient = prefill_latency_reference_coefficient * ttft / prefill_latency_reference_ttft + else: + coefficient = prefill_latency_reference_coefficient * model_active_parameter_count \ + / prefill_latency_reference_parameter_count + return coefficient * input_token_count + prefill_latency_intercept + + @dag.asset def generation_latency( model_active_parameter_count: float, @@ -71,8 +152,8 @@ def generation_latency( latency_beta: float, latency_gamma: float, request_latency: float, + prefill_latency: float, tps: Optional[float] = None, - ttft: Optional[float] = None, ) -> ValueOrRange: """ Compute the token generation latency in seconds. @@ -85,8 +166,8 @@ def generation_latency( latency_beta: Beta coefficient of the latency regression. latency_gamma: Gamma coefficient of the latency regression. request_latency: Measured request latency in seconds. + prefill_latency: Pre-fill latency (time-to-first-token) in seconds. tps: Number of tokens generated per second by the model. - ttft: Time-to-first-token latency in seconds. Returns: The token generation latency in seconds. @@ -95,8 +176,7 @@ def generation_latency( latency_per_token = latency_alpha * model_active_parameter_count + latency_beta * batch_size + latency_gamma else: latency_per_token = 1 / tps - latency_first_token = ttft or 0 - gpu_latency = output_token_count * latency_per_token + latency_first_token + gpu_latency = output_token_count * latency_per_token + prefill_latency if request_latency < gpu_latency: return request_latency return gpu_latency @@ -163,6 +243,33 @@ def server_energy( return (generation_latency / 3600) * server_power * (gpu_required_count / server_gpu_count) * (1 / batch_size) +@dag.asset +def network_energy( + generation_latency: float, + network_power: float, + server_gpu_count: int, + gpu_required_count: int, + batch_size: int +) -> float: + """ + Compute the energy consumption of the data center network equipment (firewall, router, switch). + + The network equipment power allocated to one server is shared between the requests the same way as the server + power (adapted from Impact'IA). + + Args: + generation_latency: Token generation latency in seconds. + network_power: Power consumption of the network equipment allocated to one server in kW. + server_gpu_count: Number of available GPUs in the server. + gpu_required_count: Number of required GPUs to load the model. + batch_size: Number of requests handled concurrently by the server. + + Returns: + The energy consumption of the network equipment in kWh. + """ + return (generation_latency / 3600) * network_power * (gpu_required_count / server_gpu_count) * (1 / batch_size) + + @dag.asset def request_energy( datacenter_pue: float, @@ -185,7 +292,8 @@ def request_energy( def request_it_energy( server_energy: float, gpu_required_count: int, - gpu_energy: ValueOrRange + gpu_energy: ValueOrRange, + network_energy: float ) -> ValueOrRange: """ Compute the IT energy consumption of the request before data center overhead. @@ -194,11 +302,12 @@ def request_it_energy( server_energy: Energy consumption of the server in kWh. gpu_required_count: Number of required GPUs to load the model. gpu_energy: Energy consumption of a single GPU in kWh. + network_energy: Energy consumption of the network equipment in kWh. Returns: The IT energy consumption of the request in kWh. """ - return server_energy + gpu_required_count * gpu_energy + return server_energy + gpu_required_count * gpu_energy + network_energy @dag.asset @@ -343,14 +452,14 @@ def server_gpu_embodied_pe( @dag.asset -def request_embodied_gwp( +def request_hardware_embodied_gwp( server_gpu_embodied_gwp: float, server_lifetime: float, generation_latency: ValueOrRange, batch_size: int ) -> ValueOrRange: """ - Compute the Global Warming Potential (GWP) embodied impact of the request. + Compute the Global Warming Potential (GWP) embodied impact of the server and GPUs allocated to the request. Args: server_gpu_embodied_gwp: GWP embodied impact of the server and the GPUs in kgCO2eq. @@ -359,11 +468,83 @@ def request_embodied_gwp( batch_size: Number of requests handled concurrently by the server. Returns: - The GWP embodied impact of the request in kgCO2eq. + The GWP embodied impact of the server and GPUs allocated to the request in kgCO2eq. """ return generation_latency * server_gpu_embodied_gwp / (server_lifetime * batch_size) +@dag.asset +def request_network_embodied_gwp( + network_embodied_gwp: float, + network_lifetime: float, + generation_latency: ValueOrRange, + server_gpu_count: int, + gpu_required_count: int, + batch_size: int +) -> ValueOrRange: + """ + Compute the Global Warming Potential (GWP) embodied impact of the network equipment allocated to the request. + + The network equipment embodied impacts allocated to one server are shared between the requests the same way as + the server embodied impacts (adapted from Impact'IA). + + Args: + network_embodied_gwp: GWP embodied impact of the network equipment allocated to one server in kgCO2eq. + network_lifetime: Lifetime duration of the network equipment in seconds. + generation_latency: Token generation latency in seconds. + server_gpu_count: Number of available GPUs in the server. + gpu_required_count: Number of required GPUs to load the model. + batch_size: Number of requests handled concurrently by the server. + + Returns: + The GWP embodied impact of the network equipment allocated to the request in kgCO2eq. + """ + return generation_latency * network_embodied_gwp * (gpu_required_count / server_gpu_count) \ + / (network_lifetime * batch_size) + + +@dag.asset +def request_datacenter_embodied_gwp( + request_it_energy: ValueOrRange, + datacenter_building_embodied_gwp_factor: float +) -> ValueOrRange: + """ + Compute the Global Warming Potential (GWP) embodied impact of the data center building and technical environment + (electrical and cooling equipment) allocated to the request (adapted from Impact'IA). + + Args: + request_it_energy: IT energy consumption of the request in kWh. + datacenter_building_embodied_gwp_factor: GWP embodied impact of the data center building and technical + environment per kWh of IT energy consumption in kgCO2eq / kWh. + + Returns: + The GWP embodied impact of the data center building allocated to the request in kgCO2eq. + """ + return request_it_energy * datacenter_building_embodied_gwp_factor + + +@dag.asset +def request_embodied_gwp( + request_hardware_embodied_gwp: ValueOrRange, + request_network_embodied_gwp: ValueOrRange, + request_datacenter_embodied_gwp: ValueOrRange +) -> ValueOrRange: + """ + Compute the Global Warming Potential (GWP) embodied impact of the request. + + Args: + request_hardware_embodied_gwp: GWP embodied impact of the server and GPUs allocated to the request in kgCO2eq. + request_network_embodied_gwp: GWP embodied impact of the network equipment allocated to the request in + kgCO2eq. + request_datacenter_embodied_gwp: GWP embodied impact of the data center building allocated to the request in + kgCO2eq. + + Returns: + The GWP embodied impact of the request in kgCO2eq. + """ + return request_hardware_embodied_gwp + request_network_embodied_gwp + request_datacenter_embodied_gwp + + @dag.asset def request_embodied_adpe( server_gpu_embodied_adpe: float, @@ -374,6 +555,9 @@ def request_embodied_adpe( """ Compute the Abiotic Depletion Potential for Elements (ADPe) embodied impact of the request. + Note: + Network equipment and data center building embodied impacts are not modeled for ADPe due to a lack of data. + Args: server_gpu_embodied_adpe: ADPe embodied impact of the server and the GPUs in kgSbeq. server_lifetime: Lifetime duration of the server in seconds. @@ -396,6 +580,9 @@ def request_embodied_pe( """ Compute the Primary Energy (PE) embodied impact of the request. + Note: + Network equipment and data center building embodied impacts are not modeled for PE due to a lack of data. + Args: server_gpu_embodied_pe: PE embodied impact of the server and the GPUs in MJ. server_lifetime: Lifetime duration of the server in seconds. @@ -419,6 +606,7 @@ def compute_llm_impacts_dag( if_electricity_mix_wue: float, datacenter_pue: ValueOrRange, datacenter_wue: ValueOrRange, + input_token_count: Optional[float] = None, model_quantization_bits: Optional[int] = MODEL_QUANTIZATION_BITS, gpu_energy_alpha: Optional[float] = GPU_ENERGY_ALPHA, gpu_energy_beta: Optional[float] = GPU_ENERGY_BETA, @@ -426,6 +614,10 @@ def compute_llm_impacts_dag( latency_alpha: Optional[float] = LATENCY_ALPHA, latency_beta: Optional[float] = LATENCY_BETA, latency_gamma: Optional[float] = LATENCY_GAMMA, + prefill_latency_reference_coefficient: Optional[float] = PREFILL_LATENCY_REFERENCE_COEFFICIENT, + prefill_latency_reference_ttft: Optional[float] = PREFILL_LATENCY_REFERENCE_TTFT, + prefill_latency_reference_parameter_count: Optional[float] = PREFILL_LATENCY_REFERENCE_PARAMETER_COUNT, + prefill_latency_intercept: Optional[float] = PREFILL_LATENCY_INTERCEPT, gpu_memory: Optional[float] = GPU_MEMORY, gpu_embodied_gwp: Optional[float] = GPU_EMBODIED_IMPACT_GWP, gpu_embodied_adpe: Optional[float] = GPU_EMBODIED_IMPACT_ADPE, @@ -436,6 +628,10 @@ def compute_llm_impacts_dag( server_embodied_adpe: Optional[float] = SERVER_EMBODIED_IMPACT_ADPE, server_embodied_pe: Optional[float] = SERVER_EMBODIED_IMPACT_PE, server_lifetime: Optional[float] = HARDWARE_LIFESPAN, + network_power: Optional[float] = NETWORK_POWER, + network_embodied_gwp: Optional[float] = NETWORK_EMBODIED_IMPACT_GWP, + network_lifetime: Optional[float] = NETWORK_LIFESPAN, + datacenter_building_embodied_gwp_factor: Optional[float] = DATACENTER_BUILDING_EMBODIED_GWP_FACTOR, batch_size: Optional[float] = BATCH_SIZE, tps: Optional[float] = None, ttft: Optional[float] = None, @@ -454,6 +650,7 @@ def compute_llm_impacts_dag( if_electricity_mix_wue: WCF impact factor of electricity consumption in L / kWh. datacenter_wue: Water Usage Effectiveness of the data center in L/kWh. datacenter_pue: Power Usage Effectiveness of the data center. + input_token_count: Number of input tokens (optional, used to estimate the pre-fill latency). model_quantization_bits: Number of bits used to represent the model weights. gpu_energy_alpha: Alpha coefficient of the "GPU energy" regression. gpu_energy_beta: Beta coefficient of the "GPU energy" regression. @@ -461,6 +658,10 @@ def compute_llm_impacts_dag( latency_alpha: Alpha coefficient of the "Latency" regression. latency_beta: Beta coefficient of the "Latency" regression. latency_gamma: Gamma coefficient of the "Latency" regression. + prefill_latency_reference_coefficient: Pre-fill latency per input token of the reference model in seconds. + prefill_latency_reference_ttft: Median time-to-first-token of the reference model in seconds. + prefill_latency_reference_parameter_count: Number of active parameters of the reference model (in billion). + prefill_latency_intercept: Incompressible pre-fill latency in seconds. gpu_memory: Amount of memory available on a single GPU. gpu_embodied_gwp: GWP embodied impact of a single GPU. gpu_embodied_adpe: ADPe embodied impact of a single GPU. @@ -471,6 +672,11 @@ def compute_llm_impacts_dag( server_embodied_adpe: ADPe embodied impact of the server in kgSbeq. server_embodied_pe: PE embodied impact of the server in MJ. server_lifetime: Lifetime duration of the server in seconds. + network_power: Power consumption of the network equipment allocated to one server in kW. + network_embodied_gwp: GWP embodied impact of the network equipment allocated to one server in kgCO2eq. + network_lifetime: Lifetime duration of the network equipment in seconds. + datacenter_building_embodied_gwp_factor: GWP embodied impact of the data center building per kWh of IT + energy in kgCO2eq / kWh. batch_size: Number of requests handled concurrently by the server. tps: Number of tokens generated per second by the model (optional). ttft: Time-to-first-token latency in seconds (optional). @@ -482,6 +688,7 @@ def compute_llm_impacts_dag( model_total_parameter_count=model_total_parameter_count, model_quantization_bits=model_quantization_bits, output_token_count=output_token_count, + input_token_count=input_token_count, request_latency=request_latency, if_electricity_mix_gwp=if_electricity_mix_gwp, if_electricity_mix_adpe=if_electricity_mix_adpe, @@ -495,6 +702,10 @@ def compute_llm_impacts_dag( latency_alpha=latency_alpha, latency_beta=latency_beta, latency_gamma=latency_gamma, + prefill_latency_reference_coefficient=prefill_latency_reference_coefficient, + prefill_latency_reference_ttft=prefill_latency_reference_ttft, + prefill_latency_reference_parameter_count=prefill_latency_reference_parameter_count, + prefill_latency_intercept=prefill_latency_intercept, gpu_memory=gpu_memory, gpu_embodied_gwp=gpu_embodied_gwp, gpu_embodied_adpe=gpu_embodied_adpe, @@ -505,6 +716,10 @@ def compute_llm_impacts_dag( server_embodied_adpe=server_embodied_adpe, server_embodied_pe=server_embodied_pe, server_lifetime=server_lifetime, + network_power=network_power, + network_embodied_gwp=network_embodied_gwp, + network_lifetime=network_lifetime, + datacenter_building_embodied_gwp_factor=datacenter_building_embodied_gwp_factor, batch_size=batch_size, tps=tps, ttft=ttft, @@ -512,6 +727,62 @@ def compute_llm_impacts_dag( return results +def _default(value: Optional[float], default: float) -> float: + return default if value is None else value + + +def _parameter_bounds( + model_active_parameter_count: ValueOrRange, + model_total_parameter_count: ValueOrRange +) -> tuple[list[ValueOrRange], list[ValueOrRange]]: + """ + Get the (min, max) bounds of the model parameters when at least one of them is a range, else the values as is. + """ + if not isinstance(model_active_parameter_count, RangeValue) \ + and not isinstance(model_total_parameter_count, RangeValue): + return [model_active_parameter_count], [model_total_parameter_count] + if isinstance(model_active_parameter_count, RangeValue): + active_params = [model_active_parameter_count.min, model_active_parameter_count.max] + else: + active_params = [model_active_parameter_count, model_active_parameter_count] + if isinstance(model_total_parameter_count, RangeValue): + total_params = [model_total_parameter_count.min, model_total_parameter_count.max] + else: + total_params = [model_total_parameter_count, model_total_parameter_count] + return active_params, total_params + + +def _merge_results( + results: dict[str, Union[RangeValue, float, int]], + new_results: dict[str, Any], + fields: list[str] +) -> None: + """ + Merge the results of a dag execution into the aggregated results as (min, max) ranges. + """ + for field in fields: + if field not in results: + results[field] = new_results[field] + continue + min_result = results[field] + max_result = new_results[field] + if isinstance(min_result, RangeValue): + min_result = cast(Union[float, int], min_result.min) + if isinstance(max_result, RangeValue): + max_result = cast(Union[float, int], max_result.max) + results[field] = RangeValue(min=min_result, max=max_result) + + +_INFERENCE_FIELDS = [ + "request_energy", "request_usage_gwp", "request_usage_adpe", "request_usage_pe", "request_usage_wcf", + "request_embodied_gwp", "request_embodied_adpe", "request_embodied_pe" +] +_TRAINING_FIELDS = [ + "request_training_energy", "request_training_gwp", "request_training_adpe", "request_training_pe", + "request_training_wcf" +] + + def compute_llm_impacts( model_active_parameter_count: ValueOrRange, model_total_parameter_count: ValueOrRange, @@ -523,13 +794,24 @@ def compute_llm_impacts( datacenter_pue: ValueOrRange, datacenter_wue: ValueOrRange, request_latency: Optional[float] = None, + input_token_count: Optional[float] = None, tps: Optional[float] = None, ttft: Optional[float] = None, + model_release_date: Optional[date] = None, + provider_ai_compute_capacity: Optional[float] = None, + provider_active_model_count: Optional[int] = None, + if_training_electricity_mix_adpe: Optional[float] = None, + if_training_electricity_mix_pe: Optional[float] = None, + if_training_electricity_mix_gwp: Optional[float] = None, + if_training_electricity_mix_wue: Optional[float] = None, **kwargs: Any ) -> Impacts: """ Compute the impacts of an LLM generation request. + The training phase is computed only when the model release date, the provider AI compute capacity and the + provider number of active models are all provided. Otherwise, `Impacts.training` is `None`. + Args: model_active_parameter_count: Number of active parameters of the model (in billion). model_total_parameter_count: Number of total parameters of the model (in billion). @@ -541,36 +823,47 @@ def compute_llm_impacts( datacenter_wue: Water Usage Effectiveness of the data center in L/kWh. datacenter_pue: Power Usage Effectiveness of the data center. request_latency: Measured request latency in seconds. + input_token_count: Number of input tokens (optional, used to estimate the pre-fill latency). tps: Number of tokens generated per second by the model. ttft: Time-to-first-token latency in seconds. - **kwargs: Any other optional parameter. + model_release_date: Release date of the model (optional, used for the training phase). + provider_ai_compute_capacity: AI compute capacity of the provider in kW (optional, used for the training + phase). + provider_active_model_count: Number of models of the provider actively serving requests (optional, used for + the training phase). + if_training_electricity_mix_adpe: ADPe impact factor of the electricity mix of the training location in + kgSbeq / kWh (defaults to the inference electricity mix). + if_training_electricity_mix_pe: PE impact factor of the electricity mix of the training location in + MJ / kWh (defaults to the inference electricity mix). + if_training_electricity_mix_gwp: GWP impact factor of the electricity mix of the training location in + kgCO2eq / kWh (defaults to the inference electricity mix). + if_training_electricity_mix_wue: WCF impact factor of the electricity mix of the training location in + L / kWh (defaults to the inference electricity mix). + **kwargs: Any other optional parameter of the inference or training dag. Returns: The impacts of an LLM generation request. """ if request_latency is None: request_latency = math.inf - active_params = [model_active_parameter_count] - total_params = [model_total_parameter_count] + with_training = model_release_date is not None \ + and provider_ai_compute_capacity is not None \ + and provider_active_model_count is not None + + active_params, total_params = _parameter_bounds(model_active_parameter_count, model_total_parameter_count) - if isinstance(model_active_parameter_count, RangeValue) or isinstance(model_total_parameter_count, RangeValue): - if isinstance(model_active_parameter_count, RangeValue): - active_params = [model_active_parameter_count.min, model_active_parameter_count.max] - else: - active_params = [model_active_parameter_count, model_active_parameter_count] - if isinstance(model_total_parameter_count, RangeValue): - total_params = [model_total_parameter_count.min, model_total_parameter_count.max] - else: - total_params = [model_total_parameter_count, model_total_parameter_count] + inference_kwargs = {k: v for k, v in kwargs.items() if k in _INFERENCE_DAG_PARAMETERS} + training_kwargs = {k: v for k, v in kwargs.items() if k not in _INFERENCE_DAG_PARAMETERS} + training_kwargs.setdefault("gpu_power", GPU_POWER) results: dict[str, Union[RangeValue, float, int]] = {} - fields = ["request_energy", "request_usage_gwp", "request_usage_adpe", "request_usage_pe", "request_usage_wcf", - "request_embodied_gwp", "request_embodied_adpe", "request_embodied_pe"] + fields = _INFERENCE_FIELDS + (_TRAINING_FIELDS if with_training else []) for act_param, tot_param in zip(active_params, total_params): res = compute_llm_impacts_dag( model_active_parameter_count=act_param, model_total_parameter_count=tot_param, output_token_count=output_token_count, + input_token_count=input_token_count, request_latency=request_latency, if_electricity_mix_adpe=if_electricity_mix_adpe, if_electricity_mix_pe=if_electricity_mix_pe, @@ -580,19 +873,21 @@ def compute_llm_impacts( datacenter_wue=datacenter_wue, tps=tps, ttft=ttft, - **kwargs + **inference_kwargs ) - for field in fields: - if field in results: - min_result = results[field] - max_result = res[field] - if isinstance(min_result, RangeValue): - min_result = cast(Union[float, int], min_result.min) - if isinstance(max_result, RangeValue): - max_result = cast(Union[float, int], max_result.max) - results[field] = RangeValue(min=min_result, max=max_result) - else: - results[field] = res[field] + if with_training: + res = compute_llm_training_impacts_dag( + inference_results=res, + model_release_date=cast(date, model_release_date), + provider_ai_compute_capacity=cast(float, provider_ai_compute_capacity), + provider_active_model_count=cast(int, provider_active_model_count), + if_electricity_mix_adpe=_default(if_training_electricity_mix_adpe, if_electricity_mix_adpe), + if_electricity_mix_pe=_default(if_training_electricity_mix_pe, if_electricity_mix_pe), + if_electricity_mix_gwp=_default(if_training_electricity_mix_gwp, if_electricity_mix_gwp), + if_electricity_mix_wue=_default(if_training_electricity_mix_wue, if_electricity_mix_wue), + **training_kwargs + ) + _merge_results(results, res, fields) energy = Energy(value=results["request_energy"]) gwp_usage = GWP(value=results["request_usage_gwp"]) @@ -603,6 +898,16 @@ def compute_llm_impacts( adpe_embodied = ADPe(value=results["request_embodied_adpe"]) pe_embodied = PE(value=results["request_embodied_pe"]) + training = None + if with_training: + training = Training( + energy=Energy(value=results["request_training_energy"]), + gwp=GWP(value=results["request_training_gwp"]), + adpe=ADPe(value=results["request_training_adpe"]), + pe=PE(value=results["request_training_pe"]), + wcf=WCF(value=results["request_training_wcf"]), + ) + return Impacts( energy=energy, gwp=gwp_usage + gwp_embodied, @@ -620,5 +925,9 @@ def compute_llm_impacts( gwp=gwp_embodied, adpe=adpe_embodied, pe=pe_embodied - ) + ), + training=training ) + + +_INFERENCE_DAG_PARAMETERS = set(compute_llm_impacts_dag.__annotations__.keys()) - {"return"} diff --git a/ecologits/impacts/modeling.py b/ecologits/impacts/modeling.py index af91c42f..6c49a407 100644 --- a/ecologits/impacts/modeling.py +++ b/ecologits/impacts/modeling.py @@ -1,5 +1,5 @@ from functools import total_ordering -from typing import TypeVar +from typing import Optional, TypeVar from pydantic import BaseModel @@ -206,10 +206,42 @@ class Embodied(Phase): pe: PE +class Training(Phase): + """ + Training impacts data model. + + Info: + Share of the model training impacts (final training run, research and development experiments and training + data storage) allocated to the request. Estimated with a top-down approach adapted from the + [Impact'IA](https://github.com/SNCF-ImpactIA/ImpactIA) methodology. Reported separately from the usage and + embodied phases and not included in the total impacts. + + Attributes: + type: training + name: Training + energy: Energy consumption allocated to the request + gwp: Global Warming Potential (GWP) training impact + adpe: Abiotic Depletion Potential for Elements (ADPe) training impact + pe: Primary Energy (PE) training impact + wcf: Water Consumption Footprint (WCF) training impact + """ + type: str = "training" + name: str = "Training" + energy: Energy + gwp: GWP + adpe: ADPe + pe: PE + wcf: WCF + + class Impacts(BaseModel): """ Impacts data model. + Info: + Total impacts (`energy`, `gwp`, `adpe`, `pe` and `wcf`) are the sum of the usage and embodied phases of the + inference. The training phase is reported separately and is not included in the totals. + Attributes: energy: Total energy consumption gwp: Total Global Warming Potential (GWP) impact @@ -218,6 +250,7 @@ class Impacts(BaseModel): wcf: Usage-only Water Consumption Footprint (WCF) impact usage: Impacts for the usage phase embodied: Impacts for the embodied phase + training: Impacts allocated from the model training (not included in totals, None when not available) """ energy: Energy gwp: GWP @@ -226,3 +259,4 @@ class Impacts(BaseModel): wcf: WCF usage: Usage embodied: Embodied + training: Optional[Training] = None diff --git a/ecologits/impacts/training.py b/ecologits/impacts/training.py new file mode 100644 index 00000000..35773f58 --- /dev/null +++ b/ecologits/impacts/training.py @@ -0,0 +1,510 @@ +""" +Estimation of the training impacts of an LLM allocated to a generation request. + +The methodology is adapted from the top-down approach of the Impact'IA calculator (SNCF, Wavestone, Resilio, 2026): + +1. The compute required by the final training run is estimated from the model size and release date with a + regression on Epoch AI data, then converted to energy with a compute efficiency (FLOPs per watt), the same + infrastructure overheads as the inference phase and the data center PUE. +2. Research and development experiments (test runs, ablations, etc.) are accounted for as a multiple of the final + training run. +3. The storage of the training dataset is estimated from the number of training tokens. +4. The training impacts are allocated to the request in proportion to the number of output tokens over the total + number of tokens the model is expected to generate during its lifetime. The latter is estimated top-down from the + AI compute capacity of the provider shared between its active models. + +Embodied impacts of the training infrastructure are estimated by applying the embodied impact intensity of the +inference phase (embodied impacts per kWh of IT energy) to the training energy. +""" +import math +from datetime import date +from typing import Any, Optional + +from ecologits.impacts.dag import DAG +from ecologits.utils.range_value import ValueOrRange + +TRAINING_FLOPS_REFERENCE_DATE = date(2020, 1, 1) +TRAINING_FLOPS_ALPHA = 0.0006 # log10(FLOPs) increase per day since the reference date +TRAINING_FLOPS_BETA = 17.151 # log10(FLOPs) intercept +TRAINING_FLOPS_GAMMA = 0.541 # exponent on the number of total parameters + +COMPUTE_FLOPS_PER_WATT = 1.4e12 # FLOPS / W (NVIDIA H100, dense BF16 peak, Epoch AI) +RND_TRAINING_FACTOR = 4 # R&D experiments energy as a multiple of the final training run + +TRAINING_DURATION = 100 * 24 # hours (Epoch AI) +TRAINING_BYTES_PER_TOKEN = 4 * 32 / 8 # 4 characters per token encoded on 32 bits +HDD_CAPACITY = 30 # TB (Seagate Exos 30 TB) +HDD_POWER = 0.0095 # kW +HDD_USAGE_RATIO = 0.2 + +INFERENCE_CAPACITY_RATIO = 0.8 # share of the provider AI compute capacity dedicated to inference +COMPUTE_EFFICIENCY_FACTOR = 0.5 * 0.5 * 0.7 # GPU utilization x memory-bound limitation x GPU sharing +MODEL_LIFETIME = 2 * 365.25 * 24 * 3600 # seconds + +dag = DAG() + + +@dag.asset +def training_flops( + model_total_parameter_count: float, + model_release_days: float, + training_flops_alpha: float, + training_flops_beta: float, + training_flops_gamma: float +) -> float: + """ + Estimate the compute of the final training run of the model. + + Regression from Impact'IA on Epoch AI notable models data: at equal size, recent models are trained with more + compute, and bigger models need more compute. + + Args: + model_total_parameter_count: Number of parameters of the model (in billion). + model_release_days: Number of days between the reference date (2020-01-01) and the model release date. + training_flops_alpha: Time coefficient of the regression (in log10(FLOPs) per day). + training_flops_beta: Intercept of the regression (in log10(FLOPs)). + training_flops_gamma: Exponent of the number of parameters. + + Returns: + The compute of the final training run in FLOPs. + """ + return 10 ** (training_flops_alpha * model_release_days + training_flops_beta) \ + * (model_total_parameter_count * 1e9) ** training_flops_gamma + + +@dag.asset +def infrastructure_overhead_ratio( + gpu_power: float, + server_power: float, + network_power: float, + server_gpu_count: int +) -> float: + """ + Compute the ratio between the IT power of a server (GPUs, server and network equipment) and its GPUs power. + + Used to extend the GPU energy of the training phase to the same technical scope as the inference phase. + + Args: + gpu_power: Power consumption of a single GPU in kW. + server_power: Power consumption of the server (without GPUs) in kW. + network_power: Power consumption of the network equipment allocated to one server in kW. + server_gpu_count: Number of GPUs in the server. + + Returns: + The infrastructure overhead ratio (greater than 1). + """ + return (server_gpu_count * gpu_power + server_power + network_power) / (server_gpu_count * gpu_power) + + +@dag.asset +def training_it_energy( + training_flops: float, + compute_flops_per_watt: float, + infrastructure_overhead_ratio: float +) -> float: + """ + Compute the IT energy consumption of the final training run (GPUs, servers and network equipment). + + Args: + training_flops: Compute of the final training run in FLOPs. + compute_flops_per_watt: Compute efficiency of the training hardware in FLOPS / W. + infrastructure_overhead_ratio: Ratio between the IT power and the GPUs power of a server. + + Returns: + The IT energy consumption of the final training run in kWh. + """ + return training_flops / compute_flops_per_watt / 3600 / 1000 * infrastructure_overhead_ratio + + +@dag.asset +def rnd_it_energy( + training_it_energy: float, + rnd_training_factor: float +) -> float: + """ + Compute the IT energy consumption of the research and development experiments preceding the final training run. + + Args: + training_it_energy: IT energy consumption of the final training run in kWh. + rnd_training_factor: R&D energy consumption as a multiple of the final training run. + + Returns: + The IT energy consumption of the R&D experiments in kWh. + """ + return training_it_energy * rnd_training_factor + + +@dag.asset +def training_token_count( + training_flops: float, + model_total_parameter_count: float +) -> float: + """ + Estimate the number of tokens used to train the model (Kaplan et al., 2020: FLOPs = 6 x parameters x tokens). + + Args: + training_flops: Compute of the final training run in FLOPs. + model_total_parameter_count: Number of parameters of the model (in billion). + + Returns: + The number of training tokens. + """ + return training_flops / (6 * model_total_parameter_count * 1e9) + + +@dag.asset +def training_hdd_count( + training_token_count: float, + training_bytes_per_token: float, + hdd_capacity: float +) -> int: + """ + Estimate the number of hard disk drives required to store the training dataset. + + Args: + training_token_count: Number of training tokens. + training_bytes_per_token: Storage size of a token in bytes. + hdd_capacity: Capacity of a single hard disk drive in TB. + + Returns: + The number of hard disk drives. + """ + dataset_size = math.ceil(training_token_count * training_bytes_per_token / 1024 ** 4) # TB + return max(1, math.ceil(dataset_size / hdd_capacity)) + + +@dag.asset +def storage_it_energy( + training_hdd_count: int, + hdd_power: float, + hdd_usage_ratio: float, + training_duration: float +) -> float: + """ + Compute the IT energy consumption of the training dataset storage during the training run. + + Args: + training_hdd_count: Number of hard disk drives storing the training dataset. + hdd_power: Power consumption of a single hard disk drive in kW. + hdd_usage_ratio: Effective usage ratio of a hard disk drive. + training_duration: Duration of the training run in hours. + + Returns: + The IT energy consumption of the training dataset storage in kWh. + """ + return hdd_power * training_hdd_count * hdd_usage_ratio * training_duration + + +@dag.asset +def model_lifetime_output_token_count( + provider_ai_compute_capacity: float, + inference_capacity_ratio: float, + provider_active_model_count: int, + compute_flops_per_watt: float, + compute_efficiency_factor: float, + datacenter_pue: ValueOrRange, + model_active_parameter_count: float, + model_lifetime: float +) -> ValueOrRange: + """ + Estimate the total number of tokens generated by the model during its lifetime. + + Top-down estimation from Impact'IA: the AI compute capacity of the provider dedicated to inference is shared + between its active models, converted to an effective compute (FLOPS) with a compute efficiency and utilization + factor, then to generated tokens (2 FLOPs per active parameter per token) over the model lifetime. + + Args: + provider_ai_compute_capacity: AI compute capacity of the provider in kW. + inference_capacity_ratio: Share of the compute capacity dedicated to inference. + provider_active_model_count: Number of models of the provider actively serving requests. + compute_flops_per_watt: Compute efficiency of the inference hardware in FLOPS / W. + compute_efficiency_factor: Effective utilization of the compute capacity. + datacenter_pue: Power Usage Effectiveness of the data center. + model_active_parameter_count: Number of active parameters of the model (in billion). + model_lifetime: Lifetime of the model in seconds. + + Returns: + The number of tokens generated by the model during its lifetime. + """ + model_compute_capacity = provider_ai_compute_capacity * 1000 * inference_capacity_ratio \ + / provider_active_model_count # W + model_flops = model_compute_capacity * compute_flops_per_watt * compute_efficiency_factor / datacenter_pue + return model_flops * model_lifetime / (2 * model_active_parameter_count * 1e9) + + +@dag.asset +def request_training_it_energy( + training_it_energy: float, + rnd_it_energy: float, + storage_it_energy: float, + model_lifetime_output_token_count: ValueOrRange, + output_token_count: float +) -> ValueOrRange: + """ + Compute the IT energy consumption of the training phase allocated to the request. + + Args: + training_it_energy: IT energy consumption of the final training run in kWh. + rnd_it_energy: IT energy consumption of the R&D experiments in kWh. + storage_it_energy: IT energy consumption of the training dataset storage in kWh. + model_lifetime_output_token_count: Number of tokens generated by the model during its lifetime. + output_token_count: Number of generated tokens. + + Returns: + The IT energy consumption of the training phase allocated to the request in kWh. + """ + return (training_it_energy + rnd_it_energy + storage_it_energy) * output_token_count \ + / model_lifetime_output_token_count + + +@dag.asset +def request_training_energy( + request_training_it_energy: ValueOrRange, + datacenter_pue: ValueOrRange +) -> ValueOrRange: + """ + Compute the energy consumption of the training phase allocated to the request. + + Args: + request_training_it_energy: IT energy consumption of the training phase allocated to the request in kWh. + datacenter_pue: Power Usage Effectiveness of the data center. + + Returns: + The energy consumption of the training phase allocated to the request in kWh. + """ + return request_training_it_energy * datacenter_pue + + +@dag.asset +def inference_embodied_gwp_intensity( + request_embodied_gwp: ValueOrRange, + request_it_energy: ValueOrRange +) -> ValueOrRange: + """ + Compute the Global Warming Potential (GWP) embodied impact intensity of the inference phase. + + Args: + request_embodied_gwp: GWP embodied impact of the request in kgCO2eq. + request_it_energy: IT energy consumption of the request in kWh. + + Returns: + The GWP embodied impact per kWh of IT energy in kgCO2eq / kWh. + """ + if request_it_energy == 0: + return 0 + return request_embodied_gwp / request_it_energy + + +@dag.asset +def inference_embodied_adpe_intensity( + request_embodied_adpe: ValueOrRange, + request_it_energy: ValueOrRange +) -> ValueOrRange: + """ + Compute the Abiotic Depletion Potential for Elements (ADPe) embodied impact intensity of the inference phase. + + Args: + request_embodied_adpe: ADPe embodied impact of the request in kgSbeq. + request_it_energy: IT energy consumption of the request in kWh. + + Returns: + The ADPe embodied impact per kWh of IT energy in kgSbeq / kWh. + """ + if request_it_energy == 0: + return 0 + return request_embodied_adpe / request_it_energy + + +@dag.asset +def inference_embodied_pe_intensity( + request_embodied_pe: ValueOrRange, + request_it_energy: ValueOrRange +) -> ValueOrRange: + """ + Compute the Primary Energy (PE) embodied impact intensity of the inference phase. + + Args: + request_embodied_pe: PE embodied impact of the request in MJ. + request_it_energy: IT energy consumption of the request in kWh. + + Returns: + The PE embodied impact per kWh of IT energy in MJ / kWh. + """ + if request_it_energy == 0: + return 0 + return request_embodied_pe / request_it_energy + + +@dag.asset +def request_training_gwp( + request_training_energy: ValueOrRange, + request_training_it_energy: ValueOrRange, + if_electricity_mix_gwp: float, + inference_embodied_gwp_intensity: ValueOrRange +) -> ValueOrRange: + """ + Compute the Global Warming Potential (GWP) impact of the training phase allocated to the request. + + Args: + request_training_energy: Energy consumption of the training phase allocated to the request in kWh. + request_training_it_energy: IT energy consumption of the training phase allocated to the request in kWh. + if_electricity_mix_gwp: GWP impact factor of electricity consumption in kgCO2eq / kWh. + inference_embodied_gwp_intensity: GWP embodied impact per kWh of IT energy in kgCO2eq / kWh. + + Returns: + The GWP impact (usage and embodied) of the training phase allocated to the request in kgCO2eq. + """ + return request_training_energy * if_electricity_mix_gwp \ + + request_training_it_energy * inference_embodied_gwp_intensity + + +@dag.asset +def request_training_adpe( + request_training_energy: ValueOrRange, + request_training_it_energy: ValueOrRange, + if_electricity_mix_adpe: float, + inference_embodied_adpe_intensity: ValueOrRange +) -> ValueOrRange: + """ + Compute the Abiotic Depletion Potential for Elements (ADPe) impact of the training phase allocated to the request. + + Args: + request_training_energy: Energy consumption of the training phase allocated to the request in kWh. + request_training_it_energy: IT energy consumption of the training phase allocated to the request in kWh. + if_electricity_mix_adpe: ADPe impact factor of electricity consumption in kgSbeq / kWh. + inference_embodied_adpe_intensity: ADPe embodied impact per kWh of IT energy in kgSbeq / kWh. + + Returns: + The ADPe impact (usage and embodied) of the training phase allocated to the request in kgSbeq. + """ + return request_training_energy * if_electricity_mix_adpe \ + + request_training_it_energy * inference_embodied_adpe_intensity + + +@dag.asset +def request_training_pe( + request_training_energy: ValueOrRange, + request_training_it_energy: ValueOrRange, + if_electricity_mix_pe: float, + inference_embodied_pe_intensity: ValueOrRange +) -> ValueOrRange: + """ + Compute the Primary Energy (PE) impact of the training phase allocated to the request. + + Args: + request_training_energy: Energy consumption of the training phase allocated to the request in kWh. + request_training_it_energy: IT energy consumption of the training phase allocated to the request in kWh. + if_electricity_mix_pe: PE impact factor of electricity consumption in MJ / kWh. + inference_embodied_pe_intensity: PE embodied impact per kWh of IT energy in MJ / kWh. + + Returns: + The PE impact (usage and embodied) of the training phase allocated to the request in MJ. + """ + return request_training_energy * if_electricity_mix_pe \ + + request_training_it_energy * inference_embodied_pe_intensity + + +@dag.asset +def request_training_wcf( + request_training_it_energy: ValueOrRange, + if_electricity_mix_wue: float, + datacenter_wue: ValueOrRange, + datacenter_pue: ValueOrRange +) -> ValueOrRange: + """ + Compute the water usage impact of the training phase allocated to the request. + + Args: + request_training_it_energy: IT energy consumption of the training phase allocated to the request in kWh. + if_electricity_mix_wue: WCF impact factor of electricity consumption in L / kWh. + datacenter_wue: Water Usage Effectiveness of the data center in L/kWh. + datacenter_pue: Power Usage Effectiveness of the data center. + + Returns: + The water usage impact of the training phase allocated to the request in liters. + """ + return request_training_it_energy * (datacenter_wue + datacenter_pue * if_electricity_mix_wue) + + +def compute_llm_training_impacts_dag( + inference_results: dict[str, Any], + model_release_date: date, + provider_ai_compute_capacity: float, + provider_active_model_count: int, + if_electricity_mix_adpe: float, + if_electricity_mix_pe: float, + if_electricity_mix_gwp: float, + if_electricity_mix_wue: float, + gpu_power: float, + training_flops_alpha: Optional[float] = TRAINING_FLOPS_ALPHA, + training_flops_beta: Optional[float] = TRAINING_FLOPS_BETA, + training_flops_gamma: Optional[float] = TRAINING_FLOPS_GAMMA, + compute_flops_per_watt: Optional[float] = COMPUTE_FLOPS_PER_WATT, + rnd_training_factor: Optional[float] = RND_TRAINING_FACTOR, + training_duration: Optional[float] = TRAINING_DURATION, + training_bytes_per_token: Optional[float] = TRAINING_BYTES_PER_TOKEN, + hdd_capacity: Optional[float] = HDD_CAPACITY, + hdd_power: Optional[float] = HDD_POWER, + hdd_usage_ratio: Optional[float] = HDD_USAGE_RATIO, + inference_capacity_ratio: Optional[float] = INFERENCE_CAPACITY_RATIO, + compute_efficiency_factor: Optional[float] = COMPUTE_EFFICIENCY_FACTOR, + model_lifetime: Optional[float] = MODEL_LIFETIME, + **kwargs: Any +) -> dict[str, Any]: + """ + Compute the training impacts dag of an LLM generation request. + + Args: + inference_results: Results of the inference impacts dag (see `compute_llm_impacts_dag`). + model_release_date: Release date of the model. + provider_ai_compute_capacity: AI compute capacity of the provider in kW. + provider_active_model_count: Number of models of the provider actively serving requests. + if_electricity_mix_adpe: ADPe impact factor of the training electricity mix in kgSbeq / kWh (Antimony). + if_electricity_mix_pe: PE impact factor of the training electricity mix in MJ / kWh. + if_electricity_mix_gwp: GWP impact factor of the training electricity mix in kgCO2eq / kWh. + if_electricity_mix_wue: WCF impact factor of the training electricity mix in L / kWh. + gpu_power: Power consumption of a single GPU in kW. + training_flops_alpha: Time coefficient of the training compute regression. + training_flops_beta: Intercept of the training compute regression. + training_flops_gamma: Parameters exponent of the training compute regression. + compute_flops_per_watt: Compute efficiency of the hardware in FLOPS / W. + rnd_training_factor: R&D energy consumption as a multiple of the final training run. + training_duration: Duration of the training run in hours. + training_bytes_per_token: Storage size of a training token in bytes. + hdd_capacity: Capacity of a single hard disk drive in TB. + hdd_power: Power consumption of a single hard disk drive in kW. + hdd_usage_ratio: Effective usage ratio of a hard disk drive. + inference_capacity_ratio: Share of the provider compute capacity dedicated to inference. + compute_efficiency_factor: Effective utilization of the inference compute capacity. + model_lifetime: Lifetime of the model in seconds. + **kwargs: Any other parameter of the training dag (overrides intermediate results). + + Returns: + The inference results extended with the training impacts dag intermediate states. + """ + inputs: dict[str, Any] = dict(inference_results) + inputs.update( + model_release_days=(model_release_date - TRAINING_FLOPS_REFERENCE_DATE).days, + provider_ai_compute_capacity=provider_ai_compute_capacity, + provider_active_model_count=provider_active_model_count, + if_electricity_mix_adpe=if_electricity_mix_adpe, + if_electricity_mix_pe=if_electricity_mix_pe, + if_electricity_mix_gwp=if_electricity_mix_gwp, + if_electricity_mix_wue=if_electricity_mix_wue, + gpu_power=gpu_power, + training_flops_alpha=training_flops_alpha, + training_flops_beta=training_flops_beta, + training_flops_gamma=training_flops_gamma, + compute_flops_per_watt=compute_flops_per_watt, + rnd_training_factor=rnd_training_factor, + training_duration=training_duration, + training_bytes_per_token=training_bytes_per_token, + hdd_capacity=hdd_capacity, + hdd_power=hdd_power, + hdd_usage_ratio=hdd_usage_ratio, + inference_capacity_ratio=inference_capacity_ratio, + compute_efficiency_factor=compute_efficiency_factor, + model_lifetime=model_lifetime, + ) + inputs.update(kwargs) + return dag.execute(**inputs) diff --git a/ecologits/model_repository.py b/ecologits/model_repository.py index 2caedb19..33589e7a 100644 --- a/ecologits/model_repository.py +++ b/ecologits/model_repository.py @@ -1,5 +1,7 @@ import json import os +import re +from datetime import date from enum import Enum from typing import Any, Optional, Union @@ -8,6 +10,33 @@ from ecologits.status_messages import WarningMessage from ecologits.utils.range_value import ValueOrRange +ACTIVE_MODEL_PERIOD_DAYS = 2 * 365 + +_MODEL_FAMILY_SUFFIXES = re.compile( + r"(-latest|-\d{4}-\d{2}-\d{2}|-\d{8}|-\d{4}(-rc\d+)?|-\d{3})$" +) + + +def model_family(model_name: str) -> str: + """ + Strip version suffixes (dates, snapshots, `latest`) from a model name to identify its family. + + Examples: + `gpt-4o-2024-08-06` -> `gpt-4o`, `claude-opus-4-1-20250805` -> `claude-opus-4-1`, + `mistral-medium-2508` -> `mistral-medium`, `codestral-latest` -> `codestral`. + + Args: + model_name: Name of the model. + + Returns: + The model family name. + """ + previous = None + while previous != model_name: + previous = model_name + model_name = _MODEL_FAMILY_SUFFIXES.sub("", model_name) + return model_name + class Providers(Enum): anthropic = "anthropic" @@ -55,6 +84,7 @@ class Model(BaseModel): warnings: Warnings linked to the model (e.g. "model-arch-not-released" or "model-arch-multimodal") sources: Source of the model information (website link) deployment: Deployment information (tps, ttft) + release_date: Release date of the model (used to estimate the training impacts) """ provider: Providers @@ -63,6 +93,7 @@ class Model(BaseModel): warnings: list[WarningMessage] = [] sources: list[str] = [] deployment: Deployment | None = None + release_date: date | None = None @property def has_warnings(self) -> bool: @@ -79,6 +110,9 @@ def from_json(cls, data: dict[str, Any]) -> "Model": deployment = None if "deployment" in data and data["deployment"] is not None: deployment = Deployment.model_validate(data["deployment"]) + release_date = None + if "release_date" in data and data["release_date"] is not None: + release_date = date.fromisoformat(data["release_date"]) return cls( provider=Providers(data["provider"]), name=data["name"], @@ -86,6 +120,7 @@ def from_json(cls, data: dict[str, Any]) -> "Model": warnings=warnings, sources=sources, deployment=deployment, + release_date=release_date, ) @@ -96,6 +131,7 @@ class ModelRepository: def __init__(self, models: Optional[list[Model]] = None, aliases: Optional[list[Alias]] = None) -> None: self.__models: dict[tuple[str, str], Model] = {} + self.__aliases: set[str] = set() if models is not None: for m in models: key = m.provider.value, m.name @@ -112,6 +148,7 @@ def __init__(self, models: Optional[list[Model]] = None, aliases: Optional[list[ model = self.__models[model_key].model_copy() model.name = a.name self.__models[alias_key] = model + self.__aliases.add(a.name) def add_model(self, data: dict[str, Any]) -> None: model = Model.from_json(data) @@ -126,6 +163,33 @@ def find_model(self, provider: str, model_name: str) -> Optional[Model]: def list_models(self) -> list[Model]: return list(self.__models.values()) + def count_active_models(self, provider: str, reference_date: Optional[date] = None, + active_period_days: int = ACTIVE_MODEL_PERIOD_DAYS) -> int: + """ + Count the models of a provider that are actively serving requests. + + A model is considered active during a fixed period after its release date (2 years by default), as in the + Impact'IA methodology. Dated snapshots and `latest` versions of the same model family are counted once (see + `model_family`), aliases are not counted and models without a release date are ignored. + + Args: + provider: Name of the provider. + reference_date: Date at which the count is made (today by default). + active_period_days: Duration in days after its release during which a model is considered active. + + Returns: + The number of active models of the provider. + """ + if reference_date is None: + reference_date = date.today() + families = set() + for (model_provider, name), model in self.__models.items(): + if model_provider != provider or name in self.__aliases or model.release_date is None: + continue + if 0 <= (reference_date - model.release_date).days < active_period_days: + families.add(model_family(name)) + return len(families) + @classmethod def from_json(cls, filepath: Optional[str] = None) -> "ModelRepository": if filepath is None: diff --git a/ecologits/status_messages.py b/ecologits/status_messages.py index f9dc9750..585cf770 100644 --- a/ecologits/status_messages.py +++ b/ecologits/status_messages.py @@ -81,6 +81,12 @@ class ElectricityMixWUEDefaultWarning(WarningMessage): message: str = "The electricity mix WUE factor is using the world default value, expect lower precision." +class TrainingNotModeledWarning(WarningMessage): + code: str = "training-not-modeled" + message: str = "The training impacts are not modeled for this model or provider (missing release date or " \ + "provider compute capacity), the training phase is not reported." + + class ModelNotRegisteredError(ErrorMessage): code: str = "model-not-registered" message: str = "The model is not registered in the model repository." @@ -97,6 +103,7 @@ class ZoneNotRegisteredError(ErrorMessage): "electricity-mix-adpe-world": ElectricityMixADPeDefaultWarning, "electricity-mix-pe-world": ElectricityMixPEDefaultWarning, "electricity-mix-wue-world": ElectricityMixWUEDefaultWarning, + "training-not-modeled": TrainingNotModeledWarning, } _error_codes: dict[str, type[ErrorMessage]] = { diff --git a/ecologits/tracers/anthropic_tracer.py b/ecologits/tracers/anthropic_tracer.py index 57f0f839..819f785e 100644 --- a/ecologits/tracers/anthropic_tracer.py +++ b/ecologits/tracers/anthropic_tracer.py @@ -56,6 +56,7 @@ def __stream_text__(self) -> Iterator[str]: # type: ignore[misc] provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -103,6 +104,7 @@ async def __stream_text__(self) -> AsyncIterator[str]: # type: ignore[misc] provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -193,6 +195,7 @@ def anthropic_chat_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=response.usage.output_tokens, + input_token_count=response.usage.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -237,6 +240,7 @@ async def anthropic_async_chat_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=response.usage.output_tokens, + input_token_count=response.usage.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) diff --git a/ecologits/tracers/cohere_tracer.py b/ecologits/tracers/cohere_tracer.py index 24a341ca..a1ef9b45 100644 --- a/ecologits/tracers/cohere_tracer.py +++ b/ecologits/tracers/cohere_tracer.py @@ -58,6 +58,7 @@ def cohere_chat_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=response.meta.tokens.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -102,6 +103,7 @@ async def cohere_async_chat_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=response.meta.tokens.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -149,6 +151,7 @@ def cohere_stream_chat_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=event.response.meta.tokens.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -199,6 +202,7 @@ async def cohere_async_stream_chat_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=event.response.meta.tokens.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) diff --git a/ecologits/tracers/google_genai_tracer.py b/ecologits/tracers/google_genai_tracer.py index 4c4dfc18..9e430f3d 100644 --- a/ecologits/tracers/google_genai_tracer.py +++ b/ecologits/tracers/google_genai_tracer.py @@ -50,6 +50,7 @@ def google_genai_content_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone, ) @@ -106,6 +107,7 @@ def google_genai_content_stream_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone, ) @@ -157,6 +159,7 @@ async def google_genai_async_content_wrapper( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone, ) @@ -197,6 +200,7 @@ async def _generator( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone, ) diff --git a/ecologits/tracers/huggingface_tracer.py b/ecologits/tracers/huggingface_tracer.py index 37711283..3901d176 100644 --- a/ecologits/tracers/huggingface_tracer.py +++ b/ecologits/tracers/huggingface_tracer.py @@ -91,6 +91,7 @@ def huggingface_chat_wrapper_non_stream( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -136,6 +137,7 @@ def huggingface_chat_wrapper_stream( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -201,6 +203,7 @@ async def huggingface_async_chat_wrapper_non_stream( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -246,6 +249,7 @@ async def huggingface_async_chat_wrapper_stream( provider=PROVIDER, model_name=model_name, output_token_count=output_tokens, + input_token_count=input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) diff --git a/ecologits/tracers/litellm_tracer.py b/ecologits/tracers/litellm_tracer.py index 7717f546..980bd21c 100644 --- a/ecologits/tracers/litellm_tracer.py +++ b/ecologits/tracers/litellm_tracer.py @@ -97,6 +97,7 @@ def litellm_chat_wrapper_stream( # type: ignore[misc] provider=model_match[0], model_name=model_match[1], output_token_count=token_count, + input_token_count=chunk.usage.prompt_tokens if getattr(chunk, "usage", None) else None, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -137,6 +138,7 @@ def litellm_chat_wrapper_non_stream( provider=model_match[0], model_name=model_match[1], output_token_count=response.usage.completion_tokens, + input_token_count=response.usage.prompt_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -197,6 +199,7 @@ async def litellm_async_chat_wrapper_base( provider=model_match[0], model_name=model_match[1], output_token_count=response.usage.completion_tokens, + input_token_count=response.usage.prompt_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -237,6 +240,7 @@ async def litellm_async_chat_wrapper_stream( # type: ignore[misc] provider=model_match[0], model_name=model_match[1], output_token_count=token_count, + input_token_count=chunk.usage.prompt_tokens if getattr(chunk, "usage", None) else None, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) diff --git a/ecologits/tracers/mistralai_tracer.py b/ecologits/tracers/mistralai_tracer.py index f94d73c9..81ce0cb6 100644 --- a/ecologits/tracers/mistralai_tracer.py +++ b/ecologits/tracers/mistralai_tracer.py @@ -51,6 +51,7 @@ def mistralai_chat_wrapper( provider=PROVIDER, model_name=response.model, output_token_count=response.usage.completion_tokens, + input_token_count=response.usage.prompt_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -98,6 +99,7 @@ def mistralai_chat_wrapper_stream( provider=PROVIDER, model_name=model_name, output_token_count=token_count, + input_token_count=chunk.data.usage.prompt_tokens if chunk.data.usage is not None else None, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -146,6 +148,7 @@ async def mistralai_async_chat_wrapper( provider=PROVIDER, model_name=response.model, output_token_count=response.usage.completion_tokens, + input_token_count=response.usage.prompt_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -179,6 +182,7 @@ async def _generator( provider=PROVIDER, model_name=model_name, output_token_count=token_count, + input_token_count=chunk.data.usage.prompt_tokens if chunk.data.usage is not None else None, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) diff --git a/ecologits/tracers/openai_tracer.py b/ecologits/tracers/openai_tracer.py index d397464c..3fb10690 100644 --- a/ecologits/tracers/openai_tracer.py +++ b/ecologits/tracers/openai_tracer.py @@ -75,6 +75,7 @@ def openai_chat_wrapper_non_stream( provider=PROVIDER, model_name=model_name, output_token_count=response.usage.completion_tokens, + input_token_count=response.usage.prompt_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -116,6 +117,7 @@ def openai_chat_wrapper_stream( # type: ignore[misc] provider=PROVIDER, model_name=model_name, output_token_count=output_token_count, + input_token_count=chunk.usage.prompt_tokens if getattr(chunk, "usage", None) else None, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -181,6 +183,7 @@ async def openai_async_chat_wrapper_base( provider=PROVIDER, model_name=model_name, output_token_count=response.usage.completion_tokens, + input_token_count=response.usage.prompt_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -222,6 +225,7 @@ async def openai_async_chat_wrapper_stream( # type: ignore[misc] provider=PROVIDER, model_name=model_name, output_token_count=output_token_count, + input_token_count=chunk.usage.prompt_tokens if getattr(chunk, "usage", None) else None, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -288,6 +292,7 @@ def openai_responses_wrapper_non_stream( provider=PROVIDER, model_name=model_name, output_token_count=response.usage.output_tokens, + input_token_count=response.usage.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -325,6 +330,7 @@ def openai_responses_wrapper_stream( # type: ignore[misc] provider=PROVIDER, model_name=model_name, output_token_count=output_token_count, + input_token_count=event.response.usage.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -382,6 +388,7 @@ async def openai_async_responses_wrapper_base( provider=PROVIDER, model_name=model_name, output_token_count=response.usage.output_tokens, + input_token_count=response.usage.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) @@ -419,6 +426,7 @@ async def openai_async_responses_wrapper_stream( # type: ignore[misc] provider=PROVIDER, model_name=model_name, output_token_count=output_token_count, + input_token_count=event.response.usage.input_tokens, request_latency=request_latency, electricity_mix_zone=EcoLogits.config.electricity_mix_zone ) diff --git a/ecologits/tracers/utils.py b/ecologits/tracers/utils.py index f733c6ff..b2ece82d 100644 --- a/ecologits/tracers/utils.py +++ b/ecologits/tracers/utils.py @@ -1,17 +1,26 @@ from __future__ import annotations from dataclasses import dataclass +from typing import Any from pydantic import BaseModel from ecologits.electricity_mix_repository import electricity_mixes from ecologits.impacts.llm import compute_llm_impacts -from ecologits.impacts.modeling import GWP, PE, WCF, ADPe, Embodied, Energy, Usage +from ecologits.impacts.modeling import GWP, PE, WCF, ADPe, Embodied, Energy, Training, Usage from ecologits.log import logger -from ecologits.model_repository import ParametersMoE, models -from ecologits.status_messages import ErrorMessage, ModelNotRegisteredError, WarningMessage, ZoneNotRegisteredError +from ecologits.model_repository import Model, ParametersMoE, models +from ecologits.status_messages import ( + ErrorMessage, + ModelNotRegisteredError, + TrainingNotModeledWarning, + WarningMessage, + ZoneNotRegisteredError, +) from ecologits.utils.range_value import RangeValue +TRAINING_ELECTRICITY_MIX_ZONE = "WOR" + class ImpactsOutput(BaseModel): """ @@ -25,6 +34,7 @@ class ImpactsOutput(BaseModel): wcf: Usage-only Water Consumption Footprint (WCF) impact usage: Impacts for the usage phase embodied: Impacts for the embodied phase + training: Impacts allocated from the model training (not included in totals) warnings: List of warnings errors: List of errors """ @@ -35,6 +45,7 @@ class ImpactsOutput(BaseModel): wcf: WCF | None = None usage: Usage | None = None embodied: Embodied | None = None + training: Training | None = None warnings: list[WarningMessage] | None = None errors: list[ErrorMessage] | None = None @@ -65,6 +76,7 @@ def llm_impacts( output_token_count: int, request_latency: float, electricity_mix_zone: str | None = None, + input_token_count: int | None = None, ) -> ImpactsOutput: """ High-level function to compute the impacts of an LLM generation request. @@ -75,6 +87,7 @@ def llm_impacts( output_token_count: Number of generated tokens. request_latency: Measured request latency in seconds. electricity_mix_zone: ISO 3166-1 alpha-3 code of the electricity mix zone (WOR by default). + input_token_count: Number of input tokens (optional, used to estimate the pre-fill latency). Returns: The impacts of an LLM generation request. @@ -93,24 +106,24 @@ def llm_impacts( model_total_params = model.architecture.parameters model_active_params = model.architecture.parameters - datacenter_location = PROVIDER_CONFIG_MAP[provider].datacenter_location datacenter_pue = PROVIDER_CONFIG_MAP[provider].datacenter_pue datacenter_wue = PROVIDER_CONFIG_MAP[provider].datacenter_wue if electricity_mix_zone is None: - electricity_mix_zone = datacenter_location - if electricity_mix_zone is None: - electricity_mix_zone = "WOR" + electricity_mix_zone = PROVIDER_CONFIG_MAP[provider].datacenter_location or "WOR" if_electricity_mix = electricity_mixes.find_electricity_mix(zone=electricity_mix_zone) if if_electricity_mix is None: error = ZoneNotRegisteredError(message=f"Could not find electricity mix for `{electricity_mix_zone}` zone.") logger.warning_once(str(error)) return ImpactsOutput(errors=[error]) + training_inputs = _training_inputs(provider=provider, model=model) + impacts = compute_llm_impacts( model_active_parameter_count=model_active_params, model_total_parameter_count=model_total_params, output_token_count=output_token_count, + input_token_count=input_token_count, request_latency=request_latency, if_electricity_mix_adpe=if_electricity_mix.adpe, if_electricity_mix_pe=if_electricity_mix.pe, @@ -120,27 +133,68 @@ def llm_impacts( datacenter_wue=datacenter_wue, tps=model.deployment.tps if model.deployment else None, ttft=model.deployment.ttft if model.deployment else None, + **training_inputs ) impacts = ImpactsOutput.model_validate(impacts.model_dump()) - if model.has_warnings: - for w in model.warnings: - logger.warning_once(str(w)) - impacts.add_warning(w) - - if if_electricity_mix.has_warnings: - for w in if_electricity_mix.warnings: - logger.warning_once(str(w)) - impacts.add_warning(w) + warnings = model.warnings + if_electricity_mix.warnings + if not training_inputs: + warnings.append(TrainingNotModeledWarning()) + for w in warnings: + logger.warning_once(str(w)) + impacts.add_warning(w) return impacts +def _training_inputs(provider: str, model: Model) -> dict[str, Any]: + """ + Gather the inputs required to estimate the training impacts of a model (adapted from Impact'IA). + + The training phase requires the model release date, the provider AI compute capacity and at least one active + model for the provider. The training location being unknown, the world electricity mix is used. + + Args: + provider: Name of the provider. + model: The model. + + Returns: + The keyword arguments for `compute_llm_impacts`, empty when the training impacts cannot be estimated. + """ + ai_compute_capacity = PROVIDER_CONFIG_MAP[provider].ai_compute_capacity + if model.release_date is None or ai_compute_capacity is None: + return {} + active_model_count = models.count_active_models(provider=provider) + training_electricity_mix = electricity_mixes.find_electricity_mix(zone=TRAINING_ELECTRICITY_MIX_ZONE) + if active_model_count == 0 or training_electricity_mix is None: + return {} + return { + "model_release_date": model.release_date, + "provider_ai_compute_capacity": ai_compute_capacity, + "provider_active_model_count": active_model_count, + "if_training_electricity_mix_adpe": training_electricity_mix.adpe, + "if_training_electricity_mix_pe": training_electricity_mix.pe, + "if_training_electricity_mix_gwp": training_electricity_mix.gwp, + "if_training_electricity_mix_wue": training_electricity_mix.wue, + } + + @dataclass class _ProviderConfig: + """ + Provider configuration. + + Attributes: + datacenter_location: ISO 3166-1 alpha-3 code of the default data center location. + datacenter_pue: Power Usage Effectiveness of the data centers. + datacenter_wue: Water Usage Effectiveness of the data centers in L / kWh. + ai_compute_capacity: AI compute capacity of the provider in kW (public announcements gathered by Impact'IA), + used to estimate the training impacts. None when unknown. + """ datacenter_location: str datacenter_pue: float | RangeValue datacenter_wue: float | RangeValue + ai_compute_capacity: float | None = None PROVIDER_CONFIG_MAP = { @@ -148,6 +202,7 @@ class _ProviderConfig: datacenter_location="USA", datacenter_pue=RangeValue(min=1.09, max=1.14), datacenter_wue=RangeValue(min=0.13, max=0.999), + ai_compute_capacity=1_400_000, ), "cohere": _ProviderConfig( datacenter_location="USA", @@ -158,6 +213,7 @@ class _ProviderConfig: datacenter_location="USA", datacenter_pue=1.09, datacenter_wue=0.999, + ai_compute_capacity=9_000_000, ), "huggingface_hub": _ProviderConfig( datacenter_location="USA", @@ -168,10 +224,12 @@ class _ProviderConfig: datacenter_location="SWE", datacenter_pue=1.16, datacenter_wue=0.09, + ai_compute_capacity=200_000, ), "openai": _ProviderConfig( datacenter_location="USA", datacenter_pue=1.20, datacenter_wue=0.569, + ai_compute_capacity=2_000_000, ) } diff --git a/ecologits/utils/range_value.py b/ecologits/utils/range_value.py index 86701411..fec8a495 100644 --- a/ecologits/utils/range_value.py +++ b/ecologits/utils/range_value.py @@ -55,9 +55,14 @@ def __truediv__(self, other: Union[int, float]) -> "RangeValue": max=self.max / other ) + def __rtruediv__(self, other: Union[int, float]) -> "RangeValue": + return RangeValue( + min=other / self.max, + max=other / self.min + ) + __radd__ = __add__ __rmul__ = __mul__ - __rtruediv__ = __truediv__ def __eq__(self, other: Any) -> bool: if isinstance(other, RangeValue): diff --git a/mkdocs.yml b/mkdocs.yml index 284c0583..1a783b35 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,6 +33,7 @@ nav: - 'Introduction': methodology/index.md - 'Proprietary models': methodology/proprietary_models.md - 'LLM Inference': methodology/llm_inference.md + - 'LLM Training': methodology/llm_training.md - 'Video Generation': methodology/video_generation.md - 'API Reference': reference/ - 'Blog': diff --git a/tests/test_llm_impacts_impactia.py b/tests/test_llm_impacts_impactia.py new file mode 100644 index 00000000..cd91ca5a --- /dev/null +++ b/tests/test_llm_impacts_impactia.py @@ -0,0 +1,110 @@ +import pytest + +from ecologits.impacts.llm import ( + DATACENTER_BUILDING_EMBODIED_GWP_FACTOR, + NETWORK_EMBODIED_IMPACT_GWP, + NETWORK_POWER, + PREFILL_LATENCY_INTERCEPT, + compute_llm_impacts, + compute_llm_impacts_dag, +) + +WORLD_MIX = { + "if_electricity_mix_adpe": 0.0000000737708, + "if_electricity_mix_pe": 9.988, + "if_electricity_mix_gwp": 0.590478, + "if_electricity_mix_wue": 5.04, +} + + +def _dag(**kwargs): + params = { + "model_active_parameter_count": 76, + "model_total_parameter_count": 76, + "output_token_count": 1000, + "request_latency": 1e9, + "datacenter_pue": 1.2, + "datacenter_wue": 0.569, + "tps": 40, + "ttft": 0.62, + **WORLD_MIX, + } + params.update(kwargs) + return compute_llm_impacts_dag(**params) + + +def test_network_constants(): + assert NETWORK_POWER == pytest.approx(0.16108, rel=1e-4) + assert NETWORK_EMBODIED_IMPACT_GWP == pytest.approx(660.1, rel=0.001) + + +def test_prefill_latency_without_input_tokens_uses_ttft(): + results = _dag(input_token_count=None) + assert results["prefill_latency"] == 0.62 + + +def test_prefill_latency_without_input_tokens_and_ttft(): + results = _dag(input_token_count=None, ttft=None) + assert results["prefill_latency"] == 0 + + +def test_prefill_latency_scales_with_input_tokens(): + small = _dag(input_token_count=100) + big = _dag(input_token_count=10_000) + assert small["prefill_latency"] > PREFILL_LATENCY_INTERCEPT + assert big["prefill_latency"] > small["prefill_latency"] + # Coefficient of Mistral 7B scaled by the TTFT ratio: 0.062ms * (0.62 / 0.44) per token + assert big["prefill_latency"] == pytest.approx(0.062e-3 * 0.62 / 0.44 * 10_000 + PREFILL_LATENCY_INTERCEPT) + assert big["generation_latency"] > small["generation_latency"] + + +def test_prefill_latency_scales_with_parameters_without_ttft(): + small = _dag(input_token_count=1000, ttft=None, model_active_parameter_count=7.3) + big = _dag(input_token_count=1000, ttft=None, model_active_parameter_count=73) + assert big["prefill_latency"] == pytest.approx(10 * (small["prefill_latency"] - PREFILL_LATENCY_INTERCEPT) + + PREFILL_LATENCY_INTERCEPT) + + +def test_network_energy_and_embodied(): + results = _dag(input_token_count=1000) + assert results["network_energy"] > 0 + assert results["network_energy"] == pytest.approx(results["server_energy"] * NETWORK_POWER / 1.2) + assert results["request_it_energy"] == pytest.approx( + results["server_energy"] + results["gpu_required_count"] * results["gpu_energy"] + results["network_energy"] + ) + assert results["request_network_embodied_gwp"] > 0 + assert results["request_datacenter_embodied_gwp"] == pytest.approx( + results["request_it_energy"] * DATACENTER_BUILDING_EMBODIED_GWP_FACTOR + ) + assert results["request_embodied_gwp"] == pytest.approx( + results["request_hardware_embodied_gwp"] + + results["request_network_embodied_gwp"] + + results["request_datacenter_embodied_gwp"] + ) + + +def test_network_and_building_can_be_disabled(): + results = _dag(network_power=0, network_embodied_gwp=0, datacenter_building_embodied_gwp_factor=0) + assert results["network_energy"] == 0 + assert results["request_embodied_gwp"] == results["request_hardware_embodied_gwp"] + + +def test_compute_llm_impacts_with_input_tokens(): + params = { + "model_active_parameter_count": 76, + "model_total_parameter_count": 76, + "output_token_count": 1000, + "request_latency": 100, + "datacenter_pue": 1.2, + "datacenter_wue": 0.569, + "tps": 40, + "ttft": 0.62, + **WORLD_MIX, + } + short = compute_llm_impacts(input_token_count=100, **params) + long = compute_llm_impacts(input_token_count=100_000, **params) + unknown = compute_llm_impacts(**params) + assert long.energy.value > short.energy.value + assert long.embodied.gwp.value > short.embodied.gwp.value + assert unknown.training is None + assert long.gwp == long.usage.gwp + long.embodied.gwp diff --git a/tests/test_model_repository.py b/tests/test_model_repository.py index 0299606a..03cf92f9 100644 --- a/tests/test_model_repository.py +++ b/tests/test_model_repository.py @@ -1,4 +1,6 @@ -from ecologits.model_repository import ModelRepository +from datetime import date + +from ecologits.model_repository import ModelRepository, model_family def test_create_empty_repository(): @@ -43,3 +45,62 @@ def test_add_custom_model(): } models.add_model(custom_model_data) assert models.find_model("openai", "gpt-4.1-2025-04-14") is not None + + +def test_release_date(): + models = ModelRepository.from_json() + model = models.find_model(provider="openai", model_name="gpt-4.1-mini") + assert model.release_date == date(2025, 4, 14) + alias = models.find_model(provider="anthropic", model_name="claude-opus-4-6") + assert alias.release_date == date(2026, 2, 5) + + +def test_release_date_optional(): + models = ModelRepository() + models.add_model({ + "provider": "openai", + "name": "custom-model", + "architecture": {"type": "dense", "parameters": 10} + }) + assert models.find_model("openai", "custom-model").release_date is None + + +def test_model_family(): + assert model_family("gpt-4o-2024-08-06") == "gpt-4o" + assert model_family("gpt-4o") == "gpt-4o" + assert model_family("claude-opus-4-1-20250805") == "claude-opus-4-1" + assert model_family("mistral-medium-2508") == "mistral-medium" + assert model_family("codestral-2411-rc5") == "codestral" + assert model_family("codestral-latest") == "codestral" + assert model_family("gpt-5-chat-latest") == "gpt-5-chat" + assert model_family("gemini-2.0-flash-001") == "gemini-2.0-flash" + assert model_family("gpt-3.5-turbo-0125") == "gpt-3.5-turbo" + assert model_family("gemini-3.1-pro-preview") == "gemini-3.1-pro-preview" + + +def test_count_active_models(): + models = ModelRepository() + for name, release_date in [ + ("model-a", "2025-01-01"), + ("model-a-2025-01-01", "2025-01-01"), + ("model-a-latest", "2025-06-01"), + ("model-b", "2025-06-01"), + ("model-old", "2020-01-01"), + ("model-future", "2030-01-01"), + ("model-no-date", None), + ]: + models.add_model({ + "provider": "openai", + "name": name, + "architecture": {"type": "dense", "parameters": 10}, + "release_date": release_date, + }) + assert models.count_active_models("openai", reference_date=date(2026, 1, 1)) == 2 + assert models.count_active_models("openai", reference_date=date(2028, 1, 1)) == 0 + assert models.count_active_models("anthropic", reference_date=date(2026, 1, 1)) == 0 + + +def test_count_active_models_default_repository(): + models = ModelRepository.from_json() + for provider in ["openai", "anthropic", "google_genai", "mistralai"]: + assert models.count_active_models(provider, reference_date=date(2026, 9, 1)) > 0 diff --git a/tests/test_range_value.py b/tests/test_range_value.py index 586707bf..278b2bd8 100644 --- a/tests/test_range_value.py +++ b/tests/test_range_value.py @@ -66,3 +66,9 @@ def test_value_range_compare(val_1, val_2, op, result): def test_value_range_transformation(val_1, val_2, op, exp_result): result = op(val_1, val_2) assert result.min == exp_result.min and result.max == exp_result.max + + +def test_reflected_true_division(): + result = 10 / RangeValue(min=2, max=5) + assert result.min == 2 + assert result.max == 5 diff --git a/tests/test_training_impacts.py b/tests/test_training_impacts.py new file mode 100644 index 00000000..de4ef155 --- /dev/null +++ b/tests/test_training_impacts.py @@ -0,0 +1,138 @@ +from datetime import date + +import pytest + +from ecologits.impacts.llm import compute_llm_impacts, compute_llm_impacts_dag +from ecologits.impacts.modeling import Training +from ecologits.impacts.training import compute_llm_training_impacts_dag +from ecologits.utils.range_value import RangeValue + +WORLD_MIX = { + "if_electricity_mix_adpe": 0.0000000737708, + "if_electricity_mix_pe": 9.988, + "if_electricity_mix_gwp": 0.590478, + "if_electricity_mix_wue": 5.04, +} +TRAINING_KWARGS = { + "model_release_date": date(2025, 4, 14), + "provider_ai_compute_capacity": 2_000_000, + "provider_active_model_count": 15, +} + + +def _compute(output_token_count=1000, **kwargs): + params = { + "model_active_parameter_count": 76, + "model_total_parameter_count": 76, + "output_token_count": output_token_count, + "request_latency": 100, + "datacenter_pue": 1.2, + "datacenter_wue": 0.569, + **WORLD_MIX, + **TRAINING_KWARGS, + } + params.update(kwargs) + return compute_llm_impacts(**params) + + +def test_training_phase_is_reported(): + impacts = _compute() + assert isinstance(impacts.training, Training) + assert impacts.training.energy.value > 0 + assert impacts.training.gwp.value > 0 + assert impacts.training.adpe.value > 0 + assert impacts.training.pe.value > 0 + assert impacts.training.wcf.value > 0 + + +def test_training_phase_not_included_in_totals(): + impacts = _compute() + assert impacts.gwp == impacts.usage.gwp + impacts.embodied.gwp + assert impacts.energy == impacts.usage.energy + + +@pytest.mark.parametrize("missing", ["model_release_date", "provider_ai_compute_capacity", + "provider_active_model_count"]) +def test_training_phase_requires_all_inputs(missing): + impacts = _compute(**{missing: None}) + assert impacts.training is None + + +def test_training_phase_is_proportional_to_output_tokens(): + small = _compute(output_token_count=100) + big = _compute(output_token_count=1000) + assert big.training.energy.value == pytest.approx(10 * small.training.energy.value) + assert big.training.gwp.value > small.training.gwp.value + + +def test_training_phase_with_zero_output_tokens(): + impacts = _compute(output_token_count=0) + assert impacts.training.energy.value == 0 + assert impacts.training.gwp.value == 0 + + +def test_training_phase_increases_with_model_size_and_recency(): + base = _compute() + bigger = _compute(model_active_parameter_count=400, model_total_parameter_count=400) + newer = _compute(model_release_date=date(2026, 4, 14)) + assert bigger.training.energy.value > base.training.energy.value + assert newer.training.energy.value > base.training.energy.value + + +def test_training_phase_decreases_with_provider_capacity(): + base = _compute() + more_capacity = _compute(provider_ai_compute_capacity=4_000_000) + more_models = _compute(provider_active_model_count=30) + assert more_capacity.training.energy.value == pytest.approx(base.training.energy.value / 2) + assert more_models.training.energy.value == pytest.approx(base.training.energy.value * 2) + + +def test_training_phase_uses_training_electricity_mix(): + base = _compute() + cleaner = _compute(if_training_electricity_mix_gwp=0.05) + assert cleaner.training.gwp.value < base.training.gwp.value + assert cleaner.training.energy.value == base.training.energy.value + assert cleaner.usage.gwp.value == base.usage.gwp.value + + +def test_training_phase_with_range_values(): + impacts = _compute( + model_active_parameter_count=RangeValue(min=40, max=112), + model_total_parameter_count=RangeValue(min=40, max=112), + datacenter_pue=RangeValue(min=1.09, max=1.14), + ) + assert isinstance(impacts.training.energy.value, RangeValue) + assert isinstance(impacts.training.gwp.value, RangeValue) + assert isinstance(impacts.training.wcf.value, RangeValue) + assert impacts.training.energy.value.min < impacts.training.energy.value.max + + +def test_training_dag_intermediate_values(): + inference = compute_llm_impacts_dag( + model_active_parameter_count=76, + model_total_parameter_count=76, + output_token_count=1000, + request_latency=100, + datacenter_pue=1.2, + datacenter_wue=0.569, + **WORLD_MIX, + ) + results = compute_llm_training_impacts_dag( + inference_results=inference, + gpu_power=0.7, + **WORLD_MIX, + **TRAINING_KWARGS, + ) + # Regression on Epoch AI data: 10^(0.0006 * 1930 + 17.151) * (76e9)^0.541 + assert results["training_flops"] == pytest.approx(1.566e24, rel=0.01) + assert results["infrastructure_overhead_ratio"] > 1 + assert results["rnd_it_energy"] == pytest.approx(4 * results["training_it_energy"]) + assert results["training_hdd_count"] >= 1 + assert results["model_lifetime_output_token_count"] > 0 + assert results["request_training_energy"] == pytest.approx(results["request_training_it_energy"] * 1.2) + + +def test_training_dag_override_intermediate_value(): + base = _compute() + overridden = _compute(model_lifetime_output_token_count=1e12) + assert overridden.training.energy.value != base.training.energy.value