Multiple weather years in one optimisation problem by fneum · Pull Request #1613 · PyPSA/pypsa-eur

fneum · 2025-03-26T17:19:19Z

Closes #1610.
Closes #1627.

Changes proposed in this Pull Request

TLDR: Allow multiple years (consecutive or non-consecutive) or flexible periods (e.g. July-June meteorological year) in one optimisation model.

Details: See release notes.

test myopic workflow
Because data/era5-annual-HDD-per-country.csv is currently only annually resolved, the heat demand for July-June years is currently only scaled 50% first year and 50% second year. I.e. different distributions of heat demand in first and second winter of year are not considered. -> data bundle is updated with daily resolved HDD, handling deferred to follow-up PR.

Configuration for one year

snapshots:
  start: "2019-01-01"
  end: "2020-01-01"

atlite:
  default_cutout: europe-2019-sarah3-era5

Configuration for N consecutive years

snapshots:
  start: "2019-01-01"
  end: "2021-01-01"

atlite:
  default_cutout: 
  - europe-2019-sarah3-era5
  - europe-2020-sarah3-era5

Configuration for meteorological year

snapshots:
  start: "2019-07-01"
  end: "2020-07-01"

atlite:
  default_cutout: 
  - europe-2019-sarah3-era5
  - europe-2020-sarah3-era5

Configuration for N non-contiguous years

snapshots:
  start: ["2013-01-01", "2019-01-01"]
  end: ["2014-01-01", "2020-01-01"]
  inclusive: 'left'

atlite:
  default_cutout: 
  - europe-2013-sarah3-era5
  - europe-2019-sarah3-era5

Checklist

I tested my contribution locally and it works as intended.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in config/config.default.yaml.
Changes in configuration options are documented in doc/configtables/*.csv.
Sources of newly added data are documented in doc/data_sources.rst.
A release note doc/release_notes.rst is added.

for more information, see https://pre-commit.ci

koen-vg@1dce9b0

Copilot

Pull Request Overview

This PR adds support for handling multiple weather years and concatenating cutouts, along with improvements to documentation and type hints. Key changes include:

Introducing a new load_cutout helper to support multiple cutout files and multiple time ranges
Updating various build scripts to use get_snapshots and load_cutout instead of hardcoding atlite.Cutout calls
Changing the default cutout configuration in the YAML file to "default" for several renewable energy sources

Reviewed Changes

Copilot reviewed 23 out of 35 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
scripts/build_hac_features.py	Updated imports and switched to load_cutout for cutout handling
scripts/build_egs_potentials.py	Updated get_capacity_factors signature and integrated get_snapshots
scripts/build_daily_heat_demand.py	Replaced direct atlite.Cutout call with load_cutout
scripts/build_cutout.py	Added an assert to limit snapshots to a single start/end date
scripts/build_clustered_population_layouts.py	Switched to load_cutout for cutout creation
scripts/build_central_heating_temperature_profiles/run.py	Updated snapshots generation via get_snapshots
scripts/_helpers.py	Enhanced get_snapshots for multiple time ranges and added load_cutout
config/config.default.yaml	Changed cutout references from specific strings to "default"

Files not reviewed (12)

doc/configtables/atlite.csv: Language not supported
doc/configtables/hydro.csv: Language not supported
doc/configtables/lines.csv: Language not supported
doc/configtables/offwind.csv: Language not supported
doc/configtables/onwind.csv: Language not supported
doc/configtables/snapshots.csv: Language not supported
doc/configtables/solar.csv: Language not supported
doc/release_notes.rst: Language not supported
rules/build_electricity.smk: Language not supported
rules/build_sector.smk: Language not supported
rules/common.smk: Language not supported
rules/postprocess.smk: Language not supported

Comments suppressed due to low confidence (2)

scripts/build_cutout.py:90

The assertion in build_cutout.py limits the snapshots to a single start and end date, which may conflict with the new support for multiple weather years. Consider updating the cutout rule to handle multiple time ranges or document the limitation clearly.

assert (len(snakemake.params.snapshots["start"]) == 1 and len(snakemake.params.snapshots["end"]) == 1), "The cutout rule currently only supports a single start and end date."

scripts/_helpers.py:1083

Using NamedTemporaryFile().name without managing the file lifecycle may cause issues with file availability on some platforms. Consider explicitly creating a persistent temporary file or managing the temporary file context to ensure the data file remains accessible.

cutout = atlite.Cutout(NamedTemporaryFile().name, data=combined_data)

…ify time handling

cpschau

Codewise everything looks good to me. I tested the myopic foresight and everything ran smoothly. Also the concatenation of more than 2 years worked. The only thing that failed is the plotting routine in plot_heatmap_timeseries which might needs to be skipped for multi-year optimizations.

Corrected objectives values (divided by 2 years) of run with concatenated weather years 2013 & 2019 lie in between the objectives of the single-year runs with 2019 being the less challenging year (2019) and 2013 being the more challenging.

Energy balances of network with both weather years look like this:

Energy balances of networks with single weather years:

for more information, see https://pre-commit.ci

fneum · 2025-04-04T07:06:33Z

Great, thanks @cpschau for reviewing! I will merge once CI passes.

…i-year-one

…port myopic

for more information, see https://pre-commit.ci

coroa · 2025-05-14T10:17:12Z

This PR updated the shared_resources: policy setting for tests from "test" to false, leading to repetitive recomputing of the renewable profiles for each test setting.

Is there an effort to try to bring that back?

@cpschau @fneum Do you remember how it failed?

tgilon · 2025-05-14T13:55:37Z

This PR updated the shared_resources: policy setting for tests from "test" to false, leading to repetitive recomputing of the renewable profiles for each test setting.

This might be relevant #1576.

cpschau · 2025-05-16T07:56:25Z

I think the setting was changed in be9fa8d after I tested the functionality. I do not remember the workflow failing without that change, so probably it is related to the last commits. Will probably be worth it to bring it back, as the calculation of the renewable profiles can take a lot of time.

* cutouts: use a default and allow concatenation * docs: docstrings and type hints for get_snapshots() * multiple heat years for multiple meteorological years * adjust add_import_limit_constraint for multiple years * adjust add_co2_sequestration_limit for multiple years * adjust add_biomass (potentials) for multiple years * follow-up: add up snapshot weightings in add_co2_sequestration_limit * add release notes for multiple full consecutive weather years * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * build_hydro_profile: use full-year cutouts for subannual periods * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: handle non-contiguous date range in time aggregation koen-vg@1dce9b0 * avoid that aggregated snapshot indices land on Feb 29 * support for non-consecutive weather years * update snapshots options documentation * config/test: simplify as technologies automatically go to default cutout * refactor: remove snapshots parameter from build_cutout rule and simplify time handling * refactor: Update cutout handling in solar rooftop potentials script * plto_heatmap_timeseries: leap days and non-contiguity check * switch era5-annual-hdd/runoff... to daily data and data bundle * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use highs-default for myopic and perfect test * ci: do not share resources, go back to highs-simplex, test biomass import myopic * fix: add_co2_sequestration_limit nyears with perfect foresight * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

fneum and others added 9 commits March 26, 2025 14:42

cutouts: use a default and allow concatenation

b45a32c

docs: docstrings and type hints for get_snapshots()

2ed64b6

multiple heat years for multiple meteorological years

5e4a884

adjust add_import_limit_constraint for multiple years

f9a8abf

adjust add_co2_sequestration_limit for multiple years

cbd42a1

adjust add_biomass (potentials) for multiple years

5b82bc3

follow-up: add up snapshot weightings in add_co2_sequestration_limit

05483ee

add release notes for multiple full consecutive weather years

2136c4c

[pre-commit.ci] auto fixes from pre-commit.com hooks

99ff91a

for more information, see https://pre-commit.ci

fneum mentioned this pull request Mar 27, 2025

support runoff normalization for partial years with annual normalization data PyPSA/atlite#434

Open

3 tasks

fneum and others added 7 commits March 27, 2025 08:48

build_hydro_profile: use full-year cutouts for subannual periods

cb12006

[pre-commit.ci] auto fixes from pre-commit.com hooks

276ba91

for more information, see https://pre-commit.ci

fix: handle non-contiguous date range in time aggregation

128f2be

koen-vg@1dce9b0

avoid that aggregated snapshot indices land on Feb 29

b47cc1d

support for non-consecutive weather years

28f313b

update snapshots options documentation

ed121c9

Merge branch 'master' into multi-year-one

336311c

fneum marked this pull request as ready for review March 28, 2025 11:38

fneum requested review from LukasFrankenQ, amos-schledorn and p-glaum as code owners March 28, 2025 11:38

fneum requested review from Copilot and cpschau and removed request for LukasFrankenQ, amos-schledorn and p-glaum March 28, 2025 11:38

Copilot AI reviewed Mar 28, 2025

View reviewed changes

fneum added 2 commits March 28, 2025 13:08

config/test: simplify as technologies automatically go to default cutout

00ab28d

refactor: remove snapshots parameter from build_cutout rule and simpl…

dba863c

…ify time handling

cpschau mentioned this pull request Apr 2, 2025

Sub-annual optimisation and (unsustainable) biomass potentials #1627

Closed

2 tasks

Merge branch 'master' into multi-year-one

86f0924

cpschau approved these changes Apr 3, 2025

View reviewed changes

Comment thread scripts/plot_heatmap_timeseries.py

fneum and others added 5 commits April 3, 2025 17:30

refactor: Update cutout handling in solar rooftop potentials script

246ceb0

plto_heatmap_timeseries: leap days and non-contiguity check

b5fe6f2

switch era5-annual-hdd/runoff... to daily data and data bundle

1925b31

Merge branch 'master' into multi-year-one

6e5ac7d

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c0e5ee

for more information, see https://pre-commit.ci

fneum and others added 5 commits April 4, 2025 09:36

use highs-default for myopic and perfect test

52d40d0

Merge branch 'multi-year-one' of github.com:pypsa/pypsa-eur into mult…

ccc21da

…i-year-one

ci: do not share resources, go back to highs-simplex, test biomass im…

be9fa8d

…port myopic

fix: add_co2_sequestration_limit nyears with perfect foresight

663e709

[pre-commit.ci] auto fixes from pre-commit.com hooks

04724d0

for more information, see https://pre-commit.ci

fneum added this to the v2025.04.0 milestone Apr 4, 2025

fneum merged commit 037b196 into master Apr 4, 2025

fneum deleted the multi-year-one branch April 4, 2025 10:35

cpschau mentioned this pull request Apr 16, 2025

Complement era5 files for data bundle retrieval #1648

Merged

7 tasks

fneum mentioned this pull request Sep 4, 2025

feat: add river-water and sea-water sourced heat pumps #1618

Merged

13 tasks

Navigation Menu

Uh oh!

Multiple weather years in one optimisation problem - #1613

Multiple weather years in one optimisation problem#1613
fneum merged 29 commits into
masterfrom
multi-year-one

fneum commented Mar 26, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

cpschau left a comment

Uh oh!

Uh oh!

fneum commented Apr 4, 2025

Uh oh!

coroa commented May 14, 2025

Uh oh!

tgilon commented May 14, 2025

Uh oh!

cpschau commented May 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

fneum commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

cpschau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fneum commented Apr 4, 2025

Uh oh!

coroa commented May 14, 2025

Uh oh!

tgilon commented May 14, 2025

Uh oh!

cpschau commented May 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

fneum commented Mar 26, 2025 •

edited

Loading