Troubleshooting#

Running the workflow on the machine of your choice may turn out to be not as easy as you or we would hope. We often see issues with reproducing the necessary conda execution environments and with unsuitable local configuration. Do not worry: running into these problems is completely normal and in many cases a solution is easy to reach. Here are some steps we advise you to go through:

  1. Read through our frequently asked questions below.
  2. If you did not find an answer to your question, check Snakemake's documentation.
  3. If you still did not find an answer to your question, open an issue on our issue tracker.

Frequently Asked Questions#

Where do I find the rule and/or script that generated a specific file?#

If you wonder exactly how one file of the model has been generated, you need to find the corresponding Snakemake rule (step of the workflow) and Python script. Each file is generated by exactly one rule and usually, but optionally, a script. Once you found the rule, you can identify the corresponding script.

Suppose you want to learn how the fictional model file build/data/national/preprocessed-hydro.nc is generated. Here is the fictional rule generating this file:

rule preprocess_hydro_data:
    message: "Preprocess the hydro data." # You will see this message in the Snakemake log.
    output: "build/data/{resolution}/preprocessed-hydro.nc" # This is the file you are looking for.
    script: "scripts/hydro/preprocess.py" # This is the Python script generating the file.
    ... # Other rule components.

If you are not familiar with the structure of the repository, the easiest way to find the rule is to do a file search in the entire repository using your local IDE or online on GitHub. In the example, you would search for build/data/national/preprocessed-hydro.nc but you would not find the rule because of the {resolution} wildcard in the output of the rule. Do not worry if you do not know what this means. If you do not find the file, try searching for parts of the filename. In this example, searching for preprocessed-hydro.nc will let you find the rule.

Once you found the rule, it will point you to the Python script generating the file. In this example, it is scripts/hydro/preprocess.py. You can now inspect this file to learn about the details of how hydro data is preprocessed and may as well debug the rule execution for more details.

How can I debug a workflow step?#

Most of our workflow steps are Python scripts and you can debug them with default debugging methods. For example, you may include a breakpoint() statement in a line in which you want to drop into the debugger pdb.

How long should I expect the workflow to run?#

The duration of the execution of the entire workflow depends on several factors like the compute performance of your machine, your network connection, and the response times and availability of data providers. In general, expect the workflow to run for several hours -- in particular when you run it the for the first time and all datasets must be downloaded. Once you have downloaded all datasets once -- and there is no need for a re-download -- the workflow duration is shorter.

A particularly slow workflow step can be the step which downloads runoff data from the Copernicus Data Store. This is not because the data is particularly large or because the server is particularly slow, but because you may have to queue for the data, depending on the number of other users at the time of your request.

Snakemake's execution of the workflow seems frozen. Is it?#

When the Snakemake log is not proceeding for a while you may wonder whether Snakemake froze. We have never experienced Snakemake to freeze, so chances are high that Snakemake is simply waiting for a job to terminate. You can find out which job it is waiting for in the following way: in the Snakemake log (the command line output) you can see which jobs have been started and which jobs terminated. This will tell you which jobs are still running.

Only if you are running on a cluster and you are using our cluster interface, you may have run into a known issue: the cluster scheduler could have killed your jobs without Snakemake noticing. Run bjobs on the cluster to verify whether the jobs are still running. If they are not, the cluster has killed your jobs, likely due to resource overuse. You can look up the reason by reading the corresponding log file in build/logs. Terminate Snakemake manually, increase the requested resources, and only then restart the workflow.

I have updated configuration parameters but this seems to have no effect. Did I do something wrong?#

Yes and no. This is a caveat of Snakemake: Snakemake detects the parameter changes but will not automatically rebuild the affected parts of the models. You will have to manually rebuild these parts, see Customisation of the workflow.

Can you run the workflow for several different weather years?#

Not out-of-the-box, but this feature is on our roadmap.

How to run different configurations of the workflow in parallel?#

There is no officially supported way to run the workflow for more than one configuration in parallel in the same directory. The least error-prone way to do this is to maintain individual copies of Euro-Calliope with their own configuration files.

The workflow fails when trying to create a conda environment. How can I resolve this?#

We rely on the conda-forge repository to generate reproducible conda environments. This approach has it limits and may lead to a situation in which you are unable to reproduce the execution environments in the future. You will know this happened when Snakemake fails in the initial step of preparing the environments.

Unfortunately, there is no general way to resolve this. Here are a few hints that may solve the problem for you.

  • If possible, update to the newest Euro-Calliope version. Either use the latest release workflow DOI, or use the latest development version. This may resolve the problem.
  • Isolate the problem by manually creating the environment using mamba: mamba env create -f envs/<env-name>.yaml. Mamba generates more helpful error messages than conda and this may help you understanding the conflict better. Once you have identified the conflicting packages, update (or downgrade) them in the environment file.
  • If you cannot solve the problem locally and you are using the latest release or development version of Euro-Calliope, please file a bug report so we can fix the problem.

Can I use autocompletion, live validation, and tooltips when working with the workflow configuration file?#

Yes, absolutely. We rely on the de-facto standard JSON schema which most IDEs support out-of-the-box or with plug-ins. For VS Code, use the following configuration in your workspace configuration (settings.json):

{
    "yaml.schemas": {
        "config/schema.yaml": "config/*.yaml",
    }
}