basic_users_guide_xces_new

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
basic_users_guide_xces_new [2024/01/08 14:41] – etormabasic_users_guide_xces_new [2025/05/19 11:50] (current) – [Installing the freva library] etorma
Line 1: Line 1:
 ===== Getting familiar with the XCES Platform ====== ===== Getting familiar with the XCES Platform ======
  
 +<WRAP round info>
 +  * For a **quick tour** through Freva and Freva Plugins please [[https://freva-clint.github.io/Talks/Freva_Workshop_2024/index.slides.html|take a look here]].
 +  * You also have some **hands-on exercises** to get yourself familiarised with freva and plugins [[https://gitlab.dkrz.de/freva/freva_workshop|here]].
 +</WRAP>
  
 The XCES platform is an instance of the Free Evaluation System Framework (Freva), designed by the Freva team and adapted for the scientists of the ClimXtreme project by Module D. Below you will find relevant information on how to get started with the system.  The XCES platform is an instance of the Free Evaluation System Framework (Freva), designed by the Freva team and adapted for the scientists of the ClimXtreme project by Module D. Below you will find relevant information on how to get started with the system. 
  
-The system can used either through the [[basic_users_guide_xces_new#Access through the shell|shell]], [[basic_users_guide_xces_new#Access through the web|website]], or [[basic_users_guide_xces_new#access_through_python_module_pypi_jupyter|its python module (e.g. for Jupyter)]]. We will give an overview of the capabilities of XCES on all platforms and also the necessary steps to set a working environment in the shell that will allow us [[https://freva-clint.github.io/freva/FrevaCli.html#add-new-data-to-the-databrowser|to share data]] with the rest of the ClimXtreme community or test our [[list_of_plugins|plugins]] in development stage. +The system can used either through the [[basic_users_guide_xces_new#Access through the shell|shell]], [[basic_users_guide_xces_new#Access through the web|website]], or [[basic_users_guide_xces_new#access_through_python_module_pypi_jupyter|its python module (e.g. for Jupyter)]]. We will give an overview of the capabilities of XCES on all platforms and also the necessary steps to set a working environment in the shell that will allow us [[https://freva-clint.github.io/freva/FrevaCli.html#add-new-data-to-the-databrowser|to share data]] with the rest of the ClimXtreme community or test our [[https://gitlab.dkrz.de/bm1159/plugins4freva/gitlab-profile/-/blob/main/README.md|plugins]] in development stage.  
 + 
  
  
Line 189: Line 195:
 BUT be aware that there is currently NO API to call freva functionalities within R, so it will only work as an additional R kernel, nothing more. BUT be aware that there is currently NO API to call freva functionalities within R, so it will only work as an additional R kernel, nothing more.
  
-\\ +===== Creating your own conda/mamba environment with the freva library (e.g. for plugins) ===== 
-If, on the other hand you want to use freva as a python library alongside others (e.g. in a conda environment) then you can:+ 
 +We **strongly** suggest creating your own environment to run your plugin code. There are several reasons for that, among them: 
 +  - to not rely on centrally offered environments, making the plugins easier to mantain and port other instituions / newer HPCs (e.g. the successor of Levante) 
 +  - to offer flexibility (e.g. via Makefile) to install other packages / binaries via other means (e.g. directly CRAN, other means) 
 +  - to allow more reproducibile environment (ideally if e..g the library versions are set frozen). 
 + 
 +A way of doing it is creating a ''mamba/conda'' environment, you can use the conda/mamba installer provided by loading freva module for that (''module load clint xces'')), e.g.: 
 +  * Create a New Environment in Default Location (i.e. '~/.conda/envs/' be aware that your home only has 25GB!): 
 +<code> 
 +conda create --name myenv 
 +</code> 
 +to find your conda env you simply do ''conda env list'' and ''conda activate myenv'' (you may need to add something to your .bashrc/.zshrc) 
 +  * Create a New Environment in a Specific Folder (e.g. in a plugin repo): 
 +<code> 
 +conda create --prefix /path/to/myenv 
 +</code> 
 +  * Create a New Environment with specific libraries and e.g. python version: 
 +<code> 
 +conda create --name myenv python=3.8 numpy pandas 
 +</code> 
 + 
 +You can do more elaborate installations for e.g. your plugin environments, please [[https://freva-clint.github.io/freva/developers_guide.html|take a look at this short guide for it]]. 
 + 
 +<WRAP round important> 
 +**NOTE**: since 2024 DKRZ does not allow access to ''default'' channels in conda [[https://legal.anaconda.com/policies/en/#terms-of-service| due to changes in Anaconda's terms of services]]. You need to make sure you will not use it or you will get an ''HTTP error'' message while trying to create an env. Options: 
 +  * use flag '' --override-channels'' so it does not search default. 
 +  * create ''$HOME/.condarc'' file with: 
 +<code> 
 +channels: 
 +  - conda-forge 
 +</code> 
 +  * (similar to 2): 
 +<code> 
 +# check what is currently set 
 +conda config --show channels 
 + 
 +# remove what you find  
 +conda config --remove channels defaults 
 + 
 +# add conda-forge 
 +conda config --add channels conda-forge 
 +</code> 
 + 
 +</WRAP> 
 + 
 + 
 + 
 + 
 + 
 +==== Creating a jupyter kernel out of your environment ==== 
 + 
 +For that, you will need to previously install the ''ipykernel'' library via e.g.: 
 + 
 +<code python> 
 +conda create --name myenv python=3.8 ipykernel # with conda 
 +python -m pip install ipykernel # with pip 
 +</code> 
 + 
 +And then, given you are using the python binary of your environment: 
 + 
 +<code python> 
 +python -m ipykernel install --user --name=$(KERNEL_NAME) --display-name $(KERNEL_NAME) 
 +</code> 
 + 
 +==== Installing the freva library ==== 
 + 
 + 
 +If you want to use freva as a python library alongside others (e.g. in a conda environment) then you can:
   - install the package via pypi ''pip install freva'' or via conda/mamba ''mamba install -c conda-forge freva''   - install the package via pypi ''pip install freva'' or via conda/mamba ''mamba install -c conda-forge freva''
   - add the environment config of XCES in your script, e.g.:   - add the environment config of XCES in your script, e.g.:
 <code python> <code python>
-import os,sys 
-os.environ["EVALUATION_SYSTEM_CONFIG_FILE"] = "/work/bm1159/XCES/freva/evaluation_system.conf" # this is for the environment 
-os.environ["EVALUATION_SYSTEM_CONFIG_DIR"] = "/work/bm1159/XCES/freva" # this is for the environment 
 import freva import freva
 +freva.config(config_file="/work/bm1159/XCES/freva/evaluation_system.conf")
 </code> </code>
 +that should work on a python script, python env console and jupyter notebook (e.g. [[https://jupyterhub.dkrz.de|DKRZ's jupyterhub]].
 +
 +Incidentally, you can load plugins using the same command, e.g.:
 +<code python>
 +import freva
 +freva.config(config_file="/path/to/the/freva/config/file.conf",
 +             plugin_path="~/freva/mypluing,the_plugin")
 +</code>
 +
  
 +===== Using XCES/Freva R/Python environment to locally install your own packages [NOT RECOMMENDED] =====
  
-===== Using XCES/Freva R/Python environment to locally install your own packages =====+<WRAP round info> 
 +Not recommended! Your libraries would depend on the current Freva python/R environment that will eventually be updated and, hence, get obsolete. 
 +</WRAP>
  
 xces/Freva ships with its own R and Python environment, with some basic set of libraries included already. These environments can be used out-of-the-box. To check the R/Python version please: xces/Freva ships with its own R and Python environment, with some basic set of libraries included already. These environments can be used out-of-the-box. To check the R/Python version please:
Line 243: Line 326:
  
 The package will then be downloaded, built, including dependencies, and installed under ''$HOME/.local/lib/python3.X/site-packages''. The package will then be downloaded, built, including dependencies, and installed under ''$HOME/.local/lib/python3.X/site-packages''.
 +
 +**NOTE**: conda/mamba does not allow to do something similar to that, so if the package you want to append can only be installed via conda then you will need to create your own environment.
  
 ==== 2. Installing R packages locally ==== ==== 2. Installing R packages locally ====
Line 267: Line 352:
  
 You can get some more information on e.g. how to build packages locally (I adapted some of the above mentioned text from there) [[https://nesi.github.io/hpc_training/lessons/maui-and-mahuika/installing-packages-locally|here]]. You can get some more information on e.g. how to build packages locally (I adapted some of the above mentioned text from there) [[https://nesi.github.io/hpc_training/lessons/maui-and-mahuika/installing-packages-locally|here]].
- 
-===== Creating your own conda/mamba environment with the freva library ===== 
- 
-If, on the other hand you want to use freva as a python library alongside others (e.g. in a conda environment) then you can: 
-  - install the package via pypi ''pip install freva'' or via conda/mamba ''mamba install -c conda-forge freva'' 
-  - add the environment config of XCES in your script, e.g.: 
-<code python> 
-import os,sys 
-os.environ["EVALUATION_SYSTEM_CONFIG_FILE"] = "/work/bm1159/XCES/freva/evaluation_system.conf" # this is for the environment 
-os.environ["EVALUATION_SYSTEM_CONFIG_DIR"] = "/work/bm1159/XCES/freva" # this is for the environment 
-import freva 
-</code> 
- 
  
 ===== Additional info ===== ===== Additional info =====
  • basic_users_guide_xces_new.1704724882.txt.gz
  • Last modified: 2024/01/08 14:41
  • by etorma