martin-rdz

After having achived to build the windows wheels for pyLARDA on anacoda, I tried to tackle the linux version. There it’s a bit more complicated, as the wheels are kind of distribution dependent (and are not accepted by pypi).

auditwheel (https://github.com/pypa/auditwheel) allows to put the external shared libraries into the wheel itself, however the have to be compiled with a rather ‘old’ version to be cross-platform compatible.

The manylinux project (https://github.com/pypa/manylinux) providing docker images to simplify this step:

docker pull quay.io/pypa/manylinux2014_x86_64
docker images #to get the id

The source code needs to be available somewhere in the current path, afterwards fire up docker:

docker run -it -v $(pwd):/io 90ac8ec

Then inside the docker container:

cd io/larda
PYBIN=/opt/python/cp38-cp38/bin/
# pyLARDA requires cython as a prerequisite
${PYBIN}/pip install cython
${PYBIN}/pip install .
${PYBIN}/python -m pyLARDA
${PYBIN}/python setup.py bdist_wheel
cd dist/
auditwheel show pyLARDA-3.3.2-cp38-cp38-linux_x86_64.whl
auditwheel repair pyLARDA-3.3.2-cp38-cp38-linux_x86_64.whl

Now the manylinux wheels should be available in the dist/wheelhouse/ directory.

 

 

 

My latest encounter with joys of input format change: With the change to GFSv16 in March 2021 additional height levels were added to the grib output. Especially heights below 1hPa. What is generally good news, is bad news for Flexpart 10.4 (as of 20 Nov 2019). The additional levels cause incompatability with the readwinds_gfs.f90. Public information on this issue is rare. There is an Issue in Flexparts ticketing system (https://www.flexpart.eu/ticket/305) but thats more related to the mean-wind trajectory module Flextra.

One suggestion is to drop the additional variables with wgrib2, which is somehow dissatisfying (and I could not figure it out on the short run). The only alternative posed ist, to switch to the dev branch (The link in the issue points to a version 9.3.2 legacy branch). So, for 10.4 dev you want to pull git clone https://www.flexpart.eu/gitmob/flexpart --branch dev --single-branch. With that version, Flexpart switches from grib_api to eccodes, so on ubuntu the libraries apt-get install libeccodes-tools libeccodes-dev are required.

At least for my docker setup, i needed to adapt the makefile:

INCPATH1 = /usr/include
INCPATH2 = /usr/include
LIBPATH1 = /usr/lib
F90 = gfortran

Then, compiling worked sucessfully and the binary could handle the new GFS files.

Slightly related: natural earth cartopy downloads

Seems like the default natural earth mirror is temporarily unavailable (e.g., http://naciscdn.org/naturalearth/110m/physical), but there is this fix: https://github.com/SciTools/cartopy/issues/1298#issuecomment-920843582

 

For a recent project, I wanted to include the Lagrangian particle dispersion model FLEXPART (more on that) into an easy-to-deploy project. The analysis part of that project runs on python, but uses some nifty geography libraries, which in turn require C and Fortran libraries. So, I already put this part into a docker container. FLEXPART provides the actual input for the analysis part, so it seemed logical to also include that part.

Not too long ago version 10 came available, also including netCDF output. However, with added dependencies, compiling the source became a bit more complicated. Let’s have a look at the Dockerfile:

FROM ubuntu:18.04


ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get update && apt-get install -y \
  language-pack-en openssh-server vim software-properties-common \
  build-essential make gcc g++ zlib1g-dev git python3 python3-dev python3-pip \
  pandoc python3-setuptools imagemagick\
  gfortran autoconf libtool automake flex bison cmake git-core \
  libjpeg8-dev libfreetype6-dev libhdf5-serial-dev \
  libeccodes0 libeccodes-data libeccodes-dev \
  libnetcdf-c++4 libnetcdf-c++4-dev libnetcdff-dev \
  binutils  python3-numpy python3-mysqldb \
  python3-scipy python3-sphinx libedit-dev unzip curl wget
  
  
# replaced 'libpng12-dev' by libpng-dev
RUN add-apt-repository ppa:ubuntugis/ppa \
  && apt-get update \
  && apt-get install -y libatlas-base-dev libpng-dev \
     libproj-dev libgdal-dev gdal-bin  

RUN add-apt-repository 'deb http://security.ubuntu.com/ubuntu xenial-security main' \
  && apt-get update \
  && apt-get install -y libjasper1 libjasper-dev libeccodes-tools libeccodes-dev

#ENV HTTP https://confluence.ecmwf.int/download/attachments/45757960
#ENV ECCODES eccodes-2.9.2-Source

#
# Download, modify and compile flexpart 10
#
RUN mkdir flex_src && cd flex_src \
  && wget https://www.flexpart.eu/downloads/66 \
  && tar -xvf 66 \
  && rm 66 \
  && cd flexpart_v10.4_3d7eebf/src \
  && cp makefile makefile_local \
  && sed -i '74 a INCPATH1 = /usr/include\nINCPATH2 = /usr/include\nLIBPATH1 = /usr/lib\n F90 = gfortran' makefile_local \
  && sed -i 's/LIBS = -lgrib_api_f90 -lgrib_api -lm -ljasper $(NCOPT)/LIBS = -leccodes_f90 -leccodes -lm -ljasper $(NCOPT)/' makefile_local \
  && make -f makefile_local

ENV PATH /flex_src/flexpart_v10.4_3d7eebf/src/:$PATH

# pip install some more python libraries

The interesting part happens in L34-42. The source is downloaded and unpacked. Then the makefile  needs to be modified to find all libraries (which were installed with apt-get before). Sed is of great use here. First a line is added to the makefile_local and another line is searched and replaces. Finally make is executed to compile FLEXPART.

Now the container can be ran and FLEXPART be executed from within. At least for my application with a couple of thousand particles being simulated, performance is sufficient and a small penalty is outweigh by the ease of deployment.

Further reading:

• Fortran and Docker: How to Combine Legacy Code with Cutting-Edge Components
• Docker Container based PaaS Cloud Computing Comprehensive Benchmarks using LAPACK

“Hell is other people’s code” – while trying to reprocess some data on our server, a colleagues script did not produce the desired output. The problem was, that the location attribute was missing in the input netCDF files [1]. Which leads us to the code, that checked the attributes:

#next the formatting I'm supposed to check it against:

(key_for_range_dimension,key_for_time_dimension,key_for_azimuth_variable,key_for_elevation_variable,key_for_range_variable,key_for_unix_time_variable,key_for_doppler_velocity_variable,key_for_doppler_velocity_error_variable,key_for_nyquist_velocity_variable)=variable_name_key_setting_function(location_we_want,system_we_want,location_info,formatting_translator_dictionary,verbose)
       
#this is a logical statement that includes everything, using brackets to break up inline statements and make it easier to see
formatting_is_correct=(
               'location' in that_files_attributes
               ) and (
                              'system' in that_files_attributes
                              ) and (
                                            key_for_range_dimension in that_files_dimensions
                                            ) and (
                                                           key_for_range_variable in that_files_variables
                                                           ) and (
                                                                         key_for_time_dimension in that_files_dimensions
                                                                          ) and (
                                                                                        key_for_unix_time_variable in that_files_variables
                                                                                        ) and (
                                                                                                        key_for_azimuth_variable in that_files_variables
                                                                                                        ) and (
                                                                                                                       key_for_elevation_variable in that_files_variables
                                                                                                                       ) and (
                                                                                                                                     key_for_doppler_velocity_variable in that_files_variables
                                                                                                                                     )

print("Check format keys:")
print(('location' in that_files_attributes),
    ('system' in that_files_attributes),(key_for_range_dimension in that_files_dimensions),
    (key_for_range_variable in that_files_variables),(key_for_time_dimension in that_files_dimensions),
    (key_for_unix_time_variable in that_files_variables),
    (key_for_azimuth_variable in that_files_variables),
    (key_for_elevation_variable in that_files_variables),
    (key_for_doppler_velocity_variable in that_files_variables))

[You might want to expand the code snippet to fully appreciate the >380 column lines]

At least for me, that looked like a classical broken window. The final print statement is already a owned to the debugging efforts. Two points, that I consider problematic:

1. 1. insanely long lines (not the only occurrence in that script but a prominent one)
   2. the long and nested boolean statement

Both make it hard to understand, debug and modify the code. Let’s have see, what we quickly can do about it.

var_name_key_setting = variable_name_key_setting_function(
               location_we_want, system_we_want, location_info,
               formatting_translator_dictionary, verbose)
# unpack the function return
(key_for_range_dimension, key_for_time_dimension, 
 key_for_azimuth_variable, key_for_elevation_variable, 
 key_for_range_variable, key_for_unix_time_variable, 
 key_for_doppler_velocity_variable, 
 key_for_doppler_velocity_error_variable,
 key_for_nyquist_velocity_variable) = var_name_key_setting
       
formatting_is_correct_list = [
    'location' in that_files_attributes, 
    'location in attributes',
    'system' in that_files_attributes, 
    'system in attributes',
    key_for_range_dimension in that_files_dimensions, 
    'key range dim in files_dimensions',
    key_for_range_variable in that_files_variables,
    'key range var in files_variables',
    key_for_time_dimension in that_files_dimensions,
    'key time dim in files_dimensions',
    key_for_unix_time_variable in that_files_variables,
    'key unix_time_var in files_variables',
    key_for_azimuth_variable in that_files_variables,
    'key azimuth var in files_variables',
    key_for_elevation_variable in that_files_variables,
    'key elevation var in files_variables',
    key_for_doppler_velocity_variable in that_files_variables,
    'key doppler_vel_var in files_variables',
       ]
formatting_is_correct = all(formatting_is_correct_list[::2])

print("Check format keys:", formatting_is_correct)
if not formatting_is_correct:
       [print('  ', e[0], e[1]) for e in \
            zip(formatting_is_correct_list[0:-1:2], 
                formatting_is_correct_list[1::2])]

Still not perfect, but a bit more explicit and easier to read. In fact the [::2] could also be omitted, as a string also evaluates to True in all().

Maybe it’s time to setup a compulsory linter or at least make PEP 8 and the Google Python Style Guide required reading.

[1] actually required a second pair of eyes to figure out…

Recently I wanted to configure a gunicorn backend at our newest server. The aim was to run a flask-based python application using a virtual environment in a users home. The machine is running recent Red Hat 8 and the setup of systemd was straightforward and well documented in the internet (e.g. here). However, after

systemctl daemon-reload
systemctl enable ourservice
systemctl start ourservice

a strange permission denied error occurred (also journalctl helps in that context).

After rechecking the file permission several times I finally remembered, that Red Hat now runs runs SELinux. It allows even finer access control for files, resources and actions using policies. The current status of SELinux can be shown with sudo sestatus. For testing reasons, all policies can be set to permissive using sudo setenforce permissive. The contexts of a file can be inspected with ls -Z file1.

To debug an process that fails, the /var/log/audit/audit.log file is the first place to look at. The audit2why tool translates the error messages to a more understandable format:

>
sudo cat /var/log/audit/audit.log | grep gunicorn | audit2why

Afterwards the audit2allow utility can generate rules, that would allow the denied operations:

>
sudo cat /var/log/audit/audit.log | grep gunicorn | audit2allow -M custom_rule

These rules are now stored in the format of an type enforcement (custom_rule.te) file and a policy package which can be activated using semodule -i custom_rule.pp.

Likely this process has to be repeated, as different kinds of operations might be denied.

For our setup the final type enforcement file looked like

module custom_rule 1.0;
 
require {
        type init_t;
        type unconfined_exec_t;
        type user_home_t;
        class file { append execute execute_no_trans ioctl map open read setattr };
        class lnk_file { getattr read };
}
 
#============= init_t ==============
#!!!! This avc is allowed in the current policy
allow init_t unconfined_exec_t:file { execute open read };
allow init_t user_home_t:file setattr;
 
#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow init_t user_home_t:file map;
 
#!!!! This avc is allowed in the current policy
#!!!! This av rule may have been overridden by an extended permission av rule
allow init_t user_home_t:file { append execute execute_no_trans ioctl open read };
 
#!!!! This avc is allowed in the current policy
allow init_t user_home_t:lnk_file { getattr read };

The type enforcement rule can be compiled manually

checkmodule -M -m -o custom_rule.mod custom_rule.te
semodule_package -o custom_rule.pp -m custom_rule.mod
sudo semodule -i custom_rule.pp

Further useful resources

• A sysadmin’s guide to SELinux: 42 answers to the big questions
• Red Hat documentation on SELinux
• Similar fix for ansible
• Related problems with socket permissions (stackoverflow)

 

Disclaimer

When used correctly, SELinux adds a lot to your server’s security. When you run into problems with certain commands being denied, you should first make sure that you truly understand what causes the error. Chances are very high that SELinux complains for a reason. Often you can avoid the problem altogether by rethinking where you put which files. When you are absolutely sure that you need to build a new policy package, do yourself a favor and research thoroughly what each added rule does – it is only too easy to create security holes which would defeat SELinux’ purpose.

Michael Trojanek: How to compile a SELinux policy package

UPDATE

• 2012 Red Hat Summit: SELinux For Mere Mortals (Talk by Thomas Cameron)

Recently I wrote down a quick overview on data structures for scientifc programming in python, focusing on built-in and numpy.

It is available here: https://gist.github.com/martin-rdz/1018e3b19601edeba52eeac91520d711