Scientific workflow is an important technique used in many simulation and data analysis applications. In particular, workflows automate high-throughput / high-complexity computing applications, enable code and data reuse and provenance, provide methods for validation and error tracking, and exploit application concurrency using distributed computing resources. The goal of this tutorial is to learn composing and running workflow applications using the FireWorks workflow environment (FireWorks). The GridKa School Fireworks tutorial is available on GridKa School Fireworks GitLab.
The aim of this tutorial is to prepare participants for own deployment of the ELK stack for collecting and log analysis. Participants will be taken step by step through installation, configuration and use of all needed tools for modern log analysis.
Explaining the basic concepts of the Erlang programming language and testing these by a simple game using Erlang.
Machine learning with deep neural networks has seen tremendous advances in the last few years and is now the state-of-the-art method in a broad range of fields, including computer vision and natural language processing. Deep learning shines when dealing with large bodies of high-dimensional, complex data and is thus well suited for pushing the limits in high-energy particle and astroparticle physics. This tutorial will introduce you to the fundamental concepts and some advanced techniques in deep learning and give you a hands-on introduction to designing, training and evaluating neural networks in supervised classification and regression tasks. As deep learning framework we will use TensorFlow via the Python interface (some familiarity with Python is assumed).
This course aims to equip people of limited knowledge in C++ with both, a better understanding of the scope of C++ and practical, immediately useful hints to improve code with respect to correctness, readability and maintainability. The discussed topics are mostly adapted from the celebrated "Effective... " book series from Scott Myers.
Python provides a rich ecosystem of open-source software for mathematics, science, and engineering. This tutorial will introduce you to the fundamental packages of the SciPy stack. You will learn how-to: perform fast numerical calculations in N dimensions using NumPy, analyse your data using Pandas, and visualize the results using Matplotlib. The exercises will be performed in the Jupyter Notebook environment, which you can access through your web browser. The complete course is based on Jupyter Notebooks, which contain the examples, slides and the exercises. The notebook are available on github: https://github.com/thomaskeck/GridKaSchool
HTCondor is a specialized workload management system for compute-intensive jobs.
The scope of this Workshop covers the architecture as well as the concepts of kubernetes. You are taken on a journey from your first container startup to more complex setups inside the kubernetes cluster. Basic understanding of unix and the cli is required as well as a ssh capable computer, since you need to connect via ssh to our learning environment.
Python has been widely adopted in academia, science and beyond. As the language is easy to pick up, many people use it for scripting, configuration, and prototyping. At the same time, its flexibility, breadth of application and huge ecosystem make it a powerful tool even for large projects. This course focuses on software development with Python beyond simple scripting and prototyping. Topics range from best practices for programming small and large projects, to organising and packaging frameworks as well as developing high performance applications. Each topic is presented as a mixture of general lectures and hands-on exercises. The course targets intermediate Python developers who are familiar with the language itself.
The Docker tutorial will walk you through the basic steps of setting up a Docker environment on your machine. There will be a series of exercises that will detail the various concepts presented during the plenary talk which are critical that you understand for the later part of the tutorial. The final goal of the tutorial is to build and deploy a couple of containers that replicate the usual analysis workflow in High Enery Physics: you will have a container running a XRootD server providing the storage for the data and a different container that runs the ROOT framework where you will do your analysis. The tutorial will discuss into depth the concepts of port forwarding, volumes and resource management in the context of containers with a focus on understanding the advantages of containers over traditional virtual machines.
In this workshop, we will introduce the basics of programming in Go and then work our way up to concurrency programming with this relatively new language. We'll start with the usual "Hello World" program, introduce functions, variables, packages and then interfaces. Then, we will tackle the two main tools at the disposal of the Go programmer (colloquially known as a gopher): the channels and the goroutines. This will be done by implementing a small peer to peer application transmitting text messages over the network. The workshop wraps up with a whirlwind tour of scientific and non-scientific libraries readily available, and prospects/news about the next Go version.
External references: https://github.com/sbinet/gridka-go-2017
OpenACC is a directive-based programming model for highly parallel systems, which allows for automated generation of portable GPU code. In this tutorial, we will get to know the programming model with examples, learn how to use the associated tools environment, and incorporate first strategies for performance optimization into our programs.
Based on experiences from projects in academia and industry, this tutorial introduces tools and concepts to enable maintainable software projects in collaborative environments. While the tutorial tries to give a broad overview on different topics, we also provide in-depth information. We cover topics such as version control and organisation of software with git, concepts of unit testing and test-driven development, tools supporting continuous integration as well as the integration into wikis and ticket systems. Throughout this tutorial you will learn how to efficiently integrate different tools and concepts to enable maintainable software. After the course, you will have a basic setup that can be adapted to your specific needs.
The course exemplarily focuses on Python and relevant tools to support collaborative software development.
Learn about relational and non-relational databases, and how you can exploit them for fun and profit with large data volumes.
C++ Concurrency Training Material. Note that all course material is uploaded here, however the material presented in the GridKA workshop consisted of sections 5, 6, 8 and 9 only.
External references: https://github.com/graeme-a-stewart/cpp-concurrency