Introduction to using HTCondor to run distributed compute Jobs and Workflows on Servers, Clusters, Grids, or Clouds

HTCondor (http://htcondor.org) is an open-source high-throughput computing software framework for coarse-grained distributed parallelization of computationally intensive tasks (jobs). It can be used to manage workloads on a single server, a cluster of computers, public cloud resources, or even national computing grids like the Open Science Grid (http://opensciencegrid.org/).

This workshop will introduce the concept of High Throughput Computing and how to submit large batches of jobs and also job workflows (job pipelines) to HTCondor, which will be of interest to end users. We will discuss the architecture of the system, and participants will create a unified compute cluster.

If you wish to participate in the hands-on exercises (not required, but recommended), you will need a laptop with WiFi, an SSH client (such as PuTTY if using Windows), familiarity with the Linux command-line environment (cd, less, cp, rm, mkdir, etc), and the ability to edit a file using a Linux terminal editor like vi, vim, or nano.

Edit Course

Job Submission Exercises

• Run Jobs
• Remove Jobs
• Transferring files with your job
• Use queue N to submit multiple jobs
• Submit jobs with queue matching
• Submit jobs with queue from
• Submit a job with a Docker image
• Submit a million jobs!

DAGMan Exercises

• Coordinating a Set of Jobs with a Simple DAG
• A Brief Detour Through the Mandelbrot Set
• A More Complex DAG
• Handling a DAG That Fails

Slide Decks

• User Tutorial Slides
• Architecture and Administrator Slides

Edit Metadata