Parameterized jobs on Nomad
Nomad parameterized jobs enable dynamic runtime behaviors from a single job specification. These specialized batch-type jobs act less like regular Nomad jobs and more like functions. They are reusable, can accept parameters, and run only when invoked.
Parameterized jobs provide three key benefits to end users.
Creating instances of the job without resubmitting the job specification.
Providing dynamic run-time values as parameters to the job during dispatch.
Proving, when the job allows, an opaque payload that Nomad will write to one or more of the job's tasks.
Nomad registers, but does not run, a parameterized job when you submit it to
the cluster. To run a parameterized job, end-users invoke—or dispatch—it
with the nomad job dispatch
command or Nomad Job
Dispatch API.
During dispatch, the user provides the parameter values and payload. Nomad combines these values with the registered job, creates a dispatched instance, and returns the scheduling information to the end-user. Once dispatched, the created instances are handled like regular Nomad batch jobs.
Challenge
In this tutorial, you will learn the fundamentals of Nomad parameterized jobs. You will take a Nomad batch job that renders a template, and
- Convert it to a parameterized job
- Enhance it with optional and required parameters
- Define default values for optional parameters
- Handle dispatch payloads
These fundamental practices can be used to create more complex parameterized workloads in your own environment.
Prerequisites
- Nomad v1.0 or greater
- Nomad dev agent or Nomad cluster
Configure your learning environment
Fetch the tutorial content
This tutorial uses content provided in the hashicorp/learn-nomad-jobspec
repository on GitHub. You can download a ZIP archive directly or use git
to
clone the repository.
$ wget https://github.com/hashicorp-education/learn-nomad-jobspec/archive/release.zip
Unarchive the downloaded release.
$ unzip release.zip
The unzipping process creates a directory named learn-nomad-jobspec-release
.
Change into it and into the tutorial's folder.
$ cd learn-nomad-jobspec-release/parameterized
Configure tutorial environment
This tutorial uses HCL2 variables to reduce the amount of editing that you need
to do to the job specification based on your environment. The defaults are
optimized for a Linux/macOS boxes targeting a local Nomad dev agent using the
default datacenter value of dc1
. These values can be customized using
environment variables before running the job file.
Override default task driver (optional)
This tutorial uses either the raw_exec or exec task driver (raw_exec by default) to render a text file and output it to standard output.
Because Windows and macOS do not support the exec driver and the raw_exec driver is not enabled by default on non-dev agents, this job uses an HCL2 variable to let you set your preferred driver without editing the job.
The driver variable defaults to raw_exec, suitable for all learners using a
dev agent. If you need to use the exec driver instead, set the
NOMAD_VAR_driver
environment variable to exec
.
$ export NOMAD_VAR_driver=exec
Override default datacenter (optional)
The job also allows you to specify your datacenter value if it differs from the
default value of dc1
. If you are using a non-default datacenter value,
set the NOMAD_VAR_datacenter
environment variable.
If you are unsure of a valid datacenter value for your target cluster or dev
agent, run the nomad node status
command to list all the datacenters
available.
$ nomad node status
ID DC Name Class Drain Eligibility Status
d715f8b4 mydc nomad-client-1.node.consul <none> false eligible ready
14ab9290 mydc nomad-client-2.node.consul <none> false eligible ready
0f357b26 mydc nomad-client-3.node.consul <none> false eligible ready
In this sample output, all the clients are in a datacenter named mydc
,
based on this output you would run:
$ export NOMAD_VAR_datacenter=mydc
Run the starting job
To get you started, the tutorial repository includes a Nomad batch job
specification that you will use as a base to build on. Use the nomad job run
command to run the start.nomad
job file.
$ nomad job run start.nomad
==> Monitoring evaluation "68b8b7dc"
Evaluation triggered by job "template"
Allocation "cb1d80a1" created: node "66d36f06", group "renderer"
==> Monitoring evaluation "68b8b7dc"
Allocation "cb1d80a1" status changed: "pending" -> "complete" (All tasks have completed)
Evaluation status changed: "pending" -> "complete"
==> Evaluation "68b8b7dc" finished with status "complete"
Make note of the allocation ID shown in your output. In the provided sample
output, the allocation ID is cb1d80a1
.
Fetch the output of the job
Since this job writes its output to standard output, you can retrieve it with
the nomad alloc logs
command. Run the nomad alloc logs
command, supplying
the allocation ID you noted in the previous step.
Pro Tip - You only need to provide enough of the allocation ID to The
nomad alloc logs
command to point to a distinct allocation. This works with
many identifiers in the Nomad command-line tool.
$ nomad alloc logs cb1d80a1
Hello,
Thanks for your interest. I have attached the information
you requested. I look forward to talking with you soon.
Purge the job
Nomad will not allow you to convert a batch job to a parameterized job without
purging the original version as a safety feature. When run with the -purge
flag, the nomad job stop
command deletes the job and information about any
former runs of it from the Nomad server state.
Note
The start.nomad
file defines a Nomad job named template
. The
filename and job name are intentionally different allowing you to keep copies
of the different states of the job as you progress through the tutorial.
$ nomad job stop -purge template
==> Monitoring evaluation "a0704fca"
Evaluation triggered by job "template"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "a0704fca" finished with status "complete"
Make the job parameterized
Adding a parameterized stanza converts a batch-type job to a parameterized one.
An empty parameterized
stanza creates a parameterized job that isn't
customizable at dispatch time. This is still a useful feature, since it enables
an operator or application to run an instance of the job without having to have
the job specification itself.
Add parameterized stanza
Copy the start.nomad
file to a new file named parameterized.nomad
.
$ cp start.nomad parameterized.nomad
Open parameterized.nomad
in a text editor. Add an empty parameterized
stanza
inside of the job
stanza and save the file.
parameterized {
}
Register the job
Run the parameterized version of the job.
$ nomad job run parameterized.nomad
Job registration successful
Notice that the output doesn't show any scheduling activity—no evaluation or allocation information. Parameterized jobs themselves do not run.
If you get an error
If you receive the following error, it indicates that you missed purging the
non-parameterized version of the template job. Run nomad job stop -purge template
and re-run nomad job run parameterized.nomad
to resolve it.
$ nomad job run template.nomad
Error submitting job: Unexpected response code: 500 (cannot update non-parameterized job to being parameterized)
Run the nomad job status
command to verify your parameterized job is available
for dispatch.
$ nomad job status
ID Type Priority Status Submit Date
template batch/parameterized 50 running 2021-04-11T22:01:45-04:00
Dispatch the job
Run the nomad job dispatch
command to dispatch an instance of the
parameterized job. The nomad job dispatch
command's final argument is the
registered job ID to dispatch, unlike the nomad job run
command which
uses the filename of the job specification.
$ nomad job dispatch template
Dispatched Job ID = template/dispatch-1620682960-e9dfcaf8
Evaluation ID = 897c8095
==> Monitoring evaluation "897c8095"
Evaluation triggered by job "template/dispatch-1620682960-e9dfcaf8"
Allocation "4e317cb8" created: node "66d36f06", group "renderer"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "897c8095" finished with status "complete"
Examine the output of the command. There are some key differences from the
typical output of the nomad job run
command. Notice that Nomad generates a
Dispatched Job ID
. This generated job ID refers to this specific instance of
the parameterized job, and it enables you to manage discrete instances of a
parameterized job in the same ways you manage your other Nomad batch jobs.
The output also provides scheduling information. Collect the allocation ID from
your output. In the preceding output it's 4e317cb8
. Like earlier, run the
nomad alloc logs
command for your allocation ID.
$ nomad alloc logs 4e317cb8
Hello,
Thanks for your interest. I have attached the information
you requested. I look forward to talking with you soon.
Parameterized jobs without variables can be used to provide a means for running a batch job without having to supply the job specification.
Use dispatch variables
Parameterized jobs also provide the ability to submit run-specific variables while dispatching the job. The job specification defines the variable names. They can be optional or required.
Nomad provides the dispatched variable values as environment variables. The
environment variable names that Nomad creates for these variables are prepended
with NOMAD_META_
, for example NOMAD_META_customer_email
.
Nomad will not create environment variables for unset optional variables; your workload should handle this case. You may also define defaults for optional variables as part of the job specification.
Update the job to use email.tmpl
Copy the parameterized.nomad
file to a new file named variables.nomad
.
$ cp parameterized.nomad variables.nomad
Open the variables.nomad
file in a text editor and change the HCL2 file
function to read in the email.tmpl
file. This template generates an email-like
output.
Change
${file("./templates/template.tmpl")}
to
${file("./templates/email.tmpl")}
Define job variables
Required and optional variables are defined in the meta_required
and
meta_optional
attributes of the parameterized
stanza, respectively. These
attributes take a list of quoted variable names the value.
The email.tmpl
template needs more information to render properly:
- required -
customer_email
,customer_name
- optional -
rep_name
,rep_email
,rep_title
,product_name
Define the variables by adding these attributes to the variables.nomad
file
inside of the parameterized
stanza.
meta_required = ["customer_name","customer_email"]
meta_optional = ["rep_name","rep_email","rep_title","product_name"]
Provide defaults for optional values
You can define default values for your optional variables using a job-level
meta
stanza. Use the variable name as the attribute name and set the value to
the default.
For this example, create a default for the rep_name
and the rep_email
values
to provide generic values when the rep_name
and rep_email
are not provided.
In the variables.nomad
file, add this meta
stanza inside of the job
stanza.
meta {
rep_name = "BabbageCorp"
rep_email = "hello@mechanicalcomputing.com"
}
Deploy and dispatch the job
Save the variables.nomad
file and submit it using the nomad job run
command.
$ nomad job run variables.nomad
Job registration successful
Dispatch the job with variables
Now, dispatch a copy with the nomad job dispatch
command. Use the -meta
flag
to set the customer_email
to alovelace@programmers.net
and customer_name
to Ada Lovelace
.
$ nomad job dispatch \
-meta customer_email="alovelace@programmers.net" \
-meta customer_name="Ada Lovelace" \
template
The command will output scheduling information.
Dispatched Job ID = template/dispatch-1620750077-463bca71
Evaluation ID = 1b407851
==> Monitoring evaluation "1b407851"
Evaluation triggered by job "template/dispatch-1620750077-463bca71"
==> Monitoring evaluation "1b407851"
Allocation "391fc0c0" created: node "f7bc1f2d", group "renderer"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "1b407851" finished with status "complete"
Fetch the template output using the nomad alloc logs
command for the
allocation ID output when you dispatched the job.
$ nomad alloc logs 391fc0c0
TO: Ada Lovelace <alovelace@programmers.net>
FROM: BabbageCorp <hello@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest. I look forward to talking with you soon.
Sincerely,
BabbageCorp
Test the required variables
Nomad validates that the dispatch request contains all required variables. The operator or application dispatching the job must provide all required variables when dispatching the job. If not, Nomad will return an error.
Observe this by dispatching the job with no variables set.
$ nomad job dispatch template
Failed to dispatch job: Unexpected response code: 500 (rpc error: Dispatch did not provide required meta keys: [customer_email customer_name])
Use the optional variables
Now, use the optional variables to personalize the email. Dispatch the job again with the following command.
$ nomad job dispatch \
-meta customer_email="alovelace@programmers.net" \
-meta customer_name="Ada Lovelace" \
-meta rep_name="Charles Babbage" \
-meta rep_email="cbabbage@mechanicalcomputing.com" \
-meta rep_title="Chief Innovation Officer" \
-meta product_name="Analytical Engine" \
template
The command outputs the scheduling information.
Dispatched Job ID = template/dispatch-1620750369-9431cc3b
Evaluation ID = 6312c1e3
==> Monitoring evaluation "6312c1e3"
Evaluation triggered by job "template/dispatch-1620750369-9431cc3b"
Allocation "902a3d43" created: node "f7bc1f2d", group "renderer"
==> Monitoring evaluation "6312c1e3"
Allocation "902a3d43" status changed: "pending" -> "complete" (All tasks have completed)
Evaluation status changed: "pending" -> "complete"
==> Evaluation "6312c1e3" finished with status "complete"
Fetch the output from the generated allocation using the nomad alloc logs
command.
$ nomad alloc logs 902a3d43
TO: Ada Lovelace <alovelace@programmers.net>
FROM: Charles Babbage <cbabbage@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon.
Sincerely,
Charles
Chief Innovation Officer
Consume a dispatched payload
When defining a Nomad parameterized job, you have the option to use a
payload
. A payload is a means to supply up to 16Kib of opaque data to
instances of your parameterized job during dispatch.
Note
These payloads are included in the dispatched version's job definition data and will consume a corresponding amount of server memory until the dispatched instance of the job is garbage collected.
The parameterized
stanza's payload
value determines if a payload is
required
, optional
, or forbidden
when the job is dispatched.
Nomad writes the dispatch payload into any task containing a dispatch_payload
stanza. The path
attribute specifies where to write the data relative to the
task's local directory.
Enable a mandatory payload
Copy the variables.nomad
file to a new file named payload.nomad
.
$ cp variables.nomad payload.nomad
Open payload.nomad
in your text editor. Add a payload = "required
attribute inside the parameterized
stanza.
payload = "required"
Write the payload into the task
Also, add a dispatch_payload
stanza inside the output
task stanza.
dispatch_payload {
file = "payload.txt"
}
Submit the updated job
Save the payload.nomad
file and submit the job to Nomad.
$ nomad job run payload.nomad
Job registration successful
Dispatch the job with the payload
There are two ways to provide a payload to the nomad job dispatch
command
- from a file
- from standard input
Get payload from a file
The tutorial repository contains a sample text file for you to use as the payload—engine.txt
Use the nomad job dispatch
command to submit the payload directly
$ nomad job dispatch \
-meta customer_email="alovelace@programmers.net" \
-meta customer_name="Ada Lovelace" \
-meta rep_name="Charles Babbage" \
-meta rep_email="cbabbage@mechanicalcomputing.com" \
-meta rep_title="Chief Innovation Officer" \
-meta product_name="Analytical Engine" \
template payloads/engine.txt
The command outputs the scheduling information.
Dispatched Job ID = template/dispatch-1620751626-eefe8be3
Evaluation ID = 297238d6
==> Monitoring evaluation "297238d6"
Evaluation triggered by job "template/dispatch-1620751626-eefe8be3"
Allocation "a2971fa2" created: node "f7bc1f2d", group "renderer"
==> Monitoring evaluation "297238d6"
Allocation "a2971fa2" status changed: "pending" -> "complete" (All tasks have completed)
Evaluation status changed: "pending" -> "complete"
==> Evaluation "297238d6" finished with status "complete"
Note the allocation ID; retrieve the output of the allocation using the nomad alloc logs
command.
$ nomad alloc logs a2971fa2
TO: Ada Lovelace <alovelace@programmers.net>
FROM: Charles Babbage <cbabbage@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon.
Sincerely,
Charles
Chief Innovation Officer
----
CHAPTER VIII
OF THE ANALYTICAL ENGINE
The circular arrangement of the axes of the Difference Engine round large
central wheels led to the most extended prospects. The whole of arithmetic now
appeared within the grasp of mechanism. A vague glimpse even of an Analytical
...
Get payload from standard input
Using standard input allows you submit dynamic payloads to your parameterized
jobs. Use -
as the filename in the nomad job dispatch
command to collect the
payload from standard input. This example uses the date
command to generate
a dynamic payload.
$ date | nomad job dispatch \
-meta customer_email="alovelace@programmers.net" \
-meta customer_name="Ada Lovelace" \
-meta rep_name="Charles Babbage" \
-meta rep_email="cbabbage@mechanicalcomputing.com" \
-meta rep_title="Chief Innovation Officer" \
-meta product_name="Analytical Engine" \
template -
Dispatched Job ID = template/dispatch-1620751993-b9397497
Evaluation ID = dc32bd93
==> Monitoring evaluation "dc32bd93"
Evaluation triggered by job "template/dispatch-1620751993-b9397497"
==> Monitoring evaluation "dc32bd93"
Allocation "b22837a8" created: node "f7bc1f2d", group "renderer"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "dc32bd93" finished with status "complete"
Fetch the output of the dispatched allocation using the nomad alloc logs
command.
$nomad alloc logs b22837a8
TO: Ada Lovelace <alovelace@programmers.net>
FROM: Charles Babbage <cbabbage@mechanicalcomputing.com>
SUBJ: Thanks for your interest.
------
Hello Ada,
Thanks for your interest in the Analytical Engine product. I look forward to talking with you soon.
Sincerely,
Charles
Chief Innovation Officer
Wed May 12 11:59:30 EDT 2021
Clean up
Clean up the target environment
The cleanup instructions vary depending on your target infrastructure. Select the one that applies.
Nomad dev agent
Stop the Nomad dev agent with Ctrl-C
. Nomad dev agents use ephemeral storage
and require
no further cleanup.
Nomad cluster
Since Nomad clusters maintain persistent state, you will need to perform extra steps to remove all the tutorial artifacts from the cluster.
Remove the parameterized job
Stop the template
job. You can optionally pass the -purge
flag to tell Nomad
to immediately remove it from the server state. Since no scheduling activity is
required to remove the job, there is no output generated by the command.
$ nomad job stop -purge template
Remove dispatched instances (optional)
Each time you dispatch a parameterized job, Nomad creates a distinct but related
job. These jobs will naturally be removed in time by Nomad's garbage collection
process. You can see a list of these instances by using the nomad job status
command referencing an ambiguous prefix, like template\
. These IDs will be
specific to your environment.
Pro Tip - The Nomad command-line tool displays a list of matching IDs whenever an incomplete ID matches more than one item.
$ nomad job status template/
Prefix matched multiple jobs
ID Type Priority Status Submit Date
template/dispatch-1620850743-2d3e0743 batch 50 dead 2021-05-12T16:19:03-04:00
template/dispatch-1620850966-1ba39df6 batch 50 dead 2021-05-12T16:22:46-04:00
template/dispatch-1620851039-1584df83 batch 50 dead 2021-05-12T16:23:59-04:00
template/dispatch-1620851914-68ee4baf batch 50 dead 2021-05-12T16:38:34-04:00
template/dispatch-1620851954-e0c9f384 batch 50 dead 2021-05-12T16:39:14-04:00
These jobs will naturally be removed in time by Nomad's garbage collection process.
You can also use the nomad job stop
command with the -purge
flag set to
remove individual dispatched instances immediately. For example:
$ nomad job stop -purge template/dispatch-1620748308-41f86ed3
==> Monitoring evaluation "d771b1fc"
Evaluation triggered by job "template/dispatch-1620748308-41f86ed3"
==> Monitoring evaluation "d771b1fc"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "d771b1fc" finished with status "complete"
Remove the tutorial's working directory
Ensure that you have closed any file editors that might have files in the working directory open.
Change to your home directory and recursively remove the learning environment you created in the first step.
Remove the unarchived learning environment.
$ rm -rf learn-nomad-jobspec-release
Delete the downloaded archive.
$ rm release.zip
Review what you learned
You create a parameterized job by adding the parameterized
stanza to a
batch-type Nomad job specification.
You dispatch an instance of a parameterized job by running thennomad job dispatch
command or using the Job Dispatch API.
Nomad dispatch variables are supplied to the workload as environment variables.
The environment variable's names are generated by prepending NOMAD_META_
to
the variable's name, like NOMAD_META_customer_email
.
Dispatch payloads can be up to 16KiB. Payloads are written directly into the dispatched job's definition, so they use that same amount of server RAM until the job is complete and garbage collected.
Next steps
Nomad parameterized jobs are a powerful tool, encouraging reuse of a job specification by allowing users to provide runtime specific values when dispatching the job. Parameterized jobs enable job creators to make consistent and reusable experiences by encapsulating the specifics of the job and only requiring users to provide the run-time specific information. This ultimately simplifies the user experience for both end-users and API consumers.
In this tutorial, you performed these job specification development actions:
- Converted a batch job to a parameterized job.
- Added required and optional variables to the job specification.
- Defined default values for optional variables
- Configured the job to use a dispatch payload.
You performed these end-user operations:
- Dispatched instances of a parameterized job
- Provided variables during dispatch
- Submitted a static and dynamic payload during dispatch
You performed these cluster-operator tasks:
- Retrieved logs for an allocation using the
nomad alloc logs
command
Learn more about the template language used in the .tmpl
files in the Learn
Go Template Syntax and in the consul-template
documentation. You can also learn more about parameterized jobs in the
Nomad documentation.