// MANAGED JOBS

Managed jobs

Coming soon: managed jobs aren't switched on for MatMul accounts yet. This page describes how they work so you can plan for them. Until then, rent an instance and run your command over SSH.

What a job is

An instance is a machine you keep until you delete it. A job is a command: MatMul finds the cheapest machine that fits, runs your setup and your command on it, keeps the logs, and releases the machine as soon as the command ends. You pay for the machine time in between.

matmul run --gpu H100 --max-price 3.50 --wait -- python train.py

Is it on?

Ask the API. The CLI does this before every matmul run:

curl -s https://api.matmul.cloud/v1/capabilities -H "Authorization: Bearer $MATMUL_API_KEY"
{"instances": true, "app_links": false, "jobs": false}

While jobs is false, matmul run stops with a message and the /v1/jobs endpoints return 503.

Launching

matmul run --name train --gpu H100 --gpus-per-node 2 --max-price 7 \
  --setup "git clone https://github.com/you/repo && pip install -r repo/requirements.txt" \
  --env WANDB_API_KEY=... \
  -- python repo/train.py --epochs 3

Everything after -- is the command. The same spec over the API is a POST /v1/jobs/ body:

Field	CLI flag	Rules
`name`	`--name`	Required. `^[a-z][a-z0-9-]{0,15}$`. The CLI makes one up if you don't.
`run`	after `--`	Required. Up to 20,000 characters.
`setup`	`--setup`	Runs once before the command. Up to 20,000 characters.
`envs`	`--env`	Up to 50. Values go to the machine and aren't stored; only the names are kept.
`num_nodes`	`--nodes`	1 to 8 machines (default 1).
`resources.accelerators`	`--gpu`, `--gpus-per-node`	Per machine, like `H100` or `H100:2`.
`resources.cpus`	`--cpus`	`4` or `4+`.
`resources.memory`	`--memory`	GB, `16` or `16+`.
`resources.disk_size`	`--disk`	10 to 2048 GB.
`max_hourly_price_cents`	`--max-price`	Ceiling for the whole job, all machines together (the CLI takes dollars).

Nothing else is accepted: no file uploads, mounted storage, working directories or choice of cloud. Get your code onto the machine in setup, for example with git clone or pip install.

Price, balance and limits

MatMul quotes the cheapest machine that fits. The job's price is that machine's hourly price times the number of machines.
With --max-price, a quote above your ceiling is refused (409, "the cheapest match costs $X/hr, above your max of $Y/hr") and nothing launches.
You need an hour of runway at the job's price, and a frozen account can't launch. Freezes and the grace period cover jobs the same way as instances; see Pricing & billing.
Up to 2 jobs run at a time (1 on a new account). A name can't be reused while a job with it is still active.

Following a job

matmul ls                  # all jobs: state, GPU, $/hr, runtime, cost
matmul status train        # one job
matmul logs train          # its logs, also after it ends
matmul down train          # stop it and release the machine

Refer to a job by name (the most recent job with it) or by its job_... id. With --wait, matmul run prints a line on each state change, then the logs, and exits 0 if the job succeeded and 1 if not, so it works in CI.

State	Means
`launching`	Queued or provisioning; `detail` says which.
`running`	Your setup or command is running.
`succeeded`	The command exited 0.
`failed`	It exited non-zero or couldn't start; `error` says why.
`cancelled`	Stopped with `matmul down`, or by a freeze past the grace period.
`finished`	The machine went away after the command ran, before its exit status was read.

The same operations are in the Python SDK (launch, wait, logs, down) and the REST API.

← All docs