Batch Framework

Overview

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.

A batch file defines one job task.

Once started, MinIO starts processing the job. Time to completion depends on the resources available to the deployment.

Ordinary job failures use the retry limits in the definition. The next SILO release distinguishes authorization refusal from temporary pauses as described below.

The MinIO Batch Framework supports the following job types:

Next SILO release: authorization and retries

SILO binds local batch operations to the authenticated submitter at job creation. The credentials in a definition are used for its remote connections; they do not grant local server authority. IAM changes are checked again when actions are taken. An explicit denial, expired or revoked credential, or changed submitter binding ends the job with failureReason. Statements that need request context unavailable to a background job refuse the action.

IAM startup, unavailable policy material, and an authorization plugin that cannot return a valid decision are temporary failures. Submission returns 503 when authorization cannot be decided. Expiration and key rotation pause with pausedReason, preserve completed progress, and retry in the same process with bounded exponential backoff and jitter. Immediate or streamed listing failures also pause these jobs. Rotation pauses when its selected source is in a moving pool and resumes after the conflict clears. Every retry resolves current identity and policies again. A pending or paused job is not a completed or failed report and is not removed by terminal-report cleanup. Administrator cancellation takes precedence over queued work and retries.

Push, pull, and Snowball replication also pause with pausedReason when a required authorization dependency or report read is temporarily unavailable, including an outage that begins during object processing, and restart their source scan from the beginning on retry; see Batch Replication. Other mid-run replication failures follow the existing finite object retry behavior and can end the run as failed.

Startup discovery also retries failed manifest listings and storage reads with bounded backoff. It remembers manifests already admitted during that discovery, so a later scan cannot rerun a job that finished while waiting. Cancellation fences stale discovery results. Missing, malformed, or unsupported manifests are not queued; malformed and unsupported definitions remain on disk for inspection. A storage outage during discovery does not require a restart once storage recovers.

Completed jobs remain completed after restart, including legacy jobs and jobs whose credentials later expire or change. Unfinished jobs from releases without submitter binding require resubmission. New bound definitions and authorized progress reports use formats rejected by older executors. This fences those records on rollback, but does not make a mixed cluster safe: an old node can still accept and execute its own old-format jobs. Stop submissions and drain or cancel old jobs before changing versions, and do not downgrade with pending authorized jobs.

Use mcli batch status (20261109) to distinguish pauses, terminal failures, and completion with protected work left in place. infraAdmin still has other infrastructure administration capabilities; these job checks do not turn that role into a strict data-isolation boundary.

MinIO Batch CLI

• Install the MinIO Client
• Define an alias for the MinIO deployment

The mc batch commands include

Access to mc batch

The submitting identity needs the relevant administration action and the local data permissions for the job. Credentials in the definition must separately permit its remote operations. See the next-release rules above.

The user executing the batch job must have the following permissions. You can alternatively restrict users from accessing these functions by blocking or limiting access to these actions:

admin:ListBatchJobs

Grants the user the ability to see batch jobs currently in process.

admin:DescribeBatchJobs

Grants the user the ability to see the definition details of batch job currently in process.

admin:StartBatchJob

Grants the user the ability to start a batch job. The job may be further restricted by the credentials the job uses to access either the source or target deployments.

admin:CancelBatchJob

Allows the user to stop a batch job currently in progress.

You can assign any of these actions to users independently or in any combination.

The built-in ConsoleAdmin policy includes sufficient access to perform all of these types of batch job actions.

Local Deployment

You run a batch job against a particular deployment by passing an alias to the mc batch command. The deployment you specify in the command becomes the local deployment within the context of that batch job.