Delete

Delete hooks allow you to remove files or directories from local or remote storage locations. This is useful for cleanup operations, removing temporary files, or maintaining storage quotas.

Configuration

- type: delete
  location: "aws_s3/path/to/file"   # Required: Location string
  recursive: false        # Optional: true/false. Default is false
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id               # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.

Properties

Property	Required	Description
location	Yes	The location string. Contains connection name and path.
recursive	No	Whether to delete recursively
path	Yes	The path to the file or directory to delete
on_failure	No	What to do if the deletion fails (abort/warn/quiet/skip)

Output

When the delete hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
path: "path/to/file"  # The path that was deleted

You can access these values in subsequent hooks using the following syntax (jmespath):

• {state.hook_id.status} - Status of the hook execution

• {state.hook_id.path} - The path that was deleted

Examples

Clean Up Temporary Files

Remove temporary files after successful processing:

hooks:
  post:
    - type: delete
      location: "local//tmp/processed/{run.stream.name}_{timestamp.date}/*"
      on_failure: warn

Clean Up Staging Area

Remove processed files from a staging area after successful replication:

hooks:
  post:
    - type: delete
      if: run.status == "success"
      location: "azure_blob/staging/{target.environment}/{run.stream.name}/"
      recursive: true
      on_failure: warn

Remove Failed Processing Artifacts

Clean up artifacts from failed processing attempts:

hooks:
  post:
    - type: delete
      if: run.status == "error"
      location: "gcs/failed-jobs/{timestamp.date}/{run.stream.name}/"
      on_failure: quiet

Clean Up Log Files

Remove old log files after successful processing:

hooks:
  post:
    - type: delete
      location: "local//var/log/sling/{run.stream.name}/*.log"
      on_failure: quiet