Delete

Delete hooks allow you to remove files or directories from local or remote storage locations. This is useful for cleanup operations, removing temporary files, or maintaining storage quotas.

Configuration

- type: delete
  location: "aws_s3/path/to/file"   # Required: Location string
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id               # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.

Properties

Property
Required
Description

location

Yes

path

Yes

The path to the file or directory to delete

on_failure

No

What to do if the deletion fails (abort/warn/quiet/skip)

Output

When the delete hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
path: "path/to/file"  # The path that was deleted

You can access these values in subsequent hooks using the following syntax (jmespath):

  • {state.hook_id.status} - Status of the hook execution

  • {state.hook_id.path} - The path that was deleted

Examples

Clean Up Temporary Files

Remove temporary files after successful processing:

hooks:
  post:
    - type: delete
      location: "local//tmp/processed/{run.stream.name}_{timestamp.date}/*"
      on_failure: warn

Clean Up Staging Area

Remove processed files from a staging area after successful replication:

hooks:
  post:
    - type: delete
      if: run.status == "success"
      location: "azure_blob/staging/{target.environment}/{run.stream.name}/"
      on_failure: warn

Remove Failed Processing Artifacts

Clean up artifacts from failed processing attempts:

hooks:
  post:
    - type: delete
      if: run.status == "error"
      location: "gcs/failed-jobs/{timestamp.date}/{run.stream.name}/"
      on_failure: quiet

Clean Up Log Files

Remove old log files after successful processing:

hooks:
  post:
    - type: delete
      location: "local//var/log/sling/{run.stream.name}/*.log"
      on_failure: quiet

Last updated