Copy
Copy hooks allow you to transfer files between storage locations. This is particularly useful for moving files between different storage systems, making backups, or archiving data.
Configuration
Properties
from
Yes
to
Yes
on_failure
No
What to do if the copy fails (abort/warn/quiet/skip)
Output
When the copy hook executes successfully, it returns the following output that can be accessed in subsequent hooks:
You can access these values in subsequent hooks using the following syntax (jmespath
):
{state.hook_id.status}
- Status of the hook execution{state.hook_id.from_uri}
- The normalized URI of the source file{state.hook_id.from_path}
- The source path{state.hook_id.to_uri}
- The normalized URI of the destination file{state.hook_id.to_path}
- The destination path{state.hook_id.bytes_written}
- Number of bytes written
Examples
Archive Files Between Cloud Storage
Archive files between different cloud storage providers:
Upload A Local DuckDB Database into S3
Copy a local database file into Amazon S3 after writing to it:
Copy Multiple Files Pattern
Copy multiple files matching a pattern:
Last updated