Not in the traditional sense. However, Sling is capable of the below transformations.
Custom SQL
Custom SELECT query as the source stream, using a custom SQL input as a source.stream. Here is an example::
slingrun \--src-connSTARROCKS \--src-stream"SELECT columnB, columnA FROM tbl WHERE columnB > 6000" \--tgt-connMYSQL \--tgt-objectmysql.tbl \--modefull-refresh
Column Type Casting
Column type coercion, the columns key in source.options (see here). This is especially useful for file sources. Here is an example casting types of columns:
JMESPath transformations for JSON source files: the jmespath key in source.options. Here is a example, extracting the models information from the DBT manifest file using jmespath + flatten options:
When using Sling to extract/load data in a incremental manner, it will attempt to match whatever columns are present in both the source stream and target table. If an extra column is present in the source stream, it will add it in the target table. If no columns match from source stream at all, it will error. At least the primary_key or update_key must be present in the target table.
See below for a simple example, mimicking the addition and removal of columns.
# test3.csv is missing column b, c and has extra column d$echo'a,d9,10'>test3.csv$slingrun \--src-streamfile://./test3.csv \--tgt-connpostgres \--tgt-objectpublic.test1 \--modeincremental \--primary-keya<...log output omitted>$slingrun \--src-connpostgres \--src-streampublic.test1 \--stdouta,b,c,_sling_loaded_at,d1,2,3,1707869559,4,5,6,1707869559,7,,8,1707869689,9,,,1707870320,10
We can see that sling handled the changes properly, in a non-destructive manner. If the source stream were from a database, the same rules would apply, whether a column disappeared or appeared.
Suggestions?
There are currently no plans to supports other custom transformations, in the traditional "T" sense. However, any suggestions are welcome.