What's Changed
🎉 Exciting New Features
☁️ Google cloud storage source
You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.
by @TalaatHasanin in #5334
🌬️ 🍽️ Airtable Integration
Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.
by @TalaatHasanin in #5404
✂️ Add trim reformat action to transformer block
A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.
by @cristopheridlc in #5321
🐻❄️ Enable polars dataframe in GCS data exporter
This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.
by @TalaatHasanin in #5348
Add support to scheduler name on k8s executor
This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.
by @messerzen in #5412
🐛 Bug Fixes
- [jk] Update condition for cookie secure property by @johnson-mage in #5327
- [xy] Fix NoneType in trigger settings. by @wangxiaoyou1993 in #5329
- [xy] Fix dbt block delete request with nginx. by @wangxiaoyou1993 in #5332
- [xy] Fix API for file versions. by @wangxiaoyou1993 in #5338
- [xy] Fix schema creation when schema name is wrapped with double quotes. by @wangxiaoyou1993 in #5366
- [jk] Update test ENV value by @johnson-mage in #5372
- [xh] Add retry logic and throw if open AI API returns error by @matrixstone in #5375
- [jk] Fix typo for custom frequency in trigger review by @johnson-mage in #5392
- [td] Fix Git syncs API error returning sync_on_executor_start by @tommydangerous in #5388
- #5417 Fix bug: Can't use git with Bitbucket by @kennynguyeenx in #5418
- Dynamic block fixes
- [td] Fix index used in dynamic block by @tommydangerous in #5373
- [xy] Fix the condition of checking dynamic upstreams completed 2f2eeca
- [xy] Some updates and fixes for dynamic blocks 7eba140
- [hw] Fix a missing import issue with data types in a template for Data Exporters by @csharplus in #5377
- SASL_SSL kafka Host Name Validation by @sujiplr in #5380
💅 Enhancements & Polish
- [xy] Reduce CPU usage in scheduler. by @wangxiaoyou1993 in #5331
- [xy] Pass main pod labels to workspaces. by @wangxiaoyou1993 in #5344
- enhancement: bump duckdb to 1.0.0 by @matsonj in #5358
- Pass all SSL settings through for SASL_SSL by @csharplus in #5381
- Update unit tests to use the methods compatible with Python 3.12 by @csharplus in #5398
- Introduce StrEnum and IntEnum classes to resolve unit test failures with Python 3.11+ by @csharplus in #5397
- Add NATS credentials file authentication by @ddecaro11 in #5410
- Update a few unit tests that failed on Python 3.12 by @csharplus in #5420
- Fix kafka module related unit test failure issues in Python 3.12 by @csharplus in #5424
- Add trigger name in kwargs 81e4f91
- Expose column name related config to postgres sink config c40af5f
- [xh] Add ENV variables to turn on/off AI API by @matrixstone in #5394
Other Changes
- Update amazon-s3.mdx by @Ayumi08 in #5337
- Fix spelling in compute-resource.mdx by @samacciu23 in #5356
- Update logger.py by @vffv2000 in #5383
- [hw] Update the user guide for Kafka streaming with latest changes based on testing by @csharplus in #5378
- Update the development document to resolve one issue encountered by @csharplus in #5396
- Fixing spelling error by @benjaminhawn in #5407
New Contributors
- @cristopheridlc made their first contribution in #5321
- @TalaatHasanin made their first contribution in #5348
- @samacciu23 made their first contribution in #5356
- @matsonj made their first contribution in #5358
- @vffv2000 made their first contribution in #5383
- @benjaminhawn made their first contribution in #5407
- @ddecaro11 made their first contribution in #5410
- @Jaykold made their first contribution in #5340
- @kennynguyeenx made their first contribution in #5418
- @messerzen made their first contribution in #5412
Full Changelog: 0.9.73...0.9.74
