Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Ignorance Is Bliss Section for Streaming 'ignoreX'... #11

Open
newfront opened this issue Jun 24, 2023 · 0 comments
Open

Add Ignorance Is Bliss Section for Streaming 'ignoreX'... #11

newfront opened this issue Jun 24, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@newfront
Copy link
Owner

moving fast is awesome. crashing at lightspeed is not.

  • preventing problems that have to do with "not thinking it though"
  • there is a tie in to "ooops I did it again here". Since mergeSchema: true only prevents type changes - while `overwriteSchema
(spark.read.table(...)
  .withColumnRenamed("x", "y")
  .write
  .mode("overwrite")
  .option("overwriteSchema", "true")
  .saveAsTable(...)
)

not a terrible problem - just a new name - same content

(spark.read.table(...)
.withColumn('x', date(col('x'))
.write
.mode('overwrite')
.mode('overwriteSchema', 'true')
.saveAsTable(...)
)

this is bad. this is very bad. We not only just lost precision, but we'll have type conflicts for our streaming applications (active downstream)....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

1 participant