Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

row_number is incorrectly pulled into lazy_row #356

Open
okennedy opened this issue Aug 19, 2019 · 1 comment
Open

row_number is incorrectly pulled into lazy_row #356

okennedy opened this issue Aug 19, 2019 · 1 comment
Labels

Comments

@okennedy
Copy link
Member

Should be evaluated by spark.

mimir> select row_number(), cast(h as float)*60*60+cast(m as float)*60+cast(s as float)-75359.239 from extract_warm_start;
java.lang.RuntimeException: Error Decoding ROW_NUMBER (int)
        at mimir.exec.result.LazyRow.apply(LazyRow.scala:22)
        at mimir.exec.PrettyOutputFormat$$anonfun$print$4$$anonfun$apply$1.apply$mcVI$sp(OutputFormat.scala:81)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at mimir.exec.PrettyOutputFormat$$anonfun$print$4.apply(OutputFormat.scala:80)
        at mimir.exec.PrettyOutputFormat$$anonfun$print$4.apply(OutputFormat.scala:79)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at mimir.exec.result.ResultSeq.foreach(ResultSeq.scala:5)
        at mimir.exec.PrettyOutputFormat.print(OutputFormat.scala:79)
        at mimir.Mimir$$anonfun$handleQuery$1$$anonfun$apply$mcV$sp$1.apply(Mimir.scala:204)
        at mimir.Mimir$$anonfun$handleQuery$1$$anonfun$apply$mcV$sp$1.apply(Mimir.scala:204)
        at mimir.Database.query(Database.scala:155)
        at mimir.Database.query(Database.scala:182)
        at mimir.Mimir$$anonfun$handleQuery$1.apply$mcV$sp(Mimir.scala:204)
        at mimir.Mimir$$anonfun$handleQuery$1.apply(Mimir.scala:204)
        at mimir.Mimir$$anonfun$handleQuery$1.apply(Mimir.scala:204)
        at mimir.util.Timer$.monitor(Timer.scala:22)
        at mimir.Mimir$.handleQuery(Mimir.scala:203)
        at mimir.Mimir$.handleSelect(Mimir.scala:198)
        at mimir.Mimir$.eventLoop(Mimir.scala:150)
        at mimir.Mimir$.interactiveEventLoop(Mimir.scala:121)
        at mimir.Mimir$.main(Mimir.scala:112)
        at mimir.Mimir.main(Mimir.scala)
Caused by: java.lang.UnsupportedOperationException: Cannot evaluate expression: row_number()
        at org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:258)
        at org.apache.spark.sql.catalyst.expressions.aggregate.DeclarativeAggregate.eval(interfaces.scala:350)
        at mimir.exec.spark.MimirSpark$$anonfun$registerSparkFunctions$2$$anonfun$apply$1.apply(MimirSpark.scala:245)
        at mimir.exec.spark.MimirSpark$$anonfun$registerSparkFunctions$2$$anonfun$apply$1.apply(MimirSpark.scala:238)
        at mimir.exec.EvalInlined$$anonfun$compileFunction$1.apply(EvalInlined.scala:70)
        at mimir.exec.EvalInlined$$anonfun$compileFunction$1.apply(EvalInlined.scala:70)
        at mimir.exec.EvalInlined$$anonfun$compileForLong$13.apply(EvalInlined.scala:131)
        at mimir.exec.EvalInlined$$anonfun$compileForLong$13.apply(EvalInlined.scala:131)
        at mimir.exec.EvalInlined$$anonfun$compile$2.apply(EvalInlined.scala:45)
        at mimir.exec.EvalInlined$$anonfun$compile$2.apply(EvalInlined.scala:45)
        at mimir.exec.EvalInlined$$anonfun$checkNull$1.apply(EvalInlined.scala:21)
        at mimir.exec.EvalInlined$$anonfun$checkNull$1.apply(EvalInlined.scala:20)
        at mimir.exec.result.LazyRow.apply(LazyRow.scala:17)
        ... 23 more
QUERY took: 157 ms
@okennedy okennedy added the bug label Aug 19, 2019
@okennedy okennedy self-assigned this Aug 19, 2019
@okennedy
Copy link
Member Author

okennedy commented Nov 1, 2019

Due to optimizations in Mimir, row_number doesn't actually provide consistent row numbering.

  • Provide a warning about row_number instead of blindly compiling it in
  • Revisit how to provide equivalent functionality in Mimir (e.g., TABLE_NAME.row_number)

@okennedy okennedy removed their assignment Jun 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant