Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Scala closure serialization #299

Open
asarkar opened this issue Dec 14, 2017 · 5 comments
Open

Support Scala closure serialization #299

asarkar opened this issue Dec 14, 2017 · 5 comments

Comments

@asarkar
Copy link

asarkar commented Dec 14, 2017

See https://stackoverflow.com/q/47806670/839733 for details

case class TestRequest[T](app: String, locker: MeatLocker[T])

val factory: Stats => Unit = stats => {
  stats.app shouldBe (app)
  stats.unit shouldBe (SECONDS)
  stats.startupDurations should have length (10)
}

val locker: MeatLocker[Stats => Unit] = MeatLocker(factory)

val file = SerializationUtil.write(TestRequest[Stats => Unit](app, locker))
val state = SerializationUtil.read[TestRequest[Stats => Unit]](file)

(state.locker.get) (newStats())

Could not serialize lambda
java.lang.RuntimeException: Could not serialize lambda
at com.esotericsoftware.kryo.serializers.ClosureSerializer.write(ClosureSerializer.java:67)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.twitter.chill.SerDeState.writeClassAndObject(SerDeState.java:64)
at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:116)
at com.twitter.chill.MeatLocker.(MeatLocker.scala:32)
at com.twitter.chill.MeatLocker$.apply(MeatLocker.scala:21)

@johnynek
Copy link
Collaborator

What version of kryo and scala are you using. This should work. we do it all the time in scalding.

@johnynek
Copy link
Collaborator

Actually it isn’t magic. Not everything can be serialized (things like Threads or file handles for instance). I have found that test frameworks often have such things and your example seems to capture one. I am not surprised this does not work.

@asarkar
Copy link
Author

asarkar commented Dec 14, 2017

I'll post here the same update I did on SO.

Perhaps I should clarify the actual use case that is running into this issue to provide more context. The function is a callback from Akka HTTP route like the following:

path("stats") {
  logRequest("/stats") {
    completeWith(instanceOf[List[Stats]]) { callback =>
      requestHandler ! GetStatsRequest(callback)
    }
  }
}

The handler actor persists the request until it gets a response. It may take more than one response to construct the final output.

I did some digging and it appears that the callback implementation is a CallbackRunnable.

@johnynek
Copy link
Collaborator

Can you post also the version of chill you were using and the scala version (2.11, 2.12?)

@asarkar
Copy link
Author

asarkar commented Dec 14, 2017

chill-akka_2.12:0.9.2, Scala 2.12.4. I've since reverted that code, so you won't find it in the project I linked above.

@asarkar asarkar changed the title Support Scala function serialization Support Scala closure serialization Dec 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants