Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#2060] drop exceptions thrown by @Every jobs so that they get rescheduled #1160

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion framework/src/play/jobs/Job.java
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,19 @@ public void every(String delay) {
* time in seconds
*/
public void every(int seconds) {
JobsPlugin.executor.scheduleWithFixedDelay(this, seconds, seconds, TimeUnit.SECONDS);
// Wrap this job in a Runnable that catches and ignores all thrown exception.
// If there's an unhandled exception, then the executor would suppress all
// further executions, violating the contract. Note that this does
// not interfere with the normal handling of onException().
Runnable neverThrowingRunnable = new Runnable() {
Copy link
Contributor

@flybyray flybyray Jun 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also dislike anonymous class wrappers. The reason I didn't add a new flag to the Job class is that it's not a property of the job, but how the job was scheduled. I didn't want to introduce hard-to-understand failures when a Job is scheduled multiple times by the application. For example, if someone wanted to run a Job now and every 24 hours afterward, they could write something like:

  Job j = MyJob();
  j.every("24h");
  try {
    await(j.now());
  } catch (Exception ex) {
    error("could not run the job");
  }

If Job.every() sets a flag in the Job object, then putting j.every() before the j.now() would interfere with the exception propagation, but the reverse wouldn't. To me, that seemed counter-intuitive. The current Job implementation uses the anonymous wrapper class created in Job.getJobCallingCallable() to manage the exceptions for jobs scheduled by Job.now() and Job.in() without reading/writing state in the Job, so I followed suit.

That said, the applications written by my organization never the above pattern, so we wouldn't be negatively impacted by keeping a "scheduled by every" flag within the Job object. So if that's the way you want me to go, that's the way I'll go. Please confirm.

I suppose you recommended making this behavior controllable by a new configuration option so that someone could disable it if they wanted an unhandled exception to halt the "every" execution. Are there additional documentation requirements for introducing a new configuration option, or would it be acceptable to keep it as a hidden option, available to anyone who reads the source code?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That commit, should have introduced the flag within now or a special now(boolean throwExecutionException):
45d390d#diff-34df15bef2557dbc31eb399824919cfeR138

my suggestion is:

  • fix just the commit 45d390d on line 138 which introduced the error (by using a flag, check now and in too)
  • for your own problems the use of onException should be sufficient

Or am I missing something? Overriding onException is the way to go if you want to ignore errors or special error handling.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix just the commit 45d390d on line 138 which introduced the error (by using a flag, check now and in too)

Thanks for the extra clarification. I'll rework my pull request to do this, following the pattern of your Gist, but applied to now() and in() instead of every(). It may take me a while to put the update together.

public void run() {
try {
Job.this.run();
} catch (Exception exception) {
}
}
};
JobsPlugin.executor.scheduleWithFixedDelay(neverThrowingRunnable, seconds, seconds, TimeUnit.SECONDS);
JobsPlugin.scheduledJobs.add(this);
}

Expand Down
5 changes: 3 additions & 2 deletions framework/src/play/jobs/JobsPlugin.java
Original file line number Diff line number Diff line change
Expand Up @@ -157,14 +157,15 @@ public void afterApplicationStart() {
// @Every
if (clazz.isAnnotationPresent(Every.class)) {
try {
Job job = createJob(clazz);
Job<?> job = (Job<?>) clazz.newInstance();

String value = job.getClass().getAnnotation(Every.class).value();
if (value.startsWith("cron.")) {
value = Play.configuration.getProperty(value);
}
value = Expression.evaluate(value, value).toString();
if (!"never".equalsIgnoreCase(value)) {
executor.scheduleWithFixedDelay(job, Time.parseDuration(value), Time.parseDuration(value), TimeUnit.SECONDS);
job.every(value);
}
} catch (InstantiationException | IllegalAccessException ex) {
throw new UnexpectedException("Cannot instantiate Job " + clazz.getName(), ex);
Expand Down
139 changes: 139 additions & 0 deletions samples-and-tests/just-test-cases/test/JobTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
import java.util.concurrent.atomic.AtomicLong;

import org.junit.Test;

import play.jobs.Every;
import play.jobs.Job;
import play.test.UnitTest;

/**
* Unit tests for the {@link Job} class.
*/
public class JobTest extends UnitTest {

/**
* A Job class that is annotated with {@link Every} to run every second.
*/
public static class TestJob extends Job {
private final AtomicLong totalRuns = new AtomicLong(0);
private volatile boolean throwException = false;

@Override
public void doJob() throws Exception {
totalRuns.getAndIncrement();

// To avoid logging an error every second, we only throw an
// exception when the relevant test case is running.
if (throwException) {
throwException = false;
throw new Exception("Throwing an exception");
}
}
}

/**
* A Job class that is annotated with {@link Every} to run every second.
*/
@Every("1s")
public static class RunEverySecond extends Job {
private static final AtomicLong totalRuns = new AtomicLong(0);
private static volatile boolean throwException = false;

@Override
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

override is easy! why not just override onException too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the recommendation. This unit test is designed to test what happens when someone doesn't override onException (the common case, and the case cited in [#2060]). If it overrides onException, then it would be testing something else. Could you sketch out a bit more what you had in mind?

public void doJob() throws Exception {
totalRuns.getAndIncrement();

// To avoid logging an error every second, we only throw an
// exception when the relevant test case is running.
if (throwException) {
throwException = false;
throw new Exception("Throwing an exception");
}
}
}

/**
* A Job class that is annotated with {@link Every} to never run.
*/
@Every("never")
public static class RunNever extends Job {
private static final AtomicLong totalRuns = new AtomicLong(0);

@Override
public void doJob() throws Exception {
totalRuns.getAndIncrement();
}
}

/**
* A Job class that is annotated with {@link Every} to never run (using mixed case)
*/
@Every("NeveR")
public static class RunNeverMixedCase extends Job {
private static final AtomicLong totalRuns = new AtomicLong(0);

@Override
public void doJob() throws Exception {
totalRuns.getAndIncrement();
}
}

/**
* Tests that a job which is annotated to run every second does, indeed, run every second.
*/
@Test
public void testEverySecond() {
long beforeRuns = RunEverySecond.totalRuns.get();
pause(1500); // wait long enough for the job to have run at least once
long afterRuns = RunEverySecond.totalRuns.get();
assertTrue("RunEverySecond job was never run", beforeRuns < afterRuns);
assertTrue("RunEverySecond job was run too many times", afterRuns - beforeRuns <= 2);
}

/**
* Tests that jobs which are annotated to never run have never been run.
*/
@Test
public void testEveryNever() {
assertEquals(0, RunNever.totalRuns.get());
assertEquals(0, RunNeverMixedCase.totalRuns.get());
}

/**
* Tests that throwing an exception does not halt the periodic scheduling of an {@code @Every} annotation.
* This is a regression test for Lighthouse [#2060]
*/
@Test
public void testExceptionDoesNotHaltReschedulingWithEveryAnnotation() {

// Configure RunEverySecond to throw an exception.
RunEverySecond.throwException = true;
long beforeRuns = RunEverySecond.totalRuns.get();

pause(2500); // wait long enough for the job to have run at least twice

// Make sure it threw an exception and ran multiple times.
long afterRuns = RunEverySecond.totalRuns.get();
assertFalse("RunEverySecond job never ran", RunEverySecond.throwException);
assertTrue("RunEverySecond job was not run after throwing an exception", 2 <= afterRuns - beforeRuns);
}

/**
* Tests that throwing an exception does not halt the periodic scheduling of the {@link Job#every(int)} method.
* This is a regression test for Lighthouse [#2060]
*/
@Test
public void testExceptionDoesNotHaltReschedulingEveryMethod() {

// Schedule a job to run every second.
TestJob job = new TestJob();
job.throwException = true;
job.every("1s");

pause(2500); // wait long enough for the job to have run at least twice

// Make sure it threw an exception and ran multiple times.
assertFalse("TestJob job never ran", job.throwException);
assertTrue("TestJob job was not run after throwing an exception", 2 <= job.totalRuns.get());
}
}