ConcurrentModificationException

[andrewjmc]

I have a script which frequently gives ConcurrentModificationException errors soon after starting.

I think it will be very hard to provide anything near to a reproducible example! My script uses extensive caching:

We reach the stage where it looks like two jobs have been submitted to PBS.

executor >  local (1), pbspro (2)
[f2/39ba5c] process > removeDuplicates        [100%] 1 of 1 ✔
[-        ] process > step_2_trim_quality     -
[-        ] process > step_2_code             -
[skipped  ] process > step_3_kraken_batch (3) [100%] 5 of 5, stored: 5 ✔
[skipped  ] process > step_3_code             [100%] 1 of 1, stored: 1 ✔
[skipped  ] process > step_4_hg_alignment     [100%] 1 of 1, stored: 1 ✔
[-        ] process > step_4_code             -
[-        ] process > step_5_panphlan         -
[-        ] process > step_5_code             -
[ba/633101] process > step_6_kraken_batch (4) [100%] 3 of 3, stored: 3

Actor Thread 8 crashed, but I can't tell what it was doing at the point it crashed.

Mar-26 12:11:02.793 [Actor Thread 8] ERROR nextflow.extension.DataflowHelper - @unknown
java.util.ConcurrentModificationException: null
        at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
        at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2329)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2315)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2356)
        at org.codehaus.groovy.runtime.dgm$186.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at nextflow.extension.OperatorEx$_flatMap_closure2.doCall(OperatorEx.groovy:194)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:108)
        at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
        at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
        at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
        at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Mar-26 12:11:02.799 [Actor Thread 8] DEBUG nextflow.Session - Session aborted -- Cause: java.util.ConcurrentModificationException
Mar-26 12:11:02.827 [Actor Thread 8] DEBUG nextflow.Session - The following nodes are still active:
[process] step_4_code
  status=ACTIVE
  port 0: (value) bound ; channel: version
  port 1: (value) bound ; channel: prior_versions
  port 2: (value) bound ; channel: code
  port 3: (value) bound ; channel: prior_code
  port 4: (value) bound ; channel: script_name
  port 5: (cntrl) -     ; channel: $

[process] step_5_panphlan
  status=ACTIVE
  port 0: (value) bound ; channel: files_in
  port 1: (value) bound ; channel: version
  port 2: (value) bound ; channel: prior_versions
  port 3: (value) bound ; channel: script_name
  port 4: (queue) closed; channel: organism
  port 5: (cntrl) -     ; channel: $

[process] step_5_code
  status=ACTIVE
  port 0: (value) OPEN  ; channel: version
  port 1: (value) bound ; channel: prior_versions
  port 2: (value) bound ; channel: code
  port 3: (value) OPEN  ; channel: prior_code
  port 4: (value) bound ; channel: script_name
  port 5: (cntrl) -     ; channel: $

Mar-26 12:11:02.829 [main] DEBUG nextflow.Session - Session await > all process finished
Mar-26 12:11:02.832 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-26 12:11:02.845 [main] WARN  n.processor.TaskPollingMonitor - Killing pending tasks (2)
Mar-26 12:11:03.527 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=1; submittedCount=2; runningCount=-2; retriesCount=0; abortedCount=2; succeedDuration=42ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=-40; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 MB; ]
Mar-26 12:11:04.298 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Mar-26 12:11:04.528 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

My script is long but ultimately not that complex. It is compiled from separate files which I change track, and outputs its own code with the processed data, which is the main novelty. It worked no problem for a long time, and I'm not sure what it is about the inputs or processing which is leading to this error.

I'd be very grateful for any pointers for chasing down ConcurrentModificationExceptions.

main.txt

Thanks,

Andrew

[pditommaso]

I suspect a problem with in your script. Please include a replicate test case.

[andrewjmc]

Thanks, I will give it a go. I have the heart sinking feeling that it will be very difficult to replicate, since its occurrence appears to depend upon cached outputs (it had no problem a few days ago, but now that there are new cached outputs, it is raising the error).

Is there any way to get insight into which aspect of the pipeline is triggering the error?