IT-Jobs, die dich glücklich machen

#WTF: Groovy crashes with a Java Runtime EXCEPTION_ACCESS_VIOLATION (0xc0000005)

That was a really annoying one: Regularly one of our microservices crashes with an internal JVM exception. The solution for fixing this issue was very surprising.
Java Groovy Access Violation Bug

Our Jobpushy services consist of three types of microservices (which should better be called "macroservices" ;)): The user-frontend, a crawler-component and in between the managing-server. All services are running on the Java Virtual Machine and are using Groovy (and in part Grails) as the programming language. Bye the way: Each of those macroservices can scale horizontally. Hence our software stack makes sure that we shouldn’t have any outages because of heavy load.

But surprisingly whenever our crawler-component delivers with high load new job-posting-results, the managing-server got the following Java-VM crashes from time to time:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006a1ef094, pid=1896, tid=0x00000000000041a0
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V  [jvm.dll+0x3ef094]
…..
Current CompileTask:
C1:  51916 16313   !   3       jobpushy.server.alert.CheckItemService$$EnhancerBySpringCGLIB$$e6c804da::createCheckItemsForExtraction (292 bytes)

Excerpt from the hs_err_<pid>;.log file which is created during JVM crashed

Divide and Conquer

Looking at the log-file above we’ve got a first hint: The “current CompileTask” while crashing was the method createCheckItemsForExtraction in the service CheckItemService.

Using the divide and conquer method I tried to check which part of the method causes the crash. And again a surprise: Even if removing all code inside the method, the crash still occurs. What the hell is going on? ;)

The solution

There is no source-code inside the method and it is still crashing? There must be something going on with the method-signature. The method is defined and called as follows:

void createCheckItemsForExtraction(int index, ScrapingSiteResult scarpingResult) { // empty for testing } .... Integer pageIndex = myValue createCheckItemsForExtraction(myValue, result)

A little bit more try and error testing has found the solution. The first parameter was defined as int type, but was called with an Integer type. That shouldn’t be a problem. There is a language feature called autoboxing, right? But as it turns out, changing the method-signature from int to Integer fixes the problem:

void createCheckItemsForExtraction(Integer index, ScrapingSiteResult scarpingResult) {

I still have no explanation for this. There seems to be a weird bug with Groovy 2.4, Java 1.8 and autoboxing in conjunction with method parameters.



About the #WTF posts on this blog:
These are short articles that show simple solutions to problems that lead to a headache in the first place. This serves two goals: 1.) Remembering the solution (and own stupidity) for the next time 2.) Helping other people with similar problems.

Für Hinweise, Feedback oder Fragen kontaktiere uns bitte via Twitter@Jobpushy, unserem Slack Workspace oder über unsere sonstigen Kontaktangaben.