Our Jobpushy services consist of three types of microservices (which should better be called "macroservices" ;)): The user-frontend, a crawler-component and in between the managing-server. All services are running on the Java Virtual Machine and are using Groovy (and in part Grails) as the programming language. Bye the way: Each of those macroservices can scale horizontally. Hence our software stack makes sure that we shouldn’t have any outages because of heavy load.
But surprisingly whenever our crawler-component delivers with high load new job-posting-results, the managing-server got the following Java-VM crashes from time to time:
# # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006a1ef094, pid=1896, tid=0x00000000000041a0 # JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04) # Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode windows-amd64 compressed oops) # Problematic frame: # V [jvm.dll+0x3ef094] ….. Current CompileTask: C1: 51916 16313 ! 3 jobpushy.server.alert.CheckItemService$$EnhancerBySpringCGLIB$$e6c804da::createCheckItemsForExtraction (292 bytes)
Excerpt from the hs_err_<pid>.log file which is created during JVM crashed
Looking at the log-file above we’ve got a first hint: The “current CompileTask” while crashing was the method createCheckItemsForExtraction in the service CheckItemService.
Using the divide and conquer method I tried to check which part of the method causes the crash. And again a surprise: Even if removing all code inside the method, the crash still occurs. What the hell is going on? ;)
There is no source-code inside the method and it is still crashing? There must be something going on with the method-signature. The method is defined and called as follows:
void createCheckItemsForExtraction(int index, ScrapingSiteResult scarpingResult) {
// empty for testing
}
....
Integer pageIndex = myValue
createCheckItemsForExtraction(myValue, result)
A little bit more try and error testing has found the solution. The first parameter was defined as int type, but was called with an Integer type. That shouldn’t be a problem. There is a language feature called autoboxing, right? But as it turns out, changing the method-signature from int to Integer fixes the problem:
void createCheckItemsForExtraction(Integer index, ScrapingSiteResult scarpingResult) {
I still have no explanation for this. There seems to be a weird bug with Groovy 2.4, Java 1.8 and autoboxing in conjunction with method parameters.
About the #WTF posts on this blog:
These are short articles that show simple solutions to problems that lead to a headache in the first place. This serves two goals: 1.) Remembering the solution (and own stupidity) for the next time 2.) Helping other people with similar problems.