前言

呵呵 最近在 rcp项目的中导入 spark 环境之后, 呵呵 gg 了, 项目也没有报错, 差不多是 30s 之后超时了 

断点一下 看见代码抛出异常的地方是在 spark driver 暴露 tcp 服务的地方 

呵呵 当时还是很令人费解的, 我还以为是 rcp 项目在创建 tcp 服务的时候有什么限制呢 ? 

呵呵 这里记录一下  

业务代码里面的 SpringContext.init 

spark 的死锁

java12 中引入 spark 启动不了, 卡在了 TransportServer.init(main线程)channelFuture = bootstrap.bind(address);channelFuture.syncUninterruptibly();
另外一个线程 rpc-boss-3 在执行 如下任务的时候卡住了 
register0:502, AbstractChannel$AbstractUnsafe和 EquinoxClassLoader 有关系, 加载 io.netty.bootstrap.ServerBootstrap$1$1 的时候卡住了?EclipseLazyStarter.postFindLocalClass 卡住了
呵呵 死锁, main 线程拥有了 stateChangeLock, 然后等待 rpc-boss-3 唤醒自己, 之后 rpc-boss-3 在等待 stateChangeLock
main 线程里面 start:468, Module (org.eclipse.osgi.container) 获取了 stateChangeLock 的锁
原因是 Activator 里面 start 里面初始化的 SpringContext, 导致了如上的一系列

spark 初始化, 卡在了如下位置 

新的死锁 

呵呵 今天尝试复现之前的这个问题, 呵呵 结果没有复现成功, 因为 这里似乎是变成了另外的一个死锁了 .. 

大致的情况都是一样的, 这里就不去浪费时间去复现之前的 spark 的那个死锁了 

这里的死锁主要是 Thread-4 和 Thread-5[main] 之间造成了死锁

Thread-5 持有 EquinoxReentrantLock@300c6244 在等待 ContextFinder@6ed3ccb2 

Thread-4 持有 ContextFinder@6ed3ccb2 在等待 EquinoxReentrantLock@300c6244 

Thread-4 申请 EquinoxReentrantLock@300c6244  有一个 30s 的超时设置, 因此 打印出了 如下面所看到的堆栈信息 

!ENTRY org.eclipse.osgi 2 0 2021-01-16 22:24:43.423
!MESSAGE While loading class "org.springframework.cache.CacheManager", thread "Thread[Thread-4,5,main]" timed out waiting (73888ms) for thread "Thread[main,5,main]" to finish starting bundle "MyArp-client_1.0.0.qualifier [1]". To avoid deadlock, thread "Thread[Thread-4,5,main]" is proceeding but "org.springframework.cache.CacheManager" may not be fully initialized.
!STACK 0
org.osgi.framework.BundleException: Unable to acquire the state change lock for the module: osgi.identity; type="osgi.bundle"; version:Version="1.0.0.qualifier"; osgi.identity="MyArp-client"; singleton:="true" [id=1] STARTED [STARTED]at org.eclipse.osgi.container.Module.lockStateChange(Module.java:350)at org.eclipse.osgi.container.Module.start(Module.java:419)at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at org.eclipse.osgi.internal.framework.ContextFinder.loadClass(ContextFinder.java:142)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at java.base/java.lang.Class.forName0(Native Method)at java.base/java.lang.Class.forName(Class.java:468)at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:211)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)at org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver.lambda$new$0(OnClassCondition.java:150)at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.util.concurrent.TimeoutException: Timeout after waiting 30 seconds to acquire the lock.at org.eclipse.osgi.container.Module.lockStateChange(Module.java:347)... 22 more
Caused by: org.eclipse.osgi.framework.util.ThreadInfoReport: Thread dumpThreadId: 1 ThreadName: main ThreadState: BLOCKEDBlocked On: org.eclipse.osgi.internal.framework.ContextFinder@6ed3ccb2 LockOwnerId: 26 LockOwnerName: Thread-4Synchronizers Locked: java.util.concurrent.locks.ReentrantLock$NonfairSync@356da172Monitors Locked: java.lang.Object@51340208Stack Trace: java.base@15/java.lang.Class.forName0(Native Method)java.base@15/java.lang.Class.forName(Class.java:468)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:214)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)org.springframework.boot.autoconfigure.condition.OnClassCondition.resolveOutcomesThreaded(OnClassCondition.java:69)org.springframework.boot.autoconfigure.condition.OnClassCondition.getOutcomes(OnClassCondition.java:53)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.match(FilteringSpringBootCondition.java:49)org.springframework.boot.autoconfigure.AutoConfigurationImportSelector$ConfigurationClassFilter.filter(AutoConfigurationImportSelector.java:370)org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getAutoConfigurationEntry(AutoConfigurationImportSelector.java:128)org.springframework.boot.autoconfigure.AutoConfigurationImportSelector$AutoConfigurationGroup.process(AutoConfigurationImportSelector.java:434)org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGrouping.getImports(ConfigurationClassParser.java:879)org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.processGroupImports(ConfigurationClassParser.java:809)org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorHandler.process(ConfigurationClassParser.java:780)org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:193)org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:319)org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:236)org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280)org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:96)org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:89)MyArpclient.context.SpringContext.init(SpringContext.java:52)MyArpclient.Activator.start(Activator.java:24)org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814)org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)java.base@15/java.security.AccessController.executePrivileged(AccessController.java:784)java.base@15/java.security.AccessController.doPrivileged(AccessController.java:554)org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806)org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763)org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1011)org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)org.eclipse.osgi.container.Module.doStart(Module.java:605)org.eclipse.osgi.container.Module.start(Module.java:468)org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)java.base@15/java.lang.ClassLoader.loadClass(ClassLoader.java:522)org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:616)org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.createFromBundle(ReflectionContributionFactory.java:89)org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.doCreate(ReflectionContributionFactory.java:60)org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.create(ReflectionContributionFactory.java:42)org.eclipse.e4.ui.internal.workbench.swt.E4Application.lambda$0(E4Application.java:243)org.eclipse.e4.ui.internal.workbench.swt.E4Application$$Lambda$103/0x0000000800d6ba80.accept(Unknown Source)java.base@15/java.util.Optional.ifPresent(Optional.java:176)org.eclipse.e4.ui.internal.workbench.swt.E4Application.createE4Workbench(E4Application.java:242)org.eclipse.e4.ui.internal.workbench.swt.E4Application.start(E4Application.java:153)org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)java.base@15/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)java.base@15/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)java.base@15/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.base@15/java.lang.reflect.Method.invoke(Method.java:564)app//org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)app//org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)app//org.eclipse.equinox.launcher.Main.run(Main.java:1461)app//org.eclipse.equinox.launcher.Main.main(Main.java:1434)app//MyArpclient.Bootstrap.main(Bootstrap.java:16)ThreadId: 26 ThreadName: Thread-4 ThreadState: RUNNABLEBlocked On: org.eclipse.osgi.internal.container.EquinoxReentrantLock@300c6244[Locked by thread main]Synchronizers Locked: noneMonitors Locked: org.eclipse.osgi.internal.framework.ContextFinder@6ed3ccb2Stack Trace: java.management@15/sun.management.ThreadImpl.dumpThreads0(Native Method)java.management@15/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)java.management@15/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)org.eclipse.osgi.framework.util.ThreadInfoReport.getThreadDump(ThreadInfoReport.java:30)org.eclipse.osgi.framework.util.ThreadInfoReport.<init>(ThreadInfoReport.java:23)org.eclipse.osgi.container.Module.lockStateChange(Module.java:347)org.eclipse.osgi.container.Module.start(Module.java:419)org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)java.base@15/java.lang.ClassLoader.loadClass(ClassLoader.java:522)org.eclipse.osgi.internal.framework.ContextFinder.loadClass(ContextFinder.java:142)java.base@15/java.lang.ClassLoader.loadClass(ClassLoader.java:522)java.base@15/java.lang.Class.forName0(Native Method)java.base@15/java.lang.Class.forName(Class.java:468)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:211)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver.lambda$new$0(OnClassCondition.java:150)org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver$$Lambda$197/0x0000000800e0d7a8.run(Unknown Source)java.base@15/java.lang.Thread.run(Thread.java:832)... 23 more
Root exception:
java.util.concurrent.TimeoutException: Timeout after waiting 30 seconds to acquire the lock.at org.eclipse.osgi.container.Module.lockStateChange(Module.java:347)at org.eclipse.osgi.container.Module.start(Module.java:419)at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at org.eclipse.osgi.internal.framework.ContextFinder.loadClass(ContextFinder.java:142)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at java.base/java.lang.Class.forName0(Native Method)at java.base/java.lang.Class.forName(Class.java:468)at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:211)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)at org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver.lambda$new$0(OnClassCondition.java:150)at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.eclipse.osgi.framework.util.ThreadInfoReport: Thread dump

main 持有 EquinoxReentrantLock@300c6244

main 等待 ContextFinder@6ed3ccb2 

Thread-4 持有 ContextFinder@6ed3ccb2  

Thread-4 等待 EquinoxReentrantLock@300c6244

完