javax.ejb.NoSuchEJBException after redeploying EJBs

Using Glassfish 3.0.1 ... If I have a web application accessing EJBs in another application remotely, and the remote application containing the EJBs is redeployed, I get a javax.ejb.NoSuchEJBException (see stacktrace below).

Shouldn't this work? I can see that the EJB in question was successfully deployed, using the exact same JNDI name.

Is there any other way to fix this than to restart the web application? It should be noted that in this particular example that the stacktrace is from, I'm accessing a servlet that injects the bean with CDI:

public class StatusServlet extends HttpServlet {

    @Inject
    private StatusService statusService;

    @Override
    public void doGet(final HttpServletRequest req, final HttpServletResponse res) throws IOException {
        res.getWriter().write(statusService.getStatus());
    }
}

The injection is done with the following producer to get the right EJB:

public class StatusServiceProducer extends AbstractServiceProducer {

 @EJB(name = "StatusService")
 private StatusService service;

 @Produces
 public StatusService getService(final InjectionPoint ip) {
  return service;
 }

}

A producer is used to make it easier to wrap the service in a proxy, and to make it easier to change how the EJBs are looked up.

The StatusService interface and implementation is as follows:

@Stateless(name = "StatusService")
public class StatusServiceImpl implements StatusService {
    private static final String OK = "OK";

    public String getStatus() {
        // Some code
        return OK;
    }
}
public interface StatusService {
 String getStatus();
}

Edit: Provided my own possible solution. Is there a better one?

Full stacktrace:

[#|2011-01-12T10:45:28.273+0100|WARNING|glassfish3.0.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=50;_ThreadName=http-thread-pool-8080-(1);|StandardWrapperValve[Load
Balancer status servlet]: PWC1406: Servlet.service() for servlet Load
Balancer status servlet threw exception
javax.ejb.NoSuchEJBException
       at org.example.service._StatusService_Wrapper.getStatus(org/example/service/_StatusService_Wrapper.java)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at no.evote.service.cache.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:34)
       at $Proxy760.getStatus(Unknown Source)
       at no.evote.presentation.StatusServlet.doGet(StatusServlet.java:25)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
       at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
       at net.balusc.http.multipart.MultipartFilter.doFilter(MultipartFilter.java:78)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
       at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
       at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:325)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:226)
       at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
       at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
       at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
       at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
       at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
       at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
       at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
       at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
       at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
       at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
       at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
       at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
       at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
       at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.rmi.NoSuchObjectException: CORBA OBJECT_NOT_EXIST
1330446338 No; nested exception is:
       org.omg.CORBA.OBJECT_NOT_EXIST: ----------BEGIN server-side stack
trace----------
org.omg.CORBA.OBJECT_NOT_EXIST:   vmcid: OMG  minor code: 2  completed: No
       at com.sun.corba.ee.impl.logging.OMGSystemException.noObjectAdaptor(OMGSystemException.java:3457)
       at com.sun.corba.ee.impl.logging.OMGSystemException.noObjectAdaptor(OMGSystemException.java:3475)
       at com.sun.corba.ee.impl.oa.poa.POAFactory.find(POAFactory.java:222)
       at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.findObjectAdapter(CorbaServerRequestDispatcherImpl.java:450)
       at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:209)
       at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1841)
       at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:119)
       at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:235)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:187)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
       at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
       at no.evote.service.__StatusService_Remote_DynamicStub.getStatus(no/evote/service/__StatusService_Remote_DynamicStub.java)
       at no.evote.service._StatusService_Wrapper.getStatus(no/evote/service/_StatusService_Wrapper.java)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at no.evote.service.cache.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:34)
       at $Proxy760.getStatus(Unknown Source)
       at no.evote.presentation.StatusServlet.doGet(StatusServlet.java:25)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
       at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
       at net.balusc.http.multipart.MultipartFilter.doFilter(MultipartFilter.java:78)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
       at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLoc开发者_JS百科kingStandardPipeline.java:85)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
       at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:325)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:226)
       at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
       at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
       at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
       at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
       at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
       at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
       at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
       at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
       at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
       at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
       at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
       at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
       at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
       at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
       at java.lang.Thread.run(Thread.java:662)
Caused by: org.omg.PortableServer.POAPackage.AdapterNonExistent:
IDL:omg.org/PortableServer/POA/AdapterNonExistent:1.0
       at com.sun.corba.ee.impl.oa.poa.POAImpl.find_POA(POAImpl.java:1057)
       at com.sun.corba.ee.impl.oa.poa.POAFactory.find(POAFactory.java:218)
       ... 48 more

----------END server-side stack trace----------  vmcid: OMG  minor
code: 2  completed: No
       at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:280)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
       at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
       at no.evote.service.__StatusService_Remote_DynamicStub.getStatus(no/evote/service/__StatusService_Remote_DynamicStub.java)
       ... 39 more
Caused by: org.omg.CORBA.OBJECT_NOT_EXIST: ----------BEGIN server-side
stack trace----------
org.omg.CORBA.OBJECT_NOT_EXIST:   vmcid: OMG  minor code: 2  completed: No
       at com.sun.corba.ee.impl.logging.OMGSystemException.noObjectAdaptor(OMGSystemException.java:3457)
       at com.sun.corba.ee.impl.logging.OMGSystemException.noObjectAdaptor(OMGSystemException.java:3475)
       at com.sun.corba.ee.impl.oa.poa.POAFactory.find(POAFactory.java:222)
       at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.findObjectAdapter(CorbaServerRequestDispatcherImpl.java:450)
       at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:209)
       at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1841)
       at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:119)
       at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:235)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:187)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
       at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
       at no.evote.service.__StatusService_Remote_DynamicStub.getStatus(no/evote/service/__StatusService_Remote_DynamicStub.java)
       at no.evote.service._StatusService_Wrapper.getStatus(no/evote/service/_StatusService_Wrapper.java)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at no.evote.service.cache.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:34)
       at $Proxy760.getStatus(Unknown Source)
       at no.evote.presentation.StatusServlet.doGet(StatusServlet.java:25)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
       at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
       at net.balusc.http.multipart.MultipartFilter.doFilter(MultipartFilter.java:78)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
       at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
       at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:325)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:226)
       at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
       at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
       at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
       at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
       at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
       at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
       at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
       at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
       at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
       at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
       at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
       at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
       at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
       at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
       at java.lang.Thread.run(Thread.java:662)
Caused by: org.omg.PortableServer.POAPackage.AdapterNonExistent:
IDL:omg.org/PortableServer/POA/AdapterNonExistent:1.0
       at com.sun.corba.ee.impl.oa.poa.POAImpl.find_POA(POAImpl.java:1057)
       at com.sun.corba.ee.impl.oa.poa.POAFactory.find(POAFactory.java:218)
       ... 48 more

----------END server-side stack trace----------  vmcid: OMG  minor
code: 2  completed: No
       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
       at com.sun.corba.ee.impl.protocol.giopmsgheaders.MessageBase.getSystemException(MessageBase.java:913)
       at com.sun.corba.ee.impl.protocol.giopmsgheaders.ReplyMessage_1_2.getSystemException(ReplyMessage_1_2.java:129)
       at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.getSystemExceptionReply(CorbaMessageMediatorImpl.java:681)
       at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.processResponse(CorbaClientRequestDispatcherImpl.java:510)
       at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:153)
       at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:235)
       at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:187)
       ... 42 more
|#]

The only solution would be to not use @EJB injection as a field in the servlet (where it gets injected at class loading, so application startup or first use of the servlet, but rather to request a new instance for each request. Sadly, that does mean you can't use the @EJB injection annotation and will have to handle the EJB in the old way using JNDI lookup etc.

From a discussion on the Glassfish forums:

The injected reference is updated only when a new instance that it is injected into is created by the EJB container. So if you look it up all the time, you get an instance with the updated reference, but if your code holds on to the original instance, or if it is pooled, the injected reference remains stale.

So the root problem is that the reference to the EJB is stale. I have found two solutions to this problem:

1. Look up the EJB on each request (do not use CDI)
2. Wrap EJBs in a proxy that looks up the EJB if the reference is stale

With the second solution, CDI can still be used. You'll need a producer, and an InvocationHandler. The code below is a simplified version of the code I use in my project, and it might not compile properly, but it will show the idea and concepts used in making this work. YMMV.

First, in the code that uses the EJB, you use @Inject, as follows:

public class Foo {
    @Inject
    MyEjb obj;
}

Second, you'll need a producer that simply returns a wrapped version of the EJB:

public class MyEjbProducer {
    @EJB(name = "MyEjb")
    private MyEjb obj;

    @Produces
    public MyEjb getEjb(final InjectionPoint ip) {
        InvocationHandler handler = new MyInvocationHandler(obj, MyEjb.class);
        return (MyEjb)Proxy.newProxyInstance(obj.getClass().getClassLoader(), obj.getClass().getInterfaces(), handler);
    }
}

Third, the invocation handler needs to test for NoSuchEjbException and look up the bean again:

public class MyInvocationHandler implements InvocationHandler {
    private MyEjb obj;
    private Class lookupClass;

    public MyInvocationHandler(MyEjb obj, Class lookupClass) {
        this.obj = obj;
        this.lookupClass = lookupClass;
    }

    public Object invoke (...) {
        try {
            return method.invoke(impl, args);
        } catch (final InvocationTargetException e) {
            if (e.getTargetException() instanceof NoSuchEJBException) {
                // 1. look up EJB again
                // 2. try calling invoke again                    
            }
        }
    }
}

This code is by no means complete. Some of the code, such as looking up the EJB and invoking the method again has been omitted.

Doing it this way will work. Another method would be to do something similar, but adding a separate method to the EJBs for checking for staleness, instead of using the method that the user attempts to invoke.

What do you think? Is there a better solution?

It has been corrected here:

https://wikis.oracle.com/display/GlassFish/GlassFishv3.1EJB

Retain unique app. id / ejb ids across redeployment MS5 Mahesh EJB-6-1 Done by MS3

So, Glassfish 3.1 should work ok with this.