workflow error: Manager is shut down

Hi everyone,

I am trying to copy a file from a Windows VM to VCO, using the ootb workflow "Copy file from guest to vCO". The workflow ran successfully many times, but stopped working. The error message seems to suggest that a service (on vCenter or ESXi ?) is stopped. That would explain why the workflow stopped working without code changes. The problem is apparently caused by an IllegalStateException:

Caused by: java.lang.IllegalStateException: Manager is shut down.

at org.apache.http.impl.conn.SingleClientConnManager.assertStillUp(SingleClientConnManager.java:177)

What Manager is this referring to? Any ideas what I need to check / fix?

Below are two snippets from catalina.out, first the good execution, then the bad one.

good:

2014-03-24 15:04:48.709-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowHandler] getAttributeFromCache WorkflowTokenAttribute [name=guestFilePath, type=string, value=C:\toolsProvisioning\out\20140323222438980-GENERIC.xlsx-Servers.txt]
2014-03-24 15:04:48.709-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowScriptRunner] Fetching guestFilePath
2014-03-24 15:04:48.709-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowHandler] setAttributeToCache guestFilePath
2014-03-24 15:04:48.709-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowHandler] getAttributeFromCache WorkflowTokenAttribute [name=vcoPath, type=string, value=/var/lib/vco/app-server/temp/toolsProvisioning/in/20140323222438980-GENERIC.xlsx-Servers.txt]
2014-03-24 15:04:48.709-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowScriptRunner] Fetching vcoPath
2014-03-24 15:04:48.709-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowHandler] setAttributeToCache vcoPath
2014-03-24 15:04:48.710-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WrappedJavaMethod] Calling method : public com.vmware.vim.vi4.FileTransferInformation com.vmware.vmo.plugin.vi4.model.VimGuestFileManager.initiateFileTransferFromGuest(com.vmware.vmo.plugin.vi4.model.VimVirtualMachine,com.vmware.vim.vi4.GuestAuthentication,java.lang.String) throws com.vmware.vim.vi4.VimFault,com.vmware.vim.vi4.RuntimeFault,org.apache.axis.AxisFault
2014-03-24 15:04:49.967-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WrappedJavaMethod] Pre-Result : class com.vmware.vim.vi4.FileTransferInformation - com.vmware.vim.vi4.FileTransferInformation@e373905
2014-03-24 15:04:49.967-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WrappedJavaMethod] Post-Result : ch.dunes.vso.sdk.DynamicWrapper@1a3e9cce / DynamicWrapper (Instance) : [VcFileTransferInformation]-[class com.vmware.vim.vi4.FileTransferInformation] -- VALUE : com.vmware.vim.vi4.FileTransferInformation@e373905
2014-03-24 15:04:49.968-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WrappedJavaMethod] Calling method : public java.lang.Boolean com.vmware.vmo.plugin.vi4.model.VimGuestFileManager.downloadFile(java.lang.String,com.vmware.vim.vi4.FileTransferInformation) throws org.apache.http.client.ClientProtocolException,java.io.IOException,java.security.KeyStoreException,java.security.NoSuchAlgorithmException
2014-03-24 15:04:50.100-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WrappedJavaMethod] Pre-Result : class java.lang.Boolean - true
2014-03-24 15:04:50.100-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WrappedJavaMethod] Post-Result : true
2014-03-24 15:04:50.100-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowHandler] setAttributeToCache result
2014-03-24 15:04:50.100-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowScriptRunner] Done executing script
2014-03-24 15:04:50.100-0400 [WorkflowExecutorPool-Thread-270] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f57af5a009a2} [WorkflowItemTaskRunner]     END Execution of item : Main Process / 'Scriptable task'(item1)

bad:

2014-03-24 15:32:47.223-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WorkflowHandler] getAttributeFromCache WorkflowTokenAttribute [name=guestFilePath, type=string, value=C:\toolsProvisioning\out\20140323225236480-GENERIC.xlsx-Servers.txt]

2014-03-24 15:32:47.223-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WorkflowHandler] getAttributeFromCache WorkflowTokenAttribute [name=vcoPath, type=string, value=/var/lib/vco/app-server/temp/toolsProvisioning/in/20140323225236480-GENERIC.xlsx-Servers.txt]

2014-03-24 15:32:47.224-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WrappedJavaMethod] Calling method : public com.vmware.vim.vi4.FileTransferInformation com.vmware.vmo.plugin.vi4.model.VimGuestFileManager.initiateFileTransferFromGuest(com.vmware.vmo.plugin.vi4.model.VimVirtualMachine,com.vmware.vim.vi4.GuestAuthentication,java.lang.String) throws com.vmware.vim.vi4.VimFault,com.vmware.vim.vi4.RuntimeFault,org.apache.axis.AxisFault

2014-03-24 15:32:47.880-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WrappedJavaMethod] Pre-Result : class com.vmware.vim.vi4.FileTransferInformation - com.vmware.vim.vi4.FileTransferInformation@786ca916

2014-03-24 15:32:47.880-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WrappedJavaMethod] Post-Result : ch.dunes.vso.sdk.DynamicWrapper@46c98a99 / DynamicWrapper (Instance) : [VcFileTransferInformation]-[class com.vmware.vim.vi4.FileTransferInformation] -- VALUE : com.vmware.vim.vi4.FileTransferInformation@786ca916

2014-03-24 15:32:47.880-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WrappedJavaMethod] Calling method : public java.lang.Boolean com.vmware.vmo.plugin.vi4.model.VimGuestFileManager.downloadFile(java.lang.String,com.vmware.vim.vi4.FileTransferInformation) throws org.apache.http.client.ClientProtocolException,java.io.IOException,java.security.KeyStoreException,java.security.NoSuchAlgorithmException

2014-03-24 15:32:48.007-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WrappedJavaMethod] Invocation exception during 'public java.lang.Boolean com.vmware.vmo.plugin.vi4.model.VimGuestFileManager.downloadFile(java.lang.String,com.vmware.vim.vi4.FileTransferInformation) throws org.apache.http.client.ClientProtocolException,java.io.IOException,java.security.KeyStoreException,java.security.NoSuchAlgorithmException' call on object 'GuestFileManager<guestOperationsFileManager>'

java.lang.reflect.InvocationTargetException

at sun.reflect.GeneratedMethodAccessor1246.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at ch.dunes.vso.sdk.WrappedJavaMethod.call(WrappedJavaMethod.java:142)

at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473)

at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)

at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)

at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)

at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)

at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120)

at org.mozilla.javascript.Context.evaluateString(Context.java:1079)

at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:236)

at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:223)

at ch.dunes.workflow.engine.mbean.WorkflowScriptRunner.execute(WorkflowScriptRunner.java:188)

at ch.dunes.workflow.engine.mbean.runner.WorkflowItemTaskRunner.execute(WorkflowItemTaskRunner.java:45)

at ch.dunes.workflow.engine.mbean.runner.WorkflowItemTaskRunner.execute(WorkflowItemTaskRunner.java:24)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.executeItem(WorkflowHandler.java:911)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.requestElementExecution(WorkflowHandler.java:877)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.handleWorkflowTokenNextStep(WorkflowHandler.java:629)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.executeToken(WorkflowHandler.java:501)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.handleTokenExecution(WorkflowHandler.java:416)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.access$200(WorkflowHandler.java:93)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler$1.execute(WorkflowHandler.java:326)

at ch.dunes.model.ar.AccessRightsTemplate.executeWithAccessRights(AccessRightsTemplate.java:16)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.doExecute(WorkflowHandler.java:322)

at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.run(WorkflowHandler.java:226)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.IllegalStateException: Manager is shut down.

at org.apache.http.impl.conn.SingleClientConnManager.assertStillUp(SingleClientConnManager.java:177)

at org.apache.http.impl.conn.SingleClientConnManager.releaseConnection(SingleClientConnManager.java:262)

at org.apache.http.impl.conn.AbstractClientConnAdapter.releaseConnection(AbstractClientConnAdapter.java:307)

at org.apache.http.conn.BasicManagedEntity.releaseManagedConnection(BasicManagedEntity.java:191)

at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)

at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:237)

at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:186)

at com.vmware.vmo.plugin.vi4.model.VimGuestFileManager.downloadFile(VimGuestFileManager.java:123)

... 32 more

2014-03-24 15:32:48.009-0400 [WorkflowExecutorPool-Thread-276] INFO {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [SCRIPTING_LOG] Manager is shut down. (Workflow:Copy file from guest to vCO / Scriptable task (item1)#10)

2014-03-24 15:32:48.009-0400 [WorkflowExecutorPool-Thread-276] DEBUG {vcoadmin:Main Process:30d8a446-cb4c-4117-a2f2-92484d0bb1fd:ff80808144e0e9500144f594900709dd} [WorkflowScriptRunner] Execute exception

org.mozilla.javascript.EvaluatorException: Manager is shut down. (Workflow:Copy file from guest to vCO / Scriptable task (item1)#10)

at org.mozilla.javascript.DefaultErrorReporter.runtimeError(DefaultErrorReporter.java:77)

at org.mozilla.javascript.Context.reportRuntimeError(Context.java:913)

at org.mozilla.javascript.Context.reportRuntimeError(Context.java:969)

at ch.dunes.vso.sdk.WrappedJavaMethod.call(WrappedJavaMethod.java:167)

at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473)

at script(Workflow:Copy file from guest to vCO / Scriptable task (item1):10)