Fail to start convergence server


#1

Getting Started
I’m trying to launch the convergence server but it fails.
Do you know what is the cause?

Environment

Docker

[[email protected] yyyyy]$ docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

OS

[[email protected] yyyyy]$ cat /etc/system-release
Amazon Linux AMI release 2018.03

This is ec2 instance on AWS.

Operation

$ docker run -p "8000:80" --name convergence convergencelabs/convergence-de

Error Logs

[[email protected] yyyyy]$ docker run -p "8000:80" --name convergence convergencelabs/convergence-de
2019-06-07 12:08:43,445 INFO Set uid to user 0 succeeded
2019-06-07 12:08:43,486 INFO supervisord started with pid 1
2019-06-07 12:08:44,489 INFO spawned: 'proxy' with pid 10
2019-06-07 12:08:44,491 INFO spawned: 'convergence' with pid 11
2019-06-07 12:08:44,558 INFO spawned: 'db' with pid 12
2019-06-07 12:08:45,246 INFO exited: db (exit status 1; not expected)
2019-06-07 12:08:46,313 INFO success: proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-07 12:08:46,314 INFO success: convergence entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-07 12:08:46,315 INFO spawned: 'db' with pid 64
2019-06-07 12:08:46,683 INFO exited: db (exit status 1; not expected)
2019-06-07 12:08:48,696 INFO spawned: 'db' with pid 79
2019-06-07 12:08:48,986 INFO exited: db (exit status 1; not expected)
2019-06-07 12:08:52,063 INFO spawned: 'db' with pid 85
2019-06-07 12:08:52,433 INFO exited: db (exit status 1; not expected)
2019-06-07 12:08:53,453 INFO gave up: db entered FATAL state, too many start retries too quickly
12:09:39 INFO  ConvergenceServerNode$ - Using config file: /convergence/etc/convergence-server.conf
12:09:43 INFO  ConvergenceServerNode - Convergence Server Node (1.0.0-rc.3) starting up...
12:10:05 WARN  AutoDown - Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
12:10:07 INFO  ConvergenceServerNode - Convergnece Server Roles: backend, realtimeApi, restApi
12:10:07 INFO  ConvergenceServerNode - auto-install is configured, attempting to connect to the database to determin if the convergence database is installed.
12:10:07 INFO  ConvergenceServerNode - Attempting to connect to the datatbase at uri: remote:127.0.0.1:2424
12:10:08 WARN  SystemErr - Jun 07, 2019 12:10:08 PM com.orientechnologies.common.log.OLogManager log
INFO: Detected limit of amount of simultaneously open files is 4096,  limit of open files for disk cache will be set to 1536

12:10:08 WARN  SystemErr - Jun 07, 2019 12:10:08 PM com.orientechnologies.common.log.OLogManager log
WARNING: Value of limit of simultaneously open files is too small, recommended value is 262144

12:10:08 INFO  ConvergenceServerNode - Connected to datatbase with Server Admin
12:10:08 INFO  ConvergenceServerNode - Checking for convergence database
12:10:09 ERROR ConvergenceServerNode - Could not bootstrap database
com.orientechnologies.orient.core.exception.OStorageException: Cannot connect to the remote server/database 'remote:127.0.0.1:2424/convergence'
	at com.orientechnologies.orient.client.remote.OServerAdmin.networkAdminOperation(OServerAdmin.java:483) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.OServerAdmin.connect(OServerAdmin.java:130) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.core.db.OrientDBRemote.connectEndExecute(OrientDBRemote.java:180) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.core.db.OrientDBRemote.exists(OrientDBRemote.java:199) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.core.db.OrientDB.exists(OrientDB.java:275) ~[com.orientechnologies.orientdb-core-3.0.18.jar:3.0.18]
	at com.convergencelabs.server.ConvergenceServerNode.$anonfun$bootstrapConvergenceDB$1(ConvergenceServerNode.scala:341) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at scala.util.Try$.apply(Try.scala:209) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at com.convergencelabs.server.ConvergenceServerNode.bootstrapConvergenceDB(ConvergenceServerNode.scala:318) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at com.convergencelabs.server.ConvergenceServerNode.start(ConvergenceServerNode.scala:255) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at com.convergencelabs.server.ConvergenceServerNode$.$anonfun$main$7(ConvergenceServerNode.scala:107) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at com.convergencelabs.server.ConvergenceServerNode$.$anonfun$main$7$adapted(ConvergenceServerNode.scala:105) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at scala.util.Success.$anonfun$map$1(Try.scala:251) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at scala.util.Success.map(Try.scala:209) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at com.convergencelabs.server.ConvergenceServerNode$.$anonfun$main$6(ConvergenceServerNode.scala:105) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at scala.util.Success.flatMap(Try.scala:247) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at com.convergencelabs.server.ConvergenceServerNode$.$anonfun$main$5(ConvergenceServerNode.scala:104) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at scala.util.Success.flatMap(Try.scala:247) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at com.convergencelabs.server.ConvergenceServerNode$.$anonfun$main$4(ConvergenceServerNode.scala:103) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at scala.util.Success.flatMap(Try.scala:247) ~[org.scala-lang.scala-library-2.12.6.jar:?]
	at com.convergencelabs.server.ConvergenceServerNode$.$anonfun$main$3(ConvergenceServerNode.scala:102) ~[com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at scala.util.Success.flatMap(Try.scala:247) [org.scala-lang.scala-library-2.12.6.jar:?]
	at com.convergencelabs.server.ConvergenceServerNode$.main(ConvergenceServerNode.scala:101) [com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
	at com.convergencelabs.server.ConvergenceServerNode.main(ConvergenceServerNode.scala) [com.convergencelabs.convergence-server-core-1.0.0-rc.3.jar:1.0.0-rc.3]
Caused by: com.orientechnologies.common.io.OIOException: Error on connecting to 127.0.0.1:2424/convergence
at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNetworkConnection(ORemoteConnectionPool.java:62) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:68) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:15) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:95) ~[com.orientechnologies.orientdb-core-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.acquire(ORemoteConnectionPool.java:90) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:93) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.OStorageRemote.getNetwork(OStorageRemote.java:1934) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.OServerAdmin.networkAdminOperation(OServerAdmin.java:468) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	... 23 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_191]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_191]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_191]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_191]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_191]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_191]
	at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.<init>(OChannelBinaryAsynchClient.java:72) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNetworkConnection(ORemoteConnectionPool.java:52) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:68) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:15) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:95) ~[com.orientechnologies.orientdb-core-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.acquire(ORemoteConnectionPool.java:90) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:93) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.OStorageRemote.getNetwork(OStorageRemote.java:1934) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	at com.orientechnologies.orient.client.remote.OServerAdmin.networkAdminOperation(OServerAdmin.java:468) ~[com.orientechnologies.orientdb-client-3.0.18.jar:3.0.18]
	... 23 more
12:10:09 INFO  ConvergenceServerNode$ - JVM Shutdown Hook Invoked, stopping services
2019-06-07 12:10:09,586 INFO exited: convergence (exit status 0; expected)
2019-06-07 12:10:09,597 INFO spawned: 'convergence' with pid 118
2019-06-07 12:10:10,598 INFO success: convergence entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

…Repeat this.

Although the login screen is displayed, it seems that creation of DB has failed and login is not possible.

Could you tell me how to fix it?

By the way

But succeeds on a local machine with similar environment.

Local environment

Docker

[[email protected] xxxx]$ docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

OS

[[email protected] xxxx]$ cat /etc/system-release
Amazon Linux AMI release 2017.03

This is running on vagrant.


#2

In the example above, it was t2.micro instance, but it worked when it tried it with t2.small instance.

Are there any conditions, such as the minimum required memory size?


#3

Thanks for bringing this to our attention. A t2.micro only has 1 GB of RAM. I believe this is the issue. The database will likely require about 500MB to run. We also run a few other services such as a proxy and another JVM that hosts the REST and Realtime APIs. That will use several hundred MB as well. Between that, the base OS and the docker engine, I suspect the Database didn’t have enough memory. There are some tuning options that we could explore. But I suspect 1.5GB - 2GB on the EC2 host will probably be the minimum required to run the container.


#4

I understand. Thank you for your perfect explanation.