java - Generalizing DynamoDB + JanusGraph Factory: Lock and Schema Problems - Stack Overflow
Join Stack Overflow to learn, share knowledge, and build your career.
or sign in with
I'm working on generalizing the DynamoDB + JanusGraph tutorial from AWS such that, given a standard .txt file with a standard convention, the program can ingest the data (as a Triple) and create the vertices, properties, and edges. Normally I wouldn't post such a lengthy problem, but it appears that these are all related to 4-5 lines within the same class, ObjectCreationCommand, which I created.
An example Triple looks like this: "name:Jim Henson \t isCreatorOf \t televisionshow:The Muppets"
Left Object: Jim Henson
Left Object Property: name
Relationship: isCreatorOf
Right Object: The Muppets
Right Object Property: televisionshow
Although the program compiles and runs, I get several exceptions thrown that prevent the graph from being filled. When I run the Factory program, it reads all of my triples and puts them into a Hash Set, but then the following error occurs (10 times, but this is only 1 example):
57338 [pool-10-thread-2] ERROR org.janusgraph.graphdb.database.StandardJanusGraph
- Could not commit transaction [10] due to exception
org.janusgraph.diskstorage.locking.TemporaryLockingException: tx 0xc already locked key-column ( 16-165-160-114-116- 30- 98-114- 97-110-100-116-121-112-229,
0) when tx 0x tried to lock
at com.amazon.janusgraph.diskstorage.dynamodb.AbstractDynamoDBStore.acquireLock(AbstractDynamoDBStore.java:132)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore$4.call(MetricInstrumentedStore.java:155)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore$4.call(MetricInstrumentedStore.java:153)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore.runWithMetrics(MetricInstrumentedStore.java:217)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore.acquireLock(MetricInstrumentedStore.java:152)
at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:52)
at org.janusgraph.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:255)
at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:565)
at org.janusgraph.graphdb.mit(StandardJanusGraph.java:694)
at org.janusgraph.graphdb.mit(StandardJanusGraphTx.java:1363)
at org.janusgraph.graphdb.database.mit(ManagementSystem.java:235)
at com.amazon.janusgraph.creator.ObjectCreationCommand.run(ObjectCreationCommand.java:59)
at com.amazon.janusgraph.batch.BatchCommand.run(BatchCommand.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Next, a similar exception is thrown:
57427 [pool-10-thread-10] ERROR com.amazon.janusgraph.example.MarvelGraphFactory
- Error processing line Could not commit transaction due to exception during persistence tx 0xc already locked key-column ( 16-165-160-114-116- 30- 98-114- 97-110-100-116-121-112-229,
0) when tx 0xb77 tried to lock
org.janusgraph.core.JanusGraphException: Could not commit transaction due to exception during persistence
at org.janusgraph.graphdb.mit(StandardJanusGraphTx.java:1374)
at org.janusgraph.graphdb.database.mit(ManagementSystem.java:235)
at com.amazon.janusgraph.creator.ObjectCreationCommand.run(ObjectCreationCommand.java:59)
at com.amazon.janusgraph.batch.BatchCommand.run(BatchCommand.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.janusgraph.core.JanusGraphException: Unexpected exception
at org.janusgraph.graphdb.mit(StandardJanusGraph.java:798)
at org.janusgraph.graphdb.mit(StandardJanusGraphTx.java:1363)
... 6 more
Caused by: org.janusgraph.diskstorage.locking.TemporaryLockingException: tx 0xc already locked key-column ( 16-165-160-114-116- 30- 98-114- 97-110-100-116-121-112-229,
0) when tx 0xb77 tried to lock
at com.amazon.janusgraph.diskstorage.dynamodb.AbstractDynamoDBStore.acquireLock(AbstractDynamoDBStore.java:132)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore$4.call(MetricInstrumentedStore.java:155)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore$4.call(MetricInstrumentedStore.java:153)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore.runWithMetrics(MetricInstrumentedStore.java:217)
at org.janusgraph.diskstorage.util.MetricInstrumentedStore.acquireLock(MetricInstrumentedStore.java:152)
at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:52)
at org.janusgraph.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:255)
at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:565)
at org.janusgraph.graphdb.mit(StandardJanusGraph.java:694)
... 7 more
And then a schema-related exception is thrown:
58030 [pool-10-thread-4] ERROR com.amazon.janusgraph.example.MarvelGraphFactory
- Error processing line Adding this property for key [~T$SchemaName] and value [rtbrandtype] violates a uniqueness constraint [SystemIndex#~T$SchemaName]
org.janusgraph.core.SchemaViolationException: Adding this property for key [~T$SchemaName] and value [rtbrandtype] violates a uniqueness constraint [SystemIndex#~T$SchemaName]
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:791)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:720)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.makeSchemaVertex(StandardJanusGraphTx.java:847)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.makePropertyKey(StandardJanusGraphTx.java:867)
at org.janusgraph.graphdb.types.StandardPropertyKeyMaker.make(StandardPropertyKeyMaker.java:100)
at com.amazon.janusgraph.creator.ObjectCreationCommand.run(ObjectCreationCommand.java:47)
at com.amazon.janusgraph.batch.BatchCommand.run(BatchCommand.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
And lastly, an exception that I really don't understand is thrown:
58512 [pool-10-thread-8] ERROR com.amazon.janusgraph.example.MarvelGraphFactory
- Error processing line Could not find type for id: 11529
java.lang.IllegalStateException: Could not find type for id: 11529
at mon.base.Preconditions.checkState(Preconditions.java:197)
at org.janusgraph.graphdb.types.vertices.JanusGraphSchemaVertex.name(JanusGraphSchemaVertex.java:59)
at org.janusgraph.graphdb.types.vertices.JanusGraphSchemaVertex.asIndexType(JanusGraphSchemaVertex.java:177)
at org.janusgraph.graphdb.database.management.ManagementSystem.getGraphIndexDirect(ManagementSystem.java:412)
at org.janusgraph.graphdb.database.management.ManagementSystem.getGraphIndex(ManagementSystem.java:422)
at com.amazon.janusgraph.creator.ObjectCreationCommand.run(ObjectCreationCommand.java:55)
at com.amazon.janusgraph.batch.BatchCommand.run(BatchCommand.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Because the final transaction is Null, a NullPointerException is thrown and the transactio thus, my graph is initialized but empty.
Normally I wouldn't post such a lengthy problem, but it appears that these are all related to 4-5 lines within the same class, ObjectCreationCommand, which I created.
ObjectCreationCommand.java
package com.amazon.janusgraph.
import com.amazon.janusgraph.example.TravelGraphF
import com.codahale.metrics.MetricR
import mons.lang.exception.ExceptionU
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalS
import org.apache.tinkerpop.gremlin.structure.V
import org.janusgraph.core.JanusG
import com.amazon.janusgraph.triple.T
import org.janusgraph.core.M
import org.janusgraph.core.PropertyK
import org.janusgraph.core.schema.JanusGraphM
import org.slf4j.L
import java.io.BufferedR
import java.io.FileR
import java.io.IOE
import java.util.I
import java.util.concurrent.TimeU
public class ObjectCreationCommand implements Runnable {
public static JanusG
private static T
private static MetricRegistry REGISTRY;
public static Logger LOG;
private static final String TIMER_LINE = "TravelGraphFactory.line";
private static final String TIMER_CREATE = "TravelGraphFactory.create_";
private static final String COUNTER_GET = "TravelGraphFactory.get_";
public ObjectCreationCommand(JanusGraph graph, Triple triple, MetricRegistry REGISTRY, Logger LOG) {
this.graph =
this.triple =
this.REGISTRY = REGISTRY;
this.LOG = LOG;
public void run() {
JanusGraphManagement mgmt = graph.openManagement();
if (mgmt.getGraphIndex(triple.getRightObjectProperty()) == null) {
final PropertyKey hotelKey = mgmt.makePropertyKey(triple.getRightObjectProperty()).dataType(String.class).make();
mgmt.buildIndex(triple.getRightObjectProperty(), Vertex.class).addKey(hotelKey).unique().buildCompositeIndex();
if (mgmt.getEdgeLabel(triple.getRelationship()) == null) {
mgmt.makeEdgeLabel(triple.getRelationship()).multiplicity(Multiplicity.MANY2ONE).make();
if (mgmt.getGraphIndex(triple.getLeftObjectProperty()) == null) {
final PropertyKey brandTypeKey = mgmt.makePropertyKey(triple.getLeftObjectProperty()).dataType(String.class).make();
mgmt.buildIndex(triple.getLeftObjectProperty(), Vertex.class).addKey(brandTypeKey).unique().buildCompositeIndex();
long start = System.currentTimeMillis();
String RIGHT_OBJECT_PROPERTY = triple.getRightObjectProperty();
Vertex rightObject = graph.addVertex();
rightObject.property(RIGHT_OBJECT_PROPERTY, triple.getRightObject());
REGISTRY.counter(COUNTER_GET + RIGHT_OBJECT_PROPERTY).inc();
String LEFT_OBJECT_PROPERTY = triple.getLeftObjectProperty();
Vertex leftObject = graph.addVertex();
rightObject.property(LEFT_OBJECT_PROPERTY, triple.getLeftObject());
REGISTRY.counter(COUNTER_GET + LEFT_OBJECT_PROPERTY).inc();
processRelationship(graph, triple);
} catch (Throwable e) {
Throwable rootCause = ExceptionUtils.getRootCause(e);
String rootCauseMessage = null == rootCause ? "" : rootCause.getMessage();
LOG.error("Error processing line {} {}", e.getMessage(), rootCauseMessage, e);
long end = System.currentTimeMillis();
long time = end -
REGISTRY.timer(TIMER_CREATE + RIGHT_OBJECT_PROPERTY).update(time, TimeUnit.MILLISECONDS);
private static void processRelationship(JanusGraph graph, Triple triple) {
Vertex left = get(graph, triple.getLeftObjectProperty(), triple.getLeftObject());
if (null == left) {
REGISTRY.counter("error.missingLeftObject." + triple.getLeftObject()).inc();
left = graph.addVertex();
left.property(triple.getLeftObjectProperty(), triple.getLeftObject());
Vertex right = get(graph, triple.getRightObjectProperty(), triple.getRightObject());
if (null == right) {
REGISTRY.counter("error.missingRightObject." + triple.getRightObject()).inc();
right = graph.addVertex();
right.property(triple.getRightObjectProperty(), triple.getRightObject());
left.addEdge(triple.getRelationship(), right);
private static Vertex get(final JanusGraph graph, final String key, final String value) {
final GraphTraversalSource g = graph.traversal();
final Iterator&Vertex& it = g.V().has(key, value);
return it.hasNext() ? it.next() :
The exceptions above show that all the errors come from lines 47, 55 or 59 of that class:
JanusGraphManagement mgmt = graph.openManagement();
if (mgmt.getGraphIndex(triple.getRightObjectProperty()) == null) {
[47] final PropertyKey hotelKey = mgmt.makePropertyKey(triple.getRightObjectProperty()).dataType(String.class).make();
mgmt.buildIndex(triple.getRightObjectProperty(), Vertex.class).addKey(hotelKey).unique().buildCompositeIndex();
if (mgmt.getEdgeLabel(triple.getRelationship()) == null) {
mgmt.makeEdgeLabel(triple.getRelationship()).multiplicity(Multiplicity.MANY2ONE).make();
[55] if (mgmt.getGraphIndex(triple.getLeftObjectProperty()) == null) {
final PropertyKey brandTypeKey = mgmt.makePropertyKey(triple.getLeftObjectProperty()).dataType(String.class).make();
mgmt.buildIndex(triple.getLeftObjectProperty(), Vertex.class).addKey(brandTypeKey).unique().buildCompositeIndex();
[59] mit();
Can anyone help identify what I'm doing wrong in this class? Whatever I'm doing is locking up tables and creating schema problems.
First, your original run() method is executed for each line in the text file. If the , the vertices and edges you created in the previous iteration get blown away.
Second, on the next line you tried to , even though you only need to set the schema once for the graph right after you instantiate it. This was the source of your Schema exceptions, as index names must be unique (and do not necessarily need to be named after the properties they are indexing).
Third, on lines 78 and 83, you assumed you were
of the text file you processed. The uniqueness constraint applies there just as much as it did on , where you respected the uniqueness constraint while processing the relationships.
Finally, for a graph with order ~100 vertices and ~100 edges, it is a bit over the top to setup batching and an executor. I submitted
to approach the data load at hand.
2,75811329
As stated in the , you should rollback any active transaction before attempting to create an index.
Have you tried that?
2,40021028
Your Answer
Sign up or
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Post as a guest
By posting your answer, you agree to the
Not the answer you're looking for?
Browse other questions tagged
The week's top questions and answers
Important community announcements
Questions that need answers
By subscribing, you agree to the
Stack Overflow works best with JavaScript enableddynamodb-janusgraph-storage-backend connect remotely from Java - Stack Overflow
Join Stack Overflow to learn, share knowledge, and build your career.
or sign in with
I have dynamodb-janusgraph-storage-backend deployed on AWS and I am trying to figure out how to connect to the gremlin server from Java.
I have sbt dependency of dynamodb-janusgraph-storage-backend in my project but I don't want to use the gremlin server running as part of my java application. I need it to run independently and connect java application to that.
I looked into multiple options like using Cluster (gremlin-java) and withRemote (gremlin-driver) but both have limitations. I would like to use the Java Gremlin API which I can't if I use Cluster. Using the withRemote method, I cannot figure out how to initialize the graph instance.
The examples on gremlin docs shows EmptyGraph.instance() which I cannot use if I want to use JanusGraph API.
I need this part to work with Janusgraph:
Cluster cluster = Cluster.open("conf/remote-objects.yaml"); // this has hosts and ports to gremlin server running in AWS
graph = EmptyGraph.instance();
graph.traversal().withRemote(DriverRemoteConnection.using(cluster))
I need the graph object to be JanusGraph type so I can use openManagement() and other methods. Also, using the high-level Graph type, I cannot add new vertexes. I need to be able to do create, get, update from my java code.
Currently it is not possible to invoke the JanusGraph Schema APIs via the remote driver. If you don't want to open up a JanusGraph instance from your application, you'd have to build the schema as a String and use
to send it to the Gremlin Server. You can still use the
to build and query the graph.
Cluster cluster = Cluster.open("conf/remote-objects.yaml");
// use client to create the schema
Client client = cluster.connect();
String schema = "mgmt=graph.openManagement();";
schema += "mgmt.makeVertexLabel(\"person\").make();";
schema += "mgmt.makeVertexLabel(\"person\");";
schema + "mgmt.makePropertyKey(\"name\").dataType(String.class).make();"
schema += "mit(); true";
CompletableFuture&List&Result&& results = client.submit(schema).all();
// use traversals only to interact with the graph
Graph graph = EmptyGraph.instance();
GraphTraversalSource g = graph.traversal().withRemote(DriverRemoteConnection.using(cluster));
Vertex v = g.addV("person").property("name", "pepe").next();
List&Vertex& vlist = g.V().toList();
Your Answer
Sign up or
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Post as a guest
By posting your answer, you agree to the
Not the answer you're looking for?
Browse other questions tagged
Stack Overflow works best with JavaScript enabled16:34 提问
JAVA 如何连接JanusGraph
例如SSM框架一样，通过配置文件连接到数据库。那么如何连接到JanusGraph呢，
用janusgraph-hbase这个包
按赞数排序
看看这位兄弟写的是不是能帮到你
其他相关推荐java - WebSocketGremlinRequestEncoder must produce at least one message - janusgraph-dynamodb using withRemote &sideEffect& doesn't work - Stack Overflow
Join Stack Overflow to learn, share knowledge, and build your career.
or sign in with
When I use gremlin-server connection using gremlin-driver in Java, I am not able to use "sideEffect" of GraphTraversal.
graph = EmptyGraph.instance()
cluster = Cluster.open("conf/remote-objects.yaml");
graphTraversalSource = graph.traversal().withRemote(DriverRemoteConnection.using(cluster));
My query that uses sideEffect looks like:
AtomicLong level1 = new AtomicLong(0);
graphTraversalSource.V().hasLabel("user")
.has("uuid", "1234")
.sideEffect(it -& it.get().property("level", level1.getAndIncrement())).emit().repeat(in())
.until(loops().is(5)).valueMap("uuid", "name", "level");
This query used to work when I was using janusgraph-dynamodb-storage-backend as dependency and running gremlin server within Java application and connecting to dyamodb. When i switched to using remote connection to gremlin server running in EC2, i started getting below error message:
java.pletionException: io.netty.handler.codec.EncoderException: WebSocketGremlinRequestEncoder must produce at least one message., took 3.895 sec
If I remove the sideEffect part from the above query, it works fine. I really need to add a custom property during traversal and include that in results without saving it in the database.
You have a few problems. The first problem is that you are trying to remote a lambda in the sideEffect() Lambdas can't be serialized to Gremlin bytecode - at least not in the form you've provided. However, you can do this:
gremlin& cluster = Cluster.open("conf/remote-objects.yaml")
==&localhost/127.0.0.1:8182
gremlin& g = graph.traversal().withRemote(DriverRemoteConnection.using(cluster))
==&graphtraversalsource[emptygraph[empty], standard]
gremlin& g.addV('person').as('p').addE('link').to('p')
==&e[1][0-link-&0]
gremlin& g.V().sideEffect(Lambda.function("it.get().property('level',1)")).valueMap()
==&[level:[1]]
Note that I had to import import org.apache.tinkerpop.gremlin.util.function.* to the console to make that last line work there - That will be fixed for 3.2.7/3.3.0.
So, you could pass your lambda that way, but:
I don't think your traversal will work as before because you are referencing a variable local to the client with level1 - the server is not going to know anything about that.
TinkerPop generally recommends that you avoid lambdas.
I don't quite follow what your Gremlin is doing to provide a suggestion on how to resolve this. You do give this hint:
I really need to add a custom property during traversal and include that in results without saving it in the database.
...but the Gremlin above does write the value of level1 to the database so I'm not sure of what you are after.
15.6k12255
Your Answer
Sign up or
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Post as a guest
By posting your answer, you agree to the
Not the answer you're looking for?
Browse other questions tagged
Stack Overflow works best with JavaScript enabled