RIAK TS installed nodes not connecting

Agtmaal, Joris van joris.vanagtmaal at wartsila.com
Mon Sep 12 10:27:09 EDT 2016


Hi

I'm new to Riak and followed the installation instructions to get it working on an AWS cluster (3 nodes).

So far ive been able to use Riak in pyspark (zeppelin) to create/read/write tables, but i would like to use the dataframes directly from spark, using the Spark-Riak Connector.
When following the example found here: http://docs.basho.com/riak/ts/1.4.0/add-ons/spark-riak-connector/quick-start/#python
But i run into trouble on this last part:

host= my_ip_adress_of_riak_node
pb_port = '8087'
hostAndPort = ":".join([host, pb_port])
client = riak.RiakClient(host=host, pb_port=pb_port)

df.write \
    .format('org.apache.spark.sql.riak') \
    .option('spark.riak.connection.host', hostAndPort) \
    .mode('Append') \
    .save('test')

Important to note that i'm using a local download of the Jar file that is loaded into the pyspark interpreter in zeppeling through:
%dep
z.reset()
z.load("/home/hadoop/spark-riak-connector_2.10-1.6.0.jar")

Here is the error message i get back:
Py4JJavaError: An error occurred while calling o569.save. : java.lang.NoClassDefFoundError: com/basho/riak/client/core/util/HostAndPort at com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:76) at com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:89) at org.apache.spark.sql.riak.RiakRelation$.apply(RiakRelation.scala:115) at org.apache.spark.sql.riak.DefaultSource.createRelation(DefaultSource.scala:51) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:745) (<class 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred while calling o569.save.\n', JavaObject id=o570), <traceback object at 0x7f7021bb0200>)

Hope somebody can help out.
thanks, joris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160912/c234e74f/attachment-0002.html>


More information about the riak-users mailing list