RIAK TS installed nodes not connecting

Agtmaal, Joris van joris.vanagtmaal at wartsila.com
Tue Sep 13 10:06:44 EDT 2016


Hi Alex

I had just figured out that i probably was missing something, so i tried installing an earlier version using the Uber jar. This worked fine, so my next step is to get back to the latest version of the Uber jar and i expect indeed that that was my mistake.

Thanks for the help, and your patience with my rookie mistakes.
;-)

Kind Regards,
Joris van Agtmaal
+(0) 6 25 39 39 06

From: Alex Moore [mailto:amoore at basho.com]
Sent: 13 September 2016 15:35
To: Stephen Etheridge <setheridge at basho.com>
Cc: Agtmaal, Joris van <joris.vanagtmaal at wartsila.com>; riak-users at lists.basho.com; Manu Marchal <emarchal at basho.com>
Subject: Re: RIAK TS installed nodes not connecting

Joris,

One thing to check - since you are using a downloaded jar, are you using the Uber jar that contains all the dependencies?
http://search.maven.org/remotecontent?filepath=com/basho/riak/spark-riak-connector_2.10/1.6.0/spark-riak-connector_2.10-1.6.0-uber.jar

Thanks,
Alex

On Tue, Sep 13, 2016 at 8:44 AM, Stephen Etheridge <setheridge at basho.com<mailto:setheridge at basho.com>> wrote:
Hi Joris,

I have looked at the tutorial you have been following but I confess I am confused.  In the example you are following I do not see where the spark and sql contexts are created.  I use PySpark through the Jupyter notebook and I have to specify a path to the connector on invoking the jupyter notebook. Is it possible for you to share all your code (and how you are invoking zeppelin) with me so I can trace everything through?

regards
Stephen

On Mon, Sep 12, 2016 at 3:27 PM, Agtmaal, Joris van <joris.vanagtmaal at wartsila.com<mailto:joris.vanagtmaal at wartsila.com>> wrote:
Hi

I’m new to Riak and followed the installation instructions to get it working on an AWS cluster (3 nodes).

So far ive been able to use Riak in pyspark (zeppelin) to create/read/write tables, but i would like to use the dataframes directly from spark, using the Spark-Riak Connector.
When following the example found here: http://docs.basho.com/riak/ts/1.4.0/add-ons/spark-riak-connector/quick-start/#python
But i run into trouble on this last part:

host= my_ip_adress_of_riak_node
pb_port = '8087'
hostAndPort = ":".join([host, pb_port])
client = riak.RiakClient(host=host, pb_port=pb_port)

df.write \
    .format('org.apache.spark.sql.riak') \
    .option('spark.riak.connection.host', hostAndPort) \
    .mode('Append') \
    .save('test')

Important to note that i’m using a local download of the Jar file that is loaded into the pyspark interpreter in zeppeling through:
%dep
z.reset()
z.load("/home/hadoop/spark-riak-connector_2.10-1.6.0.jar")

Here is the error message i get back:
Py4JJavaError: An error occurred while calling o569.save. : java.lang.NoClassDefFoundError: com/basho/riak/client/core/util/HostAndPort at com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:76) at com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:89) at org.apache.spark.sql.riak.RiakRelation$.apply(RiakRelation.scala:115) at org.apache.spark.sql.riak.DefaultSource.createRelation(DefaultSource.scala:51) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:745) (<class 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred while calling o569.save.\n', JavaObject id=o570), <traceback object at 0x7f7021bb0200>)

Hope somebody can help out.
thanks, joris

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



--
{ "name" : "Stephen Etheridge",
   "title" : "Solution Architect, EMEA",
   "Organisation" : "Basho Technologies, Inc",
   "Telephone" : "07814 406662",
   "email" : "mailto:setheridge at basho.com<mailto:setheridge at basho.com>",
   "github" : "http://github.com/datalemming",
   "twitter" : "@datalemming"}


_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160913/fe9a8118/attachment-0002.html>


More information about the riak-users mailing list