RIAK TS installed nodes not connecting

Alex Moore amoore at basho.com
Tue Sep 13 09:35:24 EDT 2016


Joris,

One thing to check - since you are using a downloaded jar, are you using
the Uber jar that contains all the dependencies?
http://search.maven.org/remotecontent?filepath=com/basho/riak/spark-riak-connector_2.10/1.6.0/spark-riak-connector_2.10-1.6.0-uber.jar

Thanks,
Alex

On Tue, Sep 13, 2016 at 8:44 AM, Stephen Etheridge <setheridge at basho.com>
wrote:

> Hi Joris,
>
> I have looked at the tutorial you have been following but I confess I am
> confused.  In the example you are following I do not see where the spark
> and sql contexts are created.  I use PySpark through the Jupyter notebook
> and I have to specify a path to the connector on invoking the jupyter
> notebook. Is it possible for you to share all your code (and how you are
> invoking zeppelin) with me so I can trace everything through?
>
> regards
> Stephen
>
> On Mon, Sep 12, 2016 at 3:27 PM, Agtmaal, Joris van <
> joris.vanagtmaal at wartsila.com> wrote:
>
>> Hi
>>
>>
>>
>> I’m new to Riak and followed the installation instructions to get it
>> working on an AWS cluster (3 nodes).
>>
>>
>>
>> So far ive been able to use Riak in pyspark (zeppelin) to
>> create/read/write tables, but i would like to use the dataframes directly
>> from spark, using the Spark-Riak Connector.
>>
>> When following the example found here: http://docs.basho.com/riak/ts/
>> 1.4.0/add-ons/spark-riak-connector/quick-start/#python
>>
>> But i run into trouble on this last part:
>>
>>
>>
>> host= my_ip_adress_of_riak_node
>>
>> pb_port = '8087'
>>
>> hostAndPort = ":".join([host, pb_port])
>>
>> client = riak.RiakClient(host=host, pb_port=pb_port)
>>
>>
>>
>> df.write \
>>
>>     .format('org.apache.spark.sql.riak') \
>>
>>     .option('spark.riak.connection.host', hostAndPort) \
>>
>>     .mode('Append') \
>>
>>     .save('test')
>>
>>
>>
>> Important to note that i’m using a local download of the Jar file that is
>> loaded into the pyspark interpreter in zeppeling through:
>>
>> %dep
>>
>> z.reset()
>>
>> z.load("/home/hadoop/spark-riak-connector_2.10-1.6.0.jar")
>>
>>
>>
>> Here is the error message i get back:
>>
>> Py4JJavaError: An error occurred while calling o569.save. :
>> java.lang.NoClassDefFoundError: com/basho/riak/client/core/util/HostAndPort
>> at com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:76)
>> at com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:89)
>> at org.apache.spark.sql.riak.RiakRelation$.apply(RiakRelation.scala:115)
>> at org.apache.spark.sql.riak.DefaultSource.createRelation(DefaultSource.scala:51)
>> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc
>> e$.apply(ResolvedDataSource.scala:222) at org.apache.spark.sql.DataFrame
>> Writer.save(DataFrameWriter.scala:148) at org.apache.spark.sql.DataFrame
>> Writer.save(DataFrameWriter.scala:139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606) at
>> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at
>> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at
>> py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.
>> invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79)
>> at py4j.GatewayConnection.run(GatewayConnection.java:209) at
>> java.lang.Thread.run(Thread.java:745) (<class
>> 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred while
>> calling o569.save.\n', JavaObject id=o570), <traceback object at
>> 0x7f7021bb0200>)
>>
>>
>>
>> Hope somebody can help out.
>>
>> thanks, joris
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> { "name" : "Stephen Etheridge",
>    "title" : "Solution Architect, EMEA",
>    "Organisation" : "Basho Technologies, Inc",
>    "Telephone" : "07814 406662",
>    "email" : "mailto:setheridge at basho.com",
>    "github" : "http://github.com/datalemming",
>    "twitter" : "@datalemming"}
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160913/2601f9a5/attachment-0002.html>


More information about the riak-users mailing list