riak-cs sync buckets from S3 to Riak-cs

Shunichi Shinohara shino at basho.com
Mon Nov 16 21:50:10 EST 2015


Hi Alberto,

I didn't look into boto implementation, but I suspect that COPY Object API
does NOT work between different S3-like systems.
The actual interface definition of the API is [1] and source bucket/key
is just a string in the x-amz-copy-source header. The request went into the
system that includes rk02.ejemplo.com in your example, but it did not know
anything about source bucket/key because it does not have the bucket/key.
Object contents should be transferred in some way, e.g. GET Object from
source and PUT Object (or Multipart Upload for large objects) to target
system.

[1] http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html

Thanks,
Shino

2015-11-16 21:28 GMT+09:00 Alberto Ayllon <aayllon at qdqmedia.com>:
> Hello.
>
> Thanks for your help Dmitri.
>
> Perhaps this is not the correct place to ask this, but maybe someone had
> have the same problem.
>
> I build a test environment with two nodes of RIAK-CS, not in cluster, and
> I'm trying to move objects from one to another, to a bucket with the same
> name (testbucket).
> I'm using BOTO, and the copy_key method, but it fails, I guess that problem
> is in the HEAD call, this call checks if the key exists in the destination
> bucket before make the PUT, that copy the key.
>
> Here is the code I'm using.
>
> from boto.s3.key import Key
> from boto.s3.connection import S3Connection
> from boto.s3.connection import OrdinaryCallingFormat
>
> apikey04='GFR3O0HFPXQ-BWSXEMAG'
> secretkey04='eIiigR4Rov2O2kxuSHNW7WPoJE2KmrtMpzzqlg=='
>
> apikey02='J0TT_C9MJPWPGHW-KEWY'
> secretkey02='xcLOt3ANqyNJ0kAjP8Mxx68qr7kgyXG3eqJuMA=='
> cf=OrdinaryCallingFormat()
>
> conn04=S3Connection(aws_access_key_id=apikey04,aws_secret_access_key=secretkey04,
>
> is_secure=False,host='rk04.ejemplo.com',port=8080,calling_format=cf)
>
> conn02=S3Connection(aws_access_key_id=apikey02,aws_secret_access_key=secretkey02,
>
> is_secure=False,host='rk02.ejemplo.com',port=8080,calling_format=cf)
>
> bucket04=conn04.get_bucket('testbucket')
> bucket02=conn02.get_bucket('testbucket')
>
> rs04 = bucket04.list()
>
> for k in rs04:
>     print k.name
>     bucket02.copy_key(k.key, bucket04, k.key)
>
>
> When this script is executed it returns:
>
> Traceback (most recent call last):
>   File "s3_connect_2.py", line 38, in <module>
>     bucket02.copy_key(k.key, bucket04, k.key)
>   File
> "/home/alberto/.virtualenvs/boto/local/lib/python2.7/site-packages/boto/s3/bucket.py",
> line 888, in copy_key
>     response.reason, body)
> boto.exception.S3ResponseError: S3ResponseError: 404 Not Found
> <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key
> does not exist.</Message><Resource>/<Bucket: testbucket>$
>
>
>
> The idea is copy all keys in testbucket from rk04.ejemplo.com, to testbucket
> in test02.ejemplo.com, maybe someone can help me.
>
>
> Thanks a lot.
>
>
> 2015-11-13 17:06 GMT+01:00 Dmitri Zagidulin <dzagidulin at basho.com>:
>>
>> Hi Alberto,
>>
>> From what I understand, the state of the art in terms of migration of
>> objects from Amazon S3 to Riak CS is -- writing migration scripts.
>> Either as shell scripts (using s3cmd), or language-specific libraries like
>> boto (or even just the S3 SDKs).
>> And the scripts would consist of:
>> 1) get a list of the buckets you want to migrate
>> 2) List the keys in those buckets
>> 3) Migrate each object from AWS to CS.
>>
>> You're right that mounting buckets as filesystems is a (distant)
>> possibility, but we have not seen much successful use of those (though if
>> anybody's made that work, let us know).
>>
>>
>>
>> On Thu, Nov 12, 2015 at 12:40 PM, Alberto Ayllon <aayllon at qdqmedia.com>
>> wrote:
>>>
>>> Hello.
>>>
>>> I'm new using Riak and Riak-cs, I have installed a Riak-cs cluster with 4
>>> nodes and it works fine,
>>>
>>> Here is my question,  the company where I work has some buckets in Amazon
>>> s3, and I would like migrate objects from these buckets to our Riak-cs
>>> installation, as far as I know I can do it using S3FUSE or S3BACKER,
>>> mounting buckets as a filesystem, but would like avoid mount it as
>>> filesystem. I tried it with boto python library, using the copy_key method,
>>> but it doesn't work.
>>>
>>> Has anybody try with success synchronize buckets from AS3 to Riak-CS?
>>>
>>> Thanks.
>>>
>>> P:D: Excuse for my English.
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list