riak-cs sync buckets from S3 to Riak-cs
aayllon at qdqmedia.com
Tue Nov 17 04:50:23 EST 2015
Thanks a lot for your help!
2015-11-17 3:50 GMT+01:00 Shunichi Shinohara <shino at basho.com>:
> Hi Alberto,
> I didn't look into boto implementation, but I suspect that COPY Object API
> does NOT work between different S3-like systems.
> The actual interface definition of the API is  and source bucket/key
> is just a string in the x-amz-copy-source header. The request went into the
> system that includes rk02.ejemplo.com in your example, but it did not know
> anything about source bucket/key because it does not have the bucket/key.
> Object contents should be transferred in some way, e.g. GET Object from
> source and PUT Object (or Multipart Upload for large objects) to target
>  http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
> 2015-11-16 21:28 GMT+09:00 Alberto Ayllon <aayllon at qdqmedia.com>:
> > Hello.
> > Thanks for your help Dmitri.
> > Perhaps this is not the correct place to ask this, but maybe someone had
> > have the same problem.
> > I build a test environment with two nodes of RIAK-CS, not in cluster, and
> > I'm trying to move objects from one to another, to a bucket with the same
> > name (testbucket).
> > I'm using BOTO, and the copy_key method, but it fails, I guess that
> > is in the HEAD call, this call checks if the key exists in the
> > bucket before make the PUT, that copy the key.
> > Here is the code I'm using.
> > from boto.s3.key import Key
> > from boto.s3.connection import S3Connection
> > from boto.s3.connection import OrdinaryCallingFormat
> > apikey04='GFR3O0HFPXQ-BWSXEMAG'
> > secretkey04='eIiigR4Rov2O2kxuSHNW7WPoJE2KmrtMpzzqlg=='
> > apikey02='J0TT_C9MJPWPGHW-KEWY'
> > secretkey02='xcLOt3ANqyNJ0kAjP8Mxx68qr7kgyXG3eqJuMA=='
> > cf=OrdinaryCallingFormat()
> > is_secure=False,host='rk04.ejemplo.com',port=8080,calling_format=cf)
> > is_secure=False,host='rk02.ejemplo.com',port=8080,calling_format=cf)
> > bucket04=conn04.get_bucket('testbucket')
> > bucket02=conn02.get_bucket('testbucket')
> > rs04 = bucket04.list()
> > for k in rs04:
> > print k.name
> > bucket02.copy_key(k.key, bucket04, k.key)
> > When this script is executed it returns:
> > Traceback (most recent call last):
> > File "s3_connect_2.py", line 38, in <module>
> > bucket02.copy_key(k.key, bucket04, k.key)
> > File
> > line 888, in copy_key
> > response.reason, body)
> > boto.exception.S3ResponseError: S3ResponseError: 404 Not Found
> > <?xml version="1.0"
> > encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key
> > does not exist.</Message><Resource>/<Bucket: testbucket>$
> > The idea is copy all keys in testbucket from rk04.ejemplo.com, to
> > in test02.ejemplo.com, maybe someone can help me.
> > Thanks a lot.
> > 2015-11-13 17:06 GMT+01:00 Dmitri Zagidulin <dzagidulin at basho.com>:
> >> Hi Alberto,
> >> From what I understand, the state of the art in terms of migration of
> >> objects from Amazon S3 to Riak CS is -- writing migration scripts.
> >> Either as shell scripts (using s3cmd), or language-specific libraries
> >> boto (or even just the S3 SDKs).
> >> And the scripts would consist of:
> >> 1) get a list of the buckets you want to migrate
> >> 2) List the keys in those buckets
> >> 3) Migrate each object from AWS to CS.
> >> You're right that mounting buckets as filesystems is a (distant)
> >> possibility, but we have not seen much successful use of those (though
> >> anybody's made that work, let us know).
> >> On Thu, Nov 12, 2015 at 12:40 PM, Alberto Ayllon <aayllon at qdqmedia.com>
> >> wrote:
> >>> Hello.
> >>> I'm new using Riak and Riak-cs, I have installed a Riak-cs cluster
> with 4
> >>> nodes and it works fine,
> >>> Here is my question, the company where I work has some buckets in
> >>> s3, and I would like migrate objects from these buckets to our Riak-cs
> >>> installation, as far as I know I can do it using S3FUSE or S3BACKER,
> >>> mounting buckets as a filesystem, but would like avoid mount it as
> >>> filesystem. I tried it with boto python library, using the copy_key
> >>> but it doesn't work.
> >>> Has anybody try with success synchronize buckets from AS3 to Riak-CS?
> >>> Thanks.
> >>> P:D: Excuse for my English.
> >>> _______________________________________________
> >>> riak-users mailing list
> >>> riak-users at lists.basho.com
> >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users