Luwak PUT Content-Range

John Axel Eriksson john at insane.se
Wed Nov 30 03:51:54 EST 2011


Well, I don't know what I'm doing wrong here but I'm mainly using ruby.
Excon (which Ripple uses) doesn't seem to have a way to pass an IO to a put
request (am I missing something?).

I've also tried NetHTTP in ruby which is actually supposed to be able to do
that and I've tried Javas HttpURLConnection in JRuby (I'm really not a Java
expert). None of them works unfortunately:

# I tried this

require 'net/http'

## this doesn't work and doesn't store all the data
path= "/path/to/file/here" ## 88 MB file
key = "large_file"
uri = URI("http://127.0.0.1:8098")
req = Net::HTTP::Put.new("/luwak/#{key}")
req.body_stream = File.open(path)
req.content_type = "video/mp4"
req.add_field 'Transfer-Encoding', 'chunked'
res = Net::HTTP.start(uri.host, uri.port) do |http|
  http.request(req)
end

# And this - in this case riak started thrashing like crazy until
# I just killed it (it never stored more than  1 MB of data though)
path= "/path/to/file/here" ## 88 MB file
key = "large_file"
uri = URI("http://127.0.0.1:8098")
req = Net::HTTP::Put.new("/luwak/#{key}")
req.body_stream = File.open(path)
req.content_type = "video/mp4"
req.content_length = File.size(path)
req.add_field 'Transfer-Encoding', 'chunked'
res = Net::HTTP.start(uri.host, uri.port) do |http|
  http.request(req)
end

# And in JRuby I tried Javas HttpURLConnection

require 'java'

include_class "java.net.URL"
include_class "java.io.RandomAccessFile"
include_class "java.io.FileInputStream"
include_class "java.net.HttpURLConnection"

CHUNK_LENGTH=1048576

u = URL.new('http://192.168.1.142:8098/luwak/jruby_large_file_upload')
c = u.open_connection
c.request_method = 'PUT'
c.fixed_length_streaming_mode=File.size(FILE)
# c.chunked_streaming_mode = CHUNK_LENGTH  ## also tried this
c.do_input = true
c.do_output = true
output_stream = c.output_stream
error_stream = c.error_stream
random_access_file = RandomAccessFile.new(FILE, 'r')
bytes_read = 0
buffer = Java::byte[CHUNK_LENGTH].new
begin
  while(bytes_read >= 0) do
    bytes_read = random_access_file.read(buffer)
    puts "bytes_read: #{bytes_read}"
    if bytes_read>=0
      ## the second write here gets an Exception - I think it's a 400 from
riak
      puts "writing..."
      output_stream.write(buffer,0,bytes_read)
      sleep 1
    end
  end
rescue Exception => e
  puts "Exception: #{e.message}"
end
puts c.response_code rescue nil
puts c.response_message rescue nil
random_access_file.close
output_stream.close
c.disconnect



On Tue, Nov 29, 2011 at 10:53 PM, Greg Stein <gstein at gmail.com> wrote:

>
> On Nov 29, 2011 5:08 AM, "John Axel Eriksson" <john at insane.se> wrote:
> >
> > Is it possible to incrementally add to a file in Luwak using PUT and the
> Content-Range header. I just assumed that it was but I can't seem to
> > get the expected results, it just overwrites whatever the key contents
> were before. The reason I want to do this is because we have some pretty
> > large files I don't want to load fully into memory before PUTing them.
>
> Hmm? You can read from a file and write to the http socket. There is no
> reason or need to load the entire contents into memory.
>
> I don't know what client you're using, but I do know the Python client is
> broken in this regard. It erroneously loads the full content into memory.
> But there is nothing from the Riak server that demands such an approach.
>
> Cheers,
> -g
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111130/68a48cfb/attachment.html>


More information about the riak-users mailing list