loading erlang terms

Seth Falcon seth at userprimary.net
Wed Oct 27 00:58:57 EDT 2010


On Tue, Oct 26, 2010 at 9:37 AM, Charles Blair <chas at uchicago.edu> wrote:
> Btw, the documentation has this:
>
> encode_term(Object, Term) ->
>    riakc_obj:update_value(Object, term_to_binary(Term, [compressed]),
>                           <<"application/x-erlang-term">>).
>

> This gave me the idea that perhaps by encoding the content type as
> application/x-erlang-term I could get things to work as I envisioned
> they might, but I couldn't.

I think the content-type is used to help clients treat data
appropriately.  I don't think Riak does any auto-magic transformations
itself.

Storing the result of term_to_binary makes more sense to me; you could
still have a simple escript that loads a file on disk into Riak using
the HTTP interface if that keeps things convenient.

If storing the text content of a file containing Erlang terms (as you
would use with file:consult/1) is what you want, then you can do that
too.  I think you almost had a working solution, but got tripped up by
erl_parse:parse_term expecting tokens for a single term, not many
terms.  Below is an example that gets around this by breaking up the
list of tokens returned by erl_scan:string by breaking on term
boundaries ({dot, _}).  A regex using the re module would be
expedient, but error-prone because "." can appear in a number of ways
in which it does not represent a term terminator.

-module(ep).

-compile([export_all]).

test(File) ->
    {ok, Bin} = file:read_file(File),
    {ok, Tokens, _} = erl_scan:string(binary_to_list(Bin)),
    lists:map(fun(T) ->
                      {ok, Term} = erl_parse:parse_term(T),
                      Term
              end, split_by_terms(Tokens)).

pop_term(T) ->
    pop_term(T, []).

pop_term([T | Rest], Acc) ->
    case T of
        {dot, _} ->
            {lists:reverse([T|Acc]), Rest};
        _Else ->
            pop_term(Rest, [T|Acc])
    end;
pop_term([], Acc) ->
    {lists:reverse(Acc), []}.

split_by_terms(Tokens) ->
    split_by_terms(Tokens, []).

split_by_terms(Tokens = [_H|_T], Acc) ->
    {TermTokens, Rest} = pop_term(Tokens),
    split_by_terms(Rest, [TermTokens|Acc]);
split_by_terms([], Acc) ->
    lists:reverse(Acc).


-- 
Seth Falcon | @sfalcon | http://userprimary.net/




More information about the riak-users mailing list