Upgrade path 1.4.0

Toby Corkindale toby.corkindale at strategicdata.com.au
Wed Jul 10 21:55:33 EDT 2013


On 10/07/13 23:14, Jared Morrow wrote:
> We try to post the packages to all the right places before we make
> the announcement  so I'd highly recommend you don't just auto-update
> Riak packages when they hit apt/yum.

Ah, actually we don't automatically update them, but someone was 
performing a pass of unrelated package updates and accidentally pulled 
in the riak ones too.

> I'm glad you eventually got all your nodes upgraded.

Thanks. It wasn't painful in the end; just a bit of a shock at the time 
when the cluster fell over. I'm still not sure why that was, but I 
wasn't the one performing the upgrades that seemed to cause it.

I suspect it was just too many nodes being down at once, possibly due to 
the delayed start-up during the upgrade.

T

> On Wed, Jul 10, 2013 at 5:42 AM, Toby Corkindale
> <toby.corkindale at strategicdata.com.au
> <mailto:toby.corkindale at strategicdata.com.au>> wrote:
>
>     Yeah, I didn't think 1.4.0 was into final release yet either -- yet
>     it came through on the Debian and Ubuntu apt repositories
>     automatically this evening.
>
>     ----- Original Message -----
>     From: "Guido Medina" <guido.medina at temetra.com
>     <mailto:guido.medina at temetra.com>>
>     To: "riak-users" <riak-users at lists.basho.com
>     <mailto:riak-users at lists.basho.com>>
>     Sent: Wednesday, 10 July, 2013 9:39:02 PM
>     Subject: Re: Upgrade path 1.4.0
>
>     Hi Toby,
>
>     I'm sure someone from Basho will answer soon, I just pointed you to the
>     "release notes" direction. I have only overlooked the release notes
>     until we decide to migrate to 1.4.0 when is final (Right now on rc1)
>
>     HTH,
>
>     Guido.
>
>     On 10/07/13 12:29, Toby Corkindale wrote:
>      > Thanks Guido.
>      >
>      > Looks like we've upgraded to 1.4.0 completely now and the cluster
>     is back up.
>      >
>      > I'm not sure of the exact root cause, but what we were seeing was
>     that too many nodes went down for the ring to be healthy, and then
>     when nodes were restarted they waited for the ring to appear for a
>     while, and then exited (or crashed) automatically.
>      >
>      > I *think* this is because after the package was upgraded, the
>     Riak service didn't restart after the upgrade -- and our automated
>     system to catch that and restart them wasn't kicking in quickly
>     enough compared to the rate at which the servers were being upgraded.
>      >
>      > You mention in the release notes that the init scripts have been
>     adjusted - maybe that has something to do with it?
>      >
>      > Cheers,
>      > Toby
>      >
>      > ----- Original Message -----
>      > From: "Guido Medina" <guido.medina at temetra.com
>     <mailto:guido.medina at temetra.com>>
>      > To: "riak-users" <riak-users at lists.basho.com
>     <mailto:riak-users at lists.basho.com>>
>      > Sent: Wednesday, 10 July, 2013 9:05:16 PM
>      > Subject: Re: Upgrade path 1.4.0
>      >
>      > Release notes:
>     https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md
>      >
>      > Maybe related to this?
>      > Known Issues
>      > leveldb 1.3 to 1.4 conversion
>      >
>      >
>      > The first execution of 1.4.0 leveldb using a 1.3.x or 1.2.x
>     dataset will initiate an automatic conversion that could pause the
>     startup of each node by 3 to 7 minutes. The leveldb data in "level
>     #1" is being adjusted such that "level #1" can operate as an
>     overlapped data level instead of as a sorted data level. The
>     conversion is simply the reduction of the number of files in "level
>     #1" to being less than eight via normal compaction of data from
>     "level #1" into "level #2". This is a one time conversion.
>      > HTH,
>      >
>      > Guido.
>      >
>      > On 10/07/13 11:57, Toby Corkindale wrote:
>      >
>      >
>      >
>      > Hi,
>      > some of our nodes upgraded to Riak 1.4.0, and are now refusing to
>     start and join the cluster.
>      > Is there documentation on the upgrade path from 1.3.2 to 1.4.0?
>     It appears we have accidentally begun this journey, and I don't know
>     if it's easier to go back or forwards now..
>      >
>      > PS. It would have been pleasant to receive a warning about the
>     new version on the mailing list!
>      >
>      > Thanks,
>      > Toby





More information about the riak-users mailing list