On Mon, Feb 14, 2011 at 11:46, Cédric Villemain
<cedric.villemain.debian@gmail.com> wrote:
> 2011/2/14 Stefan Kaltenbrunner <Stefan@kaltenbrunner.cc>:
>> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>>> <stefan@kaltenbrunner.cc> wrote:
>>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>>
>>>>> Magnus Hagander<magnus@hagander.net> writes:
>>>>>>
>>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>>> happened - a machine did not come back up after a reboot.
>>>>>> ...
>>>>>> We'll get back to you with more information as soon as we have it.
>>>>>
>>>>> I didn't see any followup to this?
>>>>
>>>> yeah - the hosting company managed to reboot the box for us which brought it
>>>> back to life in the middle of the night (with both magnus and me asleep).
>>>
>>> Indeed. But the good news is that once it came back up, the VM with
>>> the git server started ok :-)
>>>
>>>
>>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>>
>>>> yes it is - however we will need to schedule another maintenance window soon
>>>> to finish the stuff we actually wanted to do.
>>>
>>> So, after some discussion with Stefan, we (well, I guess I) decided we
>>> should just go ahead and declare the maintenance window not closed
>>> yet, and finish off the upgrade right now :-) Given that the majority
>>> of our commits don't happen now, we'll hopefully have it done by the
>>> time the US folks wake up again.
>>>
>>> So, maintenance window again, starting now, and we'll let you know as
>>> soon as we're done. And we're definitely hoping for the machine to
>>> come back up properly this time :-)
>>
>> and it did not... We are trying to figure out what the actual problem
>> here really is because it seems to boot just fine when powercycled just
>> not with a software initiated reboot.
>> We will notify once we have more information...
>>
>
> Does it make sense to get some console link or ipmi set up for those
> crucial parts of the infrastructure ?
This is production servers, of course they are equipped with remove consoles.
However, these consoles are only accessible from the hosting companys
internal company network or VPN, so we cannot access them directly.
It is something we are discussing with them...
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/