Re: Server unreliability - Mailing list pgsql-advocacy

Bruce,

> It is my opinion that we have to make major changes in the way we
> provide hosting for our servers.  There are several problems:

You're not the only one to have noticed this, believe it or not.  ;-)

> The location of our servers in Panama is a problem.  They are too far
> for any PostgreSQL maintainers to access.  Changing hardware or
> diagnosing problems has been too hard.  I have had like 2 days of
> downtime on my home machine in the past 12 years.  We have had more than
> 2 days of downtime in the past 6 months.  My wife would not accept such
> a reliability level.

To be fair, PostgreSQL uses an enormous bandwidth and traffic that gives our
systems very little tolerance for anominalies.   Not that having the stuff in
Panama doesn't slow down fixes, but hosting postgresql.org is a bit more
demanding than hosting, say, your or my personal web site.

> The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> server crash or power failure.  Again, I would never accept such
> problems on my home server so it is hard to fathom how a project with
> thousands of users can accept that.  Either we need to find a fix, stop
> using jails, or get another operating system, but continuing to use a
> setup with a known problem is just asking for trouble.

I agree that when we have more servers dedicated to PostgreSQL, we should use
a different approach than the jails.  Marc, what are the advantages of the
jails on a dedicated server, since we are so familiar with their drawbacks?

> We have been talking about a new web page layout for years at this
> point.  I almost don't care if they just put a dancing bear up on the
> web site.  Let's do something!

I'd suggest finding some money to sponsor somebody full-time for a month to
accomplish the migration to the new web structure, which is more-or-less
finished.   That's what's holding us up right now AFAIK.

> The archives situation is a continual problem.  Again, maybe a dancing
> bear can help.  :-)

But being solved.   The archives have been copied to CommandPrompt, and the
various archive search tools have been distributed.   Once we can change the
web interface, there should always be one tool and/or version available
regardless of individual server/hosting failures.

Further, SF.net has offered to host our mailing lists, which is tempting
because of their direct mailing-list-to-database tool, as well as their more
distributed hosting structure which prevents DNS-based failures.   I expect
to find out the terms for this next week, and then discuss it on WWW.

> Basically, with no money and no one offering servers, I don't see a good
> solution to any of these problems, but I think we need to recognize
> these are problems and that we will continue to suffer until they are
> addressed.
>
> Are there any proposals, no matter how radical, to correct these?

The big obstacle in moving anything is bandwidth.   While any number of
companies will offer to host stuff for us, we can't match the amount of
bandwidth we're currently using in Panama -- and hosting donors won't support
anywhere near that level (which runs to scores of GB per month).  So while we
can move individual, less-crucial components, the mailing lists and the main
WWW require either Hub.org's generous bandwitdh or other hosting that we
"own".

I have actually been working on raising money for additional servers.   The
obstacles which have made this a year-long process are more complicated than
I want to go over right now, but there are people thinking about it.

--
Josh Berkus
Aglio Database Solutions
San Francisco