Thread: RFC: PostgreSQL Add-On Network
Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. [plan]: http://wiki.postgresql.org/wiki/PGAN [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 [CPAN]: http://cpan.org/ [JSAN]: http://www.openjsan.org [JFDI]: http://acronyms.thefreedictionary.com/JFDI Best, David
On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler <david@kineticode.com> wrote: > Hackers, > > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days seems incredibly short sighted, and liable to lead to an endless stream of 'why doesn't this work' questions. It also does the module authors no favours, by excluding 50%+ of their potential audience, and frankly, isn't the way this project generally works. We have discussed this sort of facility at previous developer meetings, and as I recall came to the conclusion that we need to have the ability to distribute pre-built binaries, not just source code as virtually no Windows users are ever going to have a build environment setup. Similarly, neither are Mac users, if they didn't install XCode. We also discussed extension management at the DBMS level, which I believe Dimitri was working on in his spare time. You should look at what he's been doing. Finally, don't write the client in Perl. Again, that's of no use to most Windows users. C will work on all platforms from the outset, with no dependencies required. Of course, the server side doesn't matter. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Thu, 2010-01-07 at 20:36 +0000, Dave Page wrote: > On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler <david@kineticode.com> wrote: > > Hackers, > > > > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. > > Whilst the aim is a noble one, glossing over 'it won't work on > Windows' which is by far our most popular platform these days Although I understand your point, this is debatable to the point of a pointless point :P. The reason it seems like the most popular is that the only way to get the windows version is to download it from .Org/EDB. Linux/FreeBSD already has it in their own repositories. I do believe that we need to support windows, but not everything will. That is just the nature of it. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Thu, Jan 7, 2010 at 21:42, Joshua D. Drake <jd@commandprompt.com> wrote: > On Thu, 2010-01-07 at 20:36 +0000, Dave Page wrote: >> On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler <david@kineticode.com> wrote: >> > Hackers, >> > >> > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPANphilosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQLtools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very gratefulfor feedback and suggestions. >> >> Whilst the aim is a noble one, glossing over 'it won't work on >> Windows' which is by far our most popular platform these days > > Although I understand your point, this is debatable to the point of a > pointless point :P. The reason it seems like the most popular is that > the only way to get the windows version is to download it from .Org/EDB. > Linux/FreeBSD already has it in their own repositories. If you count production servers, it's certainly going to be more Linux/BSD. But if you start counting developer machines, Windows gets ahead quickly. And then start looking at all the point-of-sales systems etc that come with an embedded PostgreSQL - the vast majority of those (unfortunately) run on Windows. It's not the biggest databases. But it's the most. And particularly something like this needs to work well on the developers machines, or things won't get through the door. > I do believe that we need to support windows, but not everything will. > That is just the nature of it. True. But if it's something that's infrastructure, it should work on all major platforms. This certainly includes Windows. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
David E. Wheeler wrote: > Hackers, > > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL > extensions. I've tried to closely follow the [CPAN philosophy][] to > come up with a plan that requires a minimum-work implementation that > builds on the existing PostgreSQL tools and the examples of the [CPAN][] > and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very > grateful for feedback and suggestions. > > [plan]: http://wiki.postgresql.org/wiki/PGAN > [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 > [CPAN]: http://cpan.org/ > [JSAN]: http://www.openjsan.org > [JFDI]: http://acronyms.thefreedictionary.com/JFDI Not enough brackets. ;-) -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Dave, > Whilst the aim is a noble one, glossing over 'it won't work on > Windows' which is by far our most popular platform these days seems > incredibly short sighted, and liable to lead to an endless stream of > 'why doesn't this work' questions. It also does the module authors no > favours, by excluding 50%+ of their potential audience, and frankly, > isn't the way this project generally works. But why is that *this* project's job to solve? It's already the case that contrib modules (or PostgreSQL) are not buildable on Windows in an automated fashion. And that Windows does not ship with developer tools.That's not PGAN's fault. > We have discussed this sort of facility at previous developer > meetings, and as I recall came to the conclusion that we need to have > the ability to distribute pre-built binaries, not just source code as > virtually no Windows users are ever going to have a build environment > setup. Similarly, neither are Mac users, if they didn't install XCode. Binary distributions are a good idea, for v2+. The problem with requiring that is then you're effectively requiring every module author, or a well-funded 3rd party, to have a huge build farm capable of building binaries for a wide variety of platforms and PostgresQL versions. Requiring that from the outset is a good way to ensure that nobody *ever* builds an extension management platform. If someone is available to provide that build platform, then it's a different story, but I don't know of anyone with those resources right now. It's *already* the case that Windows users have no access to the temporal, prefix or pgmemcache projects without Visual C++ and some jiggery-pokery. How does PGAN make them worse off? > We also discussed extension management at the DBMS level, which I > believe Dimitri was working on in his spare time. You should look at > what he's been doing. This is designed to be complimentary to Dimitri's project, if you read the wiki page. > Finally, don't write the client in Perl. Again, that's of no use to > most Windows users. C will work on all platforms from the outset, with > no dependencies required. Of course, the server side doesn't matter. It would make sense to build a C version when the other issues with supporting Windows are solved. At that point, the specification for the client should be well-worn enough to make doing the C version not a halt to further development. Until then, it doesn't matter. What I'm getting from your e-mail, Dave, is "If it doesn't solve all problems for everyone in the world from Day 1, it's not worth doing." It's my experience that the way to get OSS software that works is to build a little bit at a time, each delivery of which is useful to some people and solves some problems. Projects which attempt to paint the whole world never get anywhere. David's proposal is designed to be something which he can get done *this year*, possibly before 8.5 is released, and be built on later. It'll be useful to a substantial number of our users, and will be an improvement on what we have now. --Josh Berkus
On Jan 7, 2010, at 12:36 PM, Dave Page wrote: > Whilst the aim is a noble one, glossing over 'it won't work on > Windows' which is by far our most popular platform these days seems > incredibly short sighted, and liable to lead to an endless stream of > 'why doesn't this work' questions. It also does the module authors no > favours, by excluding 50%+ of their potential audience, and frankly, > isn't the way this project generally works. That's orthogonal to the development of PGAN. The PGAN client will depend on PGXS to do the installing. If the core implementssomething that works better on Windows, the PGAN client will use it. > We have discussed this sort of facility at previous developer > meetings, and as I recall came to the conclusion that we need to have > the ability to distribute pre-built binaries, not just source code as > virtually no Windows users are ever going to have a build environment > setup. Similarly, neither are Mac users, if they didn't install XCode. It's no hardship to install XCode. People do it for Perl/CPAN installs all the time. > We also discussed extension management at the DBMS level, which I > believe Dimitri was working on in his spare time. You should look at > what he's been doing. He and I have discussed it. As I wrote in the proposal, when extensions get in, the PGAN client will be updated as appropriateto support them. > Finally, don't write the client in Perl. Again, that's of no use to > most Windows users. C will work on all platforms from the outset, with > no dependencies required. Of course, the server side doesn't matter. I'll do it in Perl because: 1. It's what I know. 2. There's a lot to draw on already (CPAN.pm, JSAN::Client, CPANPLUS) 3. JFDI. The nice thing is that anyone will be able to write a client if they want it in some other language. Best, David
On Thu, Jan 7, 2010 at 9:22 PM, Josh Berkus <josh@agliodbs.com> wrote: > Dave, > >> Whilst the aim is a noble one, glossing over 'it won't work on >> Windows' which is by far our most popular platform these days seems >> incredibly short sighted, and liable to lead to an endless stream of >> 'why doesn't this work' questions. It also does the module authors no >> favours, by excluding 50%+ of their potential audience, and frankly, >> isn't the way this project generally works. > > But why is that *this* project's job to solve? Because if we (PostgreSQL) are going to support this effort, then it should not ignore such a huge percentage of our installation base. > It's already the case > that contrib modules (or PostgreSQL) are not buildable on Windows in an > automated fashion. And that Windows does not ship with developer tools. > That's not PGAN's fault. No. But that's why we built binaries for people to use. That's pretty much my point. > Binary distributions are a good idea, for v2+. The problem with > requiring that is then you're effectively requiring every module author, > or a well-funded 3rd party, to have a huge build farm capable of > building binaries for a wide variety of platforms and PostgresQL > versions. Requiring that from the outset is a good way to ensure that > nobody *ever* builds an extension management platform. No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. Building them is no problem - authors can easily use EC2 for which we have an AMI pre-configured for next to no cost, can build on their own platform, on a community provided system, or get a friend to do it. >> We also discussed extension management at the DBMS level, which I >> believe Dimitri was working on in his spare time. You should look at >> what he's been doing. > > This is designed to be complimentary to Dimitri's project, if you read > the wiki page. > >> Finally, don't write the client in Perl. Again, that's of no use to >> most Windows users. C will work on all platforms from the outset, with >> no dependencies required. Of course, the server side doesn't matter. > > It would make sense to build a C version when the other issues with > supporting Windows are solved. At that point, the specification for the > client should be well-worn enough to make doing the C version not a halt > to further development. Until then, it doesn't matter. So you have to rewrite the code. Seems like a waste of effort. > What I'm getting from your e-mail, Dave, is "If it doesn't solve all > problems for everyone in the world from Day 1, it's not worth doing." No. The essence is, 'If you're going to do it in a way that will never work for maybe 50% or more of PostgreSQL installations, then you have fundamental design issues to overcome'. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Jan 7, 2010, at 1:31 PM, Dave Page wrote: > No, I'm suggesting the mechanism needs to support source and binary > distribution. For most *nix users, source will be fine. For Windows > binaries are required. I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. Best, David
Dave Page <dpage@pgadmin.org> writes: > We have discussed this sort of facility at previous developer > meetings, and as I recall came to the conclusion that we need to have > the ability to distribute pre-built binaries, not just source code as > virtually no Windows users are ever going to have a build environment > setup. Similarly, neither are Mac users, if they didn't install XCode. And I think that's exactly why David and you are both right. The way I see it, we need *both* a source level distribution infrastructure, which David is proposing to implement under the PGAN name, and a binary one too, which is implemented by distributions like debian, Fedora and the like, and by Devrim when it comes to PGDG RPMs repos and… by you when we talk about MacOSX and Windows binary distribution. I think the aim of the PGAN should be to make it damn easy for binary distributions to include whatever PGXS extension they whish, with about no effort. > We also discussed extension management at the DBMS level, which I > believe Dimitri was working on in his spare time. You should look at > what he's been doing. We've been talking about that together on IRC, and my only reserve here ain't technical, as we already have a somewhat advanced notion of an extension. What we miss is support for dump and restore, so my reserve is that pushing the current system to point-and-clic users when it's not ready to allow you to just pg_restore a dump when you installed any PGXS extension (contribs included) is going to have us *lose* users, not gain any. Unfortunately I was overly optimistic last May when I though I could reach a patch submission in the 8.5 cycle. I'd put it on too much work on the day and no more free time with the required energy when my son is so young (yes, that's a good problem to have). But still. What I have been able to do is to determine what features to include in a first extensions implementation, and proposing a syntax that no one refused (after less successful attempts). Also the work remaining to be done would represent easily a month of my evening and week-end time, and maybe a Tom's morning. But he wouldn't have the fun of learning anything new, and to paraphrase him, source contributors don't grow on trees :) Anyway I've tried to summary it all (with some code perspective) in the following wiki page: http://wiki.postgresql.org/wiki/ExtensionPackaging Essentially, it's all about coding a new backend function to execute commands from a file on the server, in the spirit of \i for psql, say e.g. pg_execute_commands_from_file('path/ to/file.sql'). It would not send any result back to the client because you can only describe the portal once for any given single query. Then you need to add a catalog for holding the extensions metadata, like the path of the install file, uninstall file, current version, upgrade function, custom class if any, path of the setup file (GUCs). And dependancies between extensions, and between a given extension and what core provides too (plpgsql e.g., or major postgresql version too). Now you can hack a CREATE EXTENSION command to fill-in the catalog, and the commands INSTALL EXTENSION and DROP EXTENSION to execute the given files. You have to invent a trick to be able to assign any SQL object created while executing the install file a dependency entry targeting the extension's catalog oid. As this is all done in a single given backend, my guess is that a static variable holding the oid would be enough, but you still have to touch all object creation code path. You do *not* have to care about schema in any way *BUT* to prepend the search_path with pg_extension just before executing the install file. It's a new schema to provide in template databases, so that extensions default to creating their objects there. This new schema should be installed just before pg_catalog in the search_path in places that search for objects from their non qualified name. Full stop on this topic. Oh and special bonus if CREATE EXTENSION, given a newer version of an already installed extension, will register things so as to run the upgrade function from the newer version at INSTALL EXTENSION time. Or invent yet another syntax for upgrading. And. Wait. That's it. Oh, you could also try to turn plpgsql, plperl, plpython and their unstrusted variants into built-in extensions too, and contribs too. It would be nice if contribs where all CREATEd into template0 and template1 and you only had to INSTALL EXTENSION hstore; for enjoying it. That'd be a Tom's easy evening I guess :) Anyone having the hours and skill required to do it? It's great fun, rewarding, everybody and his neighbour *want* it to be done, and I'll be as available as possible to answer mails, be there on IRC, and do the necessary political work if there's still some more to be done. Consider it as a gift I'm giving you: I'm refusing easily earned reputation by allowing you to code that for me ;) Regards, -- dim PS: that ought to do it.
On Thu, 2010-01-07 at 13:22 -0800, Josh Berkus wrote: > Dave, > What I'm getting from your e-mail, Dave, is "If it doesn't solve all > problems for everyone in the world from Day 1, it's not worth doing." I doubt that is Dave's intent because then we might as well stop work on PostgreSQL too. > David's proposal is designed to be something which he can get done *this > year*, possibly before 8.5 is released, and be built on later. It'll be > useful to a substantial number of our users, and will be an improvement > on what we have now. > I say if Wheeler wants to do it, he should do it and stop quibbling with the rudiments of the Org opinion. Guess how many people have opinions? Buildfarm succeeded because someone stepped up, evaluated, took a little feedback and ran with it. He didn't screw around waiting for everyone to be happy. So +1 on Wheeler's idea. Joshua D. Drake > --Josh Berkus > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
> Building them is no problem - authors can easily use EC2 for which we > have an AMI pre-configured for next to no cost, can build on their own > platform, on a community provided system, or get a friend to do it. So any module author, in order to submit any module, would be required to build binaries for 8-12 platforms covering up to 5 PostgreSQL versions (i.e. 30-70 different binaries)? At their own cost? Seems like a good way to not get any contributions at all. For that matter, Andrew just pointed out to me (corrected me, actually) on IRC that if a Windows user has MSVC or Mingw installed, it should be no problem supporting them. So what you're asking has nothing to do with Windows users, but is a more general "we want support for users who don't have a compiler installed". That's a different problem -- and one which the One-click installer or Stackbuilder should probably solve rather than PGAN. > No. The essence is, 'If you're going to do it in a way that will never > work for maybe 50% or more of PostgreSQL installations, then you have > fundamental design issues to overcome'. Again, that's the wrong attitude. You're saying "If it doesn't work for 100% of Windows users from Day 1, it won't ever work for them." By that logic, we should have held back version 7.4 until the windows port was done, dammit! And we shouldn't have released until it worked with Visual C++. Even if it meant releasing in 2007. And we shouldn't have released PITR until it supported HS. And we shouldn't be releasing 8.5 without Win64 support. The reason I'm harping on this is this list has a real tendency to reject simple solutions which allow room for growth in favor of overcomplicated world-spanning specifications which will never be implemented. Despite the fact that we only have some notable features because we took the simple approach: partitioning, PITR, the Buildfarm, CommitFests. And that features which there is no obvious way to implement in small steps incrementally (Windows Port, HS, HOT) are huge development problems for us. This list *also* has a real tendency to have an incredible negative attitude, which *you* are currently expressing. The constructive way for you to approach this would have been to say "I think that the general idea has merit, but that putting off Windows support is a mistake. What about supporting binary distribution at the outset? And coding the client in C?" Instead, you said "this doesn't solve problems A, B, and C, so it's stupid." Building a simple solution which doesn't initially cover all bases but can be steadily improved is a far superior strategy to trying to spec The Perfect Solution before even starting work. And if we want to keep recruiting new contributors, criticism needs to be more constructive. --Josh Berkus
On Jan 7, 2010, at 1:59 PM, Joshua D. Drake wrote: > So +1 on Wheeler's idea. Thanks! David
On tor, 2010-01-07 at 12:07 -0800, David E. Wheeler wrote: > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. Sounds good. You might want to clarify in your prose what an "extension" is. I suspect I know what you mean, but perhaps not everyone does.
On Jan 7, 2010, at 2:11 PM, Peter Eisentraut wrote: > You might want to clarify in your prose what an "extension" is. I > suspect I know what you mean, but perhaps not everyone does. Good suggestion, thanks. How about this in the FAQ? * WTF is an "extension"? An extension is a piece of software that adds functionality to PostgreSQL itself. Examples are data types (CITEXT, PERIOD),utilities (newsysviews, pgTAP), and procedural languages (PL/Ruby, PL/R), among others. An extension is *not* a pieceof software designed to run on top of PostgreSQL (Bricolage, Drupal). Best, David
"David E. Wheeler" <david@kineticode.com> writes: > On Jan 7, 2010, at 2:11 PM, Peter Eisentraut wrote: > >> You might want to clarify in your prose what an "extension" is. I >> suspect I know what you mean, but perhaps not everyone does. > > Good suggestion, thanks. How about this in the FAQ? > > * WTF is an "extension"? > > An extension is a piece of software that adds functionality to > PostgreSQL itself. Examples are data types (CITEXT, PERIOD), utilities > (newsysviews, pgTAP), and procedural languages (PL/Ruby, PL/R), among > others. An extension is *not* a piece of software designed to run on > top of PostgreSQL (Bricolage, Drupal). Maybe with a link to: http://www.postgresql.org/docs/8.4/static/extend.html In the sections that follow, we will discuss how you can extend the PostgreSQL SQL query language by adding: - functions (starting in Section 34.3) - aggregates (starting in Section 34.10) - data types (starting in Section 34.11) - operators (starting in Section 34.12) - operator classes for indexes (starting in Section 34.14) The PostgreSQL server can moreover incorporate user-written code into itself through dynamic loading. That is, the usercan specify an object code file (e.g., a shared library) that implements a new type or function, and PostgreSQL willload it as required. Code written in SQL is even more trivial to add to the server. This ability to modify its operation"on the fly" makes PostgreSQL uniquely suited for rapid prototyping of new applications and storage structures. Regards, -- dim
On Jan 7, 2010, at 2:23 PM, Dimitri Fontaine wrote: > Maybe with a link to: > http://www.postgresql.org/docs/8.4/static/extend.html Good call, thanks. David
Dimitri Fontaine <dfontaine@hi-media.com> writes:
> e.g. pg_execute_commands_from_file('path/ to/file.sql'). It would not
[...]
> Then you need to add a catalog for holding the extensions metadata, like
[...]
> Now you can hack a CREATE EXTENSION command to fill-in the catalog, and
> the commands INSTALL EXTENSION and DROP EXTENSION to execute the given
> files. You have to invent a trick to be able to assign any SQL object
> created while executing the install file a dependency entry targeting
> the extension's catalog oid. As this is all done in a single given
> backend, my guess is that a static variable holding the oid would be
> enough, but you still have to touch all object creation code path.
Now that you have dependency information tracked, don't forget to add
pg_dump support by skipping any objects installed as part of an
extension and only issuing INSTALL EXTENSION foo; instead.
Then let pg_restore run this and complain loudly when the needed
extension entry does not exists yet, you have to CREATE and INSTALL the
extension on the new database before getting to restore your dump.
> You do *not* have to care about schema in any way *BUT* to prepend the
> search_path with pg_extension just before executing the install
> file. It's a new schema to provide in template databases, so that
> extensions default to creating their objects there. This new schema
> should be installed just before pg_catalog in the search_path in places
> that search for objects from their non qualified name. Full stop on this
> topic.
>
> Oh and special bonus if CREATE EXTENSION, given a newer version of an
> already installed extension, will register things so as to run the
> upgrade function from the newer version at INSTALL EXTENSION time. Or
> invent yet another syntax for upgrading.
>
> And. Wait. That's it.
>
> Oh, you could also try to turn plpgsql, plperl, plpython and their
> unstrusted variants into built-in extensions too, and contribs too. It
> would be nice if contribs where all CREATEd into template0 and template1
> and you only had to INSTALL EXTENSION hstore; for enjoying it. That'd be
> a Tom's easy evening I guess :)
>
> Anyone having the hours and skill required to do it? It's great fun,
> rewarding, everybody and his neighbour *want* it to be done, and I'll be
> as available as possible to answer mails, be there on IRC, and do the
> necessary political work if there's still some more to be done.
>
> Consider it as a gift I'm giving you: I'm refusing easily earned
> reputation by allowing you to code that for me ;)
>
> Regards,
> --
> dim
>
> PS: that ought to do it.
Hey, there's about a week left. :)
--
dim
On Thu, Jan 07, 2010 at 12:07:19PM -0800, David Wheeler wrote: > Hackers, > > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL > extensions. I've tried to closely follow the [CPAN philosophy][] to > come up with a plan that requires a minimum-work implementation that > builds on the existing PostgreSQL tools and the examples of the > [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I > would be very grateful for feedback and suggestions. +1 for a modest PGAN to start. :) Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
* Josh Berkus (josh@agliodbs.com) wrote:
> What I'm getting from your e-mail, Dave, is "If it doesn't solve all
> problems for everyone in the world from Day 1, it's not worth doing."
> It's my experience that the way to get OSS software that works is to
> build a little bit at a time, each delivery of which is useful to some
> people and solves some problems. Projects which attempt to paint the
> whole world never get anywhere.
+1 from me. I was just discussing this with some other folks at my
company and had to make the point that if all you do is show the CEO a
big fat $ figure, he's gonna say no. If you actually want to make it
happen, you have to show a progression to get there from where we are
today, with the incremental costs required along the way and the
functionality that you get for that.
> David's proposal is designed to be something which he can get done *this
> year*, possibly before 8.5 is released, and be built on later. It'll be
> useful to a substantial number of our users, and will be an improvement
> on what we have now.
Sounds excellent to me.
Thanks,
Stephen
* Dave Page (dpage@pgadmin.org) wrote:
> Because if we (PostgreSQL) are going to support this effort, then it
> should not ignore such a huge percentage of our installation base.
Not doing it day 1 is not ignoring. It's using what resources *are*
being made available to the best extent we can. If you're offering to
do the work for Windows, great!
> > It would make sense to build a C version when the other issues with
> > supporting Windows are solved. At that point, the specification for the
> > client should be well-worn enough to make doing the C version not a halt
> > to further development. Until then, it doesn't matter.
>
> So you have to rewrite the code. Seems like a waste of effort.
So the options are some Perl code that works for quite a few users or..
nothing because he's not a C hacker or doesn't want to write it in C?
Sounds like #1 is a win to me. If David's happy to write it in C to
begin with (presuming he has to write anything- if there's existing Perl
code that does 90% of what he needs, you're asking for alot more),
great. I'm even happy to encourage him to do that if it's anywhere
close to the same level of effort.
> No. The essence is, 'If you're going to do it in a way that will never
> work for maybe 50% or more of PostgreSQL installations, then you have
> fundamental design issues to overcome'.
And my vote is that you have to start somewhere and I strongly disagree
that what you're concerned with are serious *design* issues. What David
has described includes alot of implementation details, let's not confuse
the two. If the server-side had to be scrapped entirely and rewritten
to support Windows, you might have a leg to stand on. If adding Windows
support is an incremental change to the existing system (as a whole,
which, yes, I'd consider the port of a perl client app to C to be an
incremental change), then it's not a design issue.
Thanks,
Stephen
On Thu, 2010-01-07 at 20:36 +0000, Dave Page wrote: > On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler <david@kineticode.com> wrote: > > Hackers, > > > > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. > > Whilst the aim is a noble one, glossing over 'it won't work on > Windows' which is by far our most popular platform these days Although I understand your point, this is debatable to the point of a pointless point :P. The reason it seems like the most popular is that the only way to get the windows version is to download it from .Org/EDB. Linux/FreeBSD already has it in their own repositories. I do believe that we need to support windows, but not everything will. That is just the nature of it. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
David E. Wheeler wrote: > On Jan 7, 2010, at 1:31 PM, Dave Page wrote: >> No, I'm suggesting the mechanism needs to support source and binary >> distribution. For most *nix users, source will be fine. For Windows >> binaries are required. > > I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. +1. They did a nice job.
On Thu, 2010-01-07 at 13:22 -0800, Josh Berkus wrote: > Dave, > What I'm getting from your e-mail, Dave, is "If it doesn't solve all > problems for everyone in the world from Day 1, it's not worth doing." I doubt that is Dave's intent because then we might as well stop work on PostgreSQL too. > David's proposal is designed to be something which he can get done *this > year*, possibly before 8.5 is released, and be built on later. It'll be > useful to a substantial number of our users, and will be an improvement > on what we have now. > I say if Wheeler wants to do it, he should do it and stop quibbling with the rudiments of the Org opinion. Guess how many people have opinions? Buildfarm succeeded because someone stepped up, evaluated, took a little feedback and ran with it. He didn't screw around waiting for everyone to be happy. So +1 on Wheeler's idea. Joshua D. Drake > --Josh Berkus > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Jan 7, 2010, at 4:07 PM, Josh Berkus wrote: > Building a simple solution which doesn't initially cover all bases but > can be steadily improved is a far superior strategy to trying to spec > The Perfect Solution before even starting work. And if we want to keep > recruiting new contributors, criticism needs to be more constructive. I think we should call it Pecan. And paint it yellow! ;P +1 from me. -- Jim C. Nasby, Database Architect jim@nasby.net 512.569.9461 (cell) http://jim.nasby.net
On Fri, Jan 8, 2010 at 05:22, Ron Mayer <rm_pg@cheapcomplexdevices.com> wrote: > David E. Wheeler wrote: >> On Jan 7, 2010, at 1:31 PM, Dave Page wrote: >>> No, I'm suggesting the mechanism needs to support source and binary >>> distribution. For most *nix users, source will be fine. For Windows >>> binaries are required. >> >> I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. > > +1. They did a nice job. For those of us who have no idea what Strawberry Perl did (other than not shipping Microsoft compatible libraries, and is thus useless for PostgreSQL), could someone explain it? -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Thu, Jan 7, 2010 at 10:07 PM, Josh Berkus <josh@agliodbs.com> wrote: >> Building them is no problem - authors can easily use EC2 for which we >> have an AMI pre-configured for next to no cost, can build on their own >> platform, on a community provided system, or get a friend to do it. > > So any module author, in order to submit any module, would be required > to build binaries for 8-12 platforms covering up to 5 PostgreSQL > versions (i.e. 30-70 different binaries)? At their own cost? Seems > like a good way to not get any contributions at all. No, I'm suggesting that for the module authors that want to support Windows, there are a variety of easy ways to get binaries built. It shouldn't be a requirement of *them*. > For that matter, Andrew just pointed out to me (corrected me, actually) > on IRC that if a Windows user has MSVC or Mingw installed, it should be > no problem supporting them. If it were likely that Windows users would have a suitable compiler installed, I wouldn't be complaining. > So what you're asking has nothing to do with Windows users, but is a > more general "we want support for users who don't have a compiler > installed". That's a different problem -- and one which the One-click > installer or Stackbuilder should probably solve rather than PGAN. It already does - on Windows, Linux and Mac. I don't need this feature, and nor do the users of my distributions. I just thought I was being helpful by pointing out some problems in David's design which I assumed was the point of the RFC. >> No. The essence is, 'If you're going to do it in a way that will never >> work for maybe 50% or more of PostgreSQL installations, then you have >> fundamental design issues to overcome'. > > Again, that's the wrong attitude. You're saying "If it doesn't work for > 100% of Windows users from Day 1, it won't ever work for them." By that > logic, we should have held back version 7.4 until the windows port was > done, dammit! And we shouldn't have released until it worked with > Visual C++. Even if it meant releasing in 2007. And we shouldn't have > released PITR until it supported HS. And we shouldn't be releasing 8.5 > without Win64 support. Please re-read what I said before churning out ridiculous arguments that in no way relate to what I wrote. I am saying that if the design won't ever work without requiring painful dependency installation that users will likely not want to bother with, then it is fundamentally broken. Better to write one system that can _eventually_ work everywhere, than one that works for some, and then another that works elsewhere. > This list *also* has a real tendency to have an incredible negative > attitude, which *you* are currently expressing. The constructive way > for you to approach this would have been to say "I think that the > general idea has merit, but that putting off Windows support is a > mistake. What about supporting binary distribution at the outset? Again, please read what I wrote. That is almost exactly what I did write. I said it was a good idea, I explained why I thought ignoring Windows was an issue, and I noted that we had previously discussed binary distribution. > coding the client in C?" Instead, you said "this doesn't solve problems > A, B, and C, so it's stupid." No - I said it won't work for most windows users, use C instead as that'll work everywhere. At no point did I say or imply it was "stupid", just that it wasn't a universal solution. > Building a simple solution which doesn't initially cover all bases but > can be steadily improved is a far superior strategy to trying to spec > The Perfect Solution before even starting work. And if we want to keep > recruiting new contributors, criticism needs to be more constructive. For every criticism I made, I gave my reasoning, and offered a solution or an idea that would resolve my concern. In what bloody way is that not constructive? Remember, David posted an RFC. I'm not going to fill this list with grovelling bullshit about how every line he wrote is genius. I commented on the parts I considered a problem, and gave an overall nod to the rest. If all you want on this list is positive comments, then I won't bother any more as that won't help anyone. /D
Hi, Josh Berkus wrote: > Dave wrote: >> and frankly, >> isn't the way this project generally works. Isn't it? We didn't even support Windows for quite a long time. We still have lots more active Unix developers and knowledge that Windows ones. And isn't there some "scratch your own itch" philosophy in every OSS project? > But why is that *this* project's job to solve? It's already the case > that contrib modules (or PostgreSQL) are not buildable on Windows in an > automated fashion. And that Windows does not ship with developer tools. > That's not PGAN's fault. I'm with Josh here, get a PGAN up and running. Make it accessible via web to promote extensibility of Postgres and availability of extensions. Whether the first incarnation of the PGAN client works on Windows or Linux doesn't matter for now. Regards Markus Wanner
On Fri, Jan 8, 2010 at 10:40 AM, Markus Wanner <markus@bluegap.ch> wrote: > Hi, > > Josh Berkus wrote: >> >> Dave wrote: >>> >>> and frankly, >>> isn't the way this project generally works. > > Isn't it? We didn't even support Windows for quite a long time. No, it's quite different for the PostgreSQL not to support a platform at all, than for it to say "it's supported on FOO/OS, but you can't use Window Functions" or "on BAR/OS, you can't use PITR". The only reason we ever offer different functionality on different platforms is when there are external reasons forcing us to - for example, lack of reparse points in NTFS on Windows NT 4.0 prevented us offering table space support, and for some time we had no Win32 port of libuuid, so couldn't offer the UUID contrib module. There are no such problems with this project that I can see - at least that cannot be overcome with a little thought and appropriate tool selection. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Dave Page wrote: > The only reason we ever offer different functionality on different > platforms is when there are external reasons forcing us to - for > example, lack of reparse points in NTFS on Windows NT 4.0 prevented us > offering table space support, and for some time we had no Win32 port > of libuuid, so couldn't offer the UUID contrib module. There are no > such problems with this project that I can see - at least that cannot > be overcome with a little thought and appropriate tool selection. > Dave, Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. I have long spoken against making Windows a second class citizen. But I don't think David is going to do that (and I'll hound him if he does). But that doesn't mean it has to be fully supported from day one. Personally, I'd like to see us with a build service for such modules (c.f. OpenSuse's build service). cheers andrew
Magnus Hagander wrote: > On Fri, Jan 8, 2010 at 05:22, Ron Mayer <rm_pg@cheapcomplexdevices.com> wrote: >> David E. Wheeler wrote: >>> On Jan 7, 2010, at 1:31 PM, Dave Page wrote: >>>> No, I'm suggesting the mechanism needs to support source and binary >>>> distribution. For most *nix users, source will be fine. For Windows >>>> binaries are required. >>> I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. >> +1. They did a nice job. > > For those of us who have no idea what Strawberry Perl did (other than > not shipping Microsoft compatible libraries, and is thus useless for > PostgreSQL), could someone explain it? > As far as I can tell they shipped the minimal set of tools that they needed to build extensions rather than distribute binaries. http://en.wikipedia.org/wiki/Strawberry_Perl I don't know the details, but it works smoothly for them.
On Fri, Jan 8, 2010 at 15:14, Ron Mayer <rm_pg@cheapcomplexdevices.com> wrote: > Magnus Hagander wrote: >> On Fri, Jan 8, 2010 at 05:22, Ron Mayer <rm_pg@cheapcomplexdevices.com> wrote: >>> David E. Wheeler wrote: >>>> On Jan 7, 2010, at 1:31 PM, Dave Page wrote: >>>>> No, I'm suggesting the mechanism needs to support source and binary >>>>> distribution. For most *nix users, source will be fine. For Windows >>>>> binaries are required. >>>> I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. >>> +1. They did a nice job. >> >> For those of us who have no idea what Strawberry Perl did (other than >> not shipping Microsoft compatible libraries, and is thus useless for >> PostgreSQL), could someone explain it? >> > > As far as I can tell they shipped the minimal set of tools that they needed > to build extensions rather than distribute binaries. > http://en.wikipedia.org/wiki/Strawberry_Perl > I don't know the details, but it works smoothly for them. Right. They are msys compatible, not Microsoft compatible. And msys uses non-Windows-standard format of the import libraries. and the import libraries are needed for people to link to the interpreter, like pl/perl. Also a quick re-check shows that there appears to be no 64-bit version available. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Hey Andrew On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > Windows came late to the buildfarm. According to the CVS log, the buildfarm > client was first checked in in Sept 2004, got initial Mingw support in Jan > 2005 and MSVC support in March 2007, when we finally got some of the tools > sorted out. Right - but the buildfarm isn't a feature being offered to end users. > I have long spoken against making Windows a second class citizen. But I > don't think David is going to do that (and I'll hound him if he does). But > that doesn't mean it has to be fully supported from day one. I'm not saying it should be supported from day 1, but I think the initial plan will make it very difficult to add Windows support later without a great deal of rewriting/redesign. It's lack of forward planning I was objecting to. > Personally, I'd like to see us with a build service for such modules (c.f. > OpenSuse's build service). +1 -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Fri, Jan 8, 2010 at 10:12 AM, Dave Page <dpage@pgadmin.org> wrote: >> I have long spoken against making Windows a second class citizen. But I >> don't think David is going to do that (and I'll hound him if he does). But >> that doesn't mean it has to be fully supported from day one. > > I'm not saying it should be supported from day 1, but I think the > initial plan will make it very difficult to add Windows support later > without a great deal of rewriting/redesign. It's lack of forward > planning I was objecting to. I personally suspect that the client is not the most important part of this project. I think the value of CPAN is for searching, more than auto-installing. Personally, I never use the auto-install feature because I always want more control than you get that way. I just use the site to find possible modules and browse the docs, and then if I find something I like I check with I can pull it from the Red Hat repos with rpm, and if not I download it and look it over to see if it DWIW, and then if so I usually make a private SRPM for it and install from that. I'd be happy if we just had a good search-and-download site. That having been said, we should consider our filesystem layout carefully however to make sure that if we want to provide things like Windows installers in the future, we have a clean way to do that. ...Robert
On Fri, 2010-01-08 at 15:12 +0000, Dave Page wrote: > Hey Andrew > > On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > > > Windows came late to the buildfarm. According to the CVS log, the buildfarm > > client was first checked in in Sept 2004, got initial Mingw support in Jan > > 2005 and MSVC support in March 2007, when we finally got some of the tools > > sorted out. > > Right - but the buildfarm isn't a feature being offered to end users. > Neither is this. These people are developers or DBAs after all. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > On Fri, 2010-01-08 at 15:12 +0000, Dave Page wrote: >> Hey Andrew >> >> On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: >> > >> > Windows came late to the buildfarm. According to the CVS log, the buildfarm >> > client was first checked in in Sept 2004, got initial Mingw support in Jan >> > 2005 and MSVC support in March 2007, when we finally got some of the tools >> > sorted out. >> >> Right - but the buildfarm isn't a feature being offered to end users. >> > > Neither is this. These people are developers or DBAs after all. I know I'm a little slower than normal this week as I've had some sort of virus, but say what? I was kinda under the impression that developers/DBAs are end users of PostgreSQL. They make up a significant percentage of people that I know than install, configure and use it. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Fri, 2010-01-08 at 16:33 +0000, Dave Page wrote: > On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > > On Fri, 2010-01-08 at 15:12 +0000, Dave Page wrote: > >> Hey Andrew > >> > >> On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > >> > > >> > Windows came late to the buildfarm. According to the CVS log, the buildfarm > >> > client was first checked in in Sept 2004, got initial Mingw support in Jan > >> > 2005 and MSVC support in March 2007, when we finally got some of the tools > >> > sorted out. > >> > >> Right - but the buildfarm isn't a feature being offered to end users. > >> > > > > Neither is this. These people are developers or DBAs after all. > > I know I'm a little slower than normal this week as I've had some sort > of virus, but say what? I was kinda under the impression that > developers/DBAs are end users of PostgreSQL. They make up a > significant percentage of people that I know than install, configure > and use it. My point is... if they are a developer or DBA, they aren't going to have any trouble dealing with these issues. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Fri, Jan 8, 2010 at 4:34 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > On Fri, 2010-01-08 at 16:33 +0000, Dave Page wrote: >> On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake <jd@commandprompt.com> wrote: >> > On Fri, 2010-01-08 at 15:12 +0000, Dave Page wrote: >> >> Hey Andrew >> >> >> >> On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: >> >> > >> >> > Windows came late to the buildfarm. According to the CVS log, the buildfarm >> >> > client was first checked in in Sept 2004, got initial Mingw support in Jan >> >> > 2005 and MSVC support in March 2007, when we finally got some of the tools >> >> > sorted out. >> >> >> >> Right - but the buildfarm isn't a feature being offered to end users. >> >> >> > >> > Neither is this. These people are developers or DBAs after all. >> >> I know I'm a little slower than normal this week as I've had some sort >> of virus, but say what? I was kinda under the impression that >> developers/DBAs are end users of PostgreSQL. They make up a >> significant percentage of people that I know than install, configure >> and use it. > > My point is... if they are a developer or DBA, they aren't going to have > any trouble dealing with these issues. What issues? Lack of support from PGAN, or having to download and install a complex build environment weighing in at over a gigabyte of downloads, just to install an add-on? -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Dave,
* Dave Page (dpage@pgadmin.org) wrote:
> Right - but the buildfarm isn't a feature being offered to end users.
And this network isn't a feature of the core code either, nor, do I
believe, is it being designed in a way that would require an overhaul
down the road to support Windows. To be honest, I really don't feel
this compares to features in the core code to begin with- I feel like
generally we don't like things which only work on a subset of platforms
because it's a small incremental change, and typically just requires
good programming practice, to make it work for all supported platforms.
That isn't true for this.
> I'm not saying it should be supported from day 1, but I think the
> initial plan will make it very difficult to add Windows support later
> without a great deal of rewriting/redesign. It's lack of forward
> planning I was objecting to.
I disagree with this, but in the end I don't think that it really
matters. If all we have resources for to work on this is a Perl expect,
then by golly let that be a dependency for the initial work. If there
are other resources who are willing to write the initial version in C,
or what have you, let them step up and offer. This isn't a zero-sum
game here and I'm strongly against telling someone "please don't work on
this" when it's something which will be of benefit to many users.
Thanks,
Stephen
On Jan 8, 2010, at 1:35 AM, Dave Page wrote: > I am saying that if the design won't ever work without requiring > painful dependency installation that users will likely not want to > bother with, then it is fundamentally broken. Better to write one > system that can _eventually_ work everywhere, than one that works for > some, and then another that works elsewhere. This whole bit about Windows is a red herring. Perhaps I should not have phrased it the way I did WRT Windows. So I'm goingto change it to: > The PGAN client will make no other assumptions about how to build and install extensions, leaving such to the PostgreSQLcore. To the extent that PGXS-powered `make` works on a given platform, the client will support it. Discussing it with Andrew, that means it should work if you have mingw, and we might have to tweak it a bit to work with`src/tools/msvc`. So the point is: the PGAN client (which is just one part of this project, after all), will *not* include a build system.It will use whatever build system is supported by the community. Right now that's PGXS. If core switches to somethinglater, or provides binary builds for Windows, the client will be easily adapted to take advantage of it. No sweat. The upshot is this: PGAN does *not* ignore Windows or any other platform. Rather, it relies on others to create the appropriatecommunity-supported installers for all platforms. The issue of build systems and installers is not within itsdomain. Thus, I've also changed the FAQ to: > * '''What about Windows?''' The PGAN client will always follow the lead of the PostgreSQL core on installing extensions.If support for installing extensions on Windows improves such that a compiler is no longer required, the PGANclient will be modified as appropriate to take advantage of it. This applies not specifically to Windows, but to theability of the core intaller (or any future community-supported installer) to work on ''any'' platform. Please let the Windows thread die now. PGAN doesn't ignore Windows; it ignores installer development. Best, David
On Fri, Jan 8, 2010 at 5:13 PM, David E. Wheeler <david@kineticode.com> wrote: > This whole bit about Windows is a red herring. Perhaps I should not have phrased it the way I did WRT Windows. So I'm goingto change it to: > >> The PGAN client will make no other assumptions about how to build and install extensions, leaving such to the PostgreSQLcore. To the extent that PGXS-powered `make` works on a given platform, the client will support it. > > Discussing it with Andrew, that means it should work if you have mingw, and we might have to tweak it a bit to work with`src/tools/msvc`. > > So the point is: the PGAN client (which is just one part of this project, after all), will *not* include a build system.It will use whatever build system is supported by the community. Right now that's PGXS. If core switches to somethinglater, or provides binary builds for Windows, the client will be easily adapted to take advantage of it. No sweat. > > The upshot is this: PGAN does *not* ignore Windows or any other platform. Rather, it relies on others to create the appropriatecommunity-supported installers for all platforms. The issue of build systems and installers is not within itsdomain. Thus, I've also changed the FAQ to: If that is the goal of your project then I withdraw my previous comments, which were written on the belief that you were proposing a generic distribution/build/installation system for PostgreSQL users. Sorry for the noise! -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Fri, Jan 8, 2010 at 5:13 PM, David E. Wheeler <david@kineticode.com> wrote: > Please let the Windows thread die now. PGAN doesn't ignore Windows; it ignores installer development. > yeah, I think there are two quite separable projects here. It's quite possible that once the binary installer people have a source package with all the meta data and computer-readable build instructions that PGAN will need that they'll be able to implement support for binary pre-built modules as well. But just tackling the source package problem solves a lot of the problems already. The binary installer people will have their own set of problems to solve as well, they'll have to deal with environment dependencies such as OS versions, external binary versions, privileges, etc. These are all additional problems on top of having a source package already. To make an analogy configure and make are still useful even after you have dpkg -- you still need to be able to build the individual source package first before you get to put it into a binary package. -- greg
On Thu, Jan 7, 2010 at 21:07, David E. Wheeler <david@kineticode.com> wrote: > Hackers, > > I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. > > [plan]: http://wiki.postgresql.org/wiki/PGAN > [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 > [CPAN]: http://cpan.org/ > [JSAN]: http://www.openjsan.org > [JFDI]: http://acronyms.thefreedictionary.com/JFDI One note on a completely different topic from where this thread appears to have gone.. :-) Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Fri, Jan 8, 2010 at 5:38 PM, Magnus Hagander <magnus@hagander.net> wrote: > On Thu, Jan 7, 2010 at 21:07, David E. Wheeler <david@kineticode.com> wrote: >> Hackers, >> >> I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. >> >> [plan]: http://wiki.postgresql.org/wiki/PGAN >> [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 >> [CPAN]: http://cpan.org/ >> [JSAN]: http://www.openjsan.org >> [JFDI]: http://acronyms.thefreedictionary.com/JFDI > > One note on a completely different topic from where this thread > appears to have gone.. :-) > > Is there a particular reason not to use the existing mirroring network > to distribute the files? If not, then I suggest using them should be > part of the design. With StackBuilder, a package can either be listed as a path, relative to the FTP root on the mirror network, or using an alternate URL for something stored elsewhere. The current set of active mirrors can always be found at http://www.postgresql.org/mirrors.xml, so you can build URLs on the mirror network using the protocol, host, port and path from the mirror list, and then the relative path for the file. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Fri, Jan 8, 2010 at 18:44, Dave Page <dpage@pgadmin.org> wrote: > On Fri, Jan 8, 2010 at 5:38 PM, Magnus Hagander <magnus@hagander.net> wrote: >> On Thu, Jan 7, 2010 at 21:07, David E. Wheeler <david@kineticode.com> wrote: >>> Hackers, >>> >>> I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][]to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL toolsand the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedbackand suggestions. >>> >>> [plan]: http://wiki.postgresql.org/wiki/PGAN >>> [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 >>> [CPAN]: http://cpan.org/ >>> [JSAN]: http://www.openjsan.org >>> [JFDI]: http://acronyms.thefreedictionary.com/JFDI >> >> One note on a completely different topic from where this thread >> appears to have gone.. :-) >> >> Is there a particular reason not to use the existing mirroring network >> to distribute the files? If not, then I suggest using them should be >> part of the design. > > With StackBuilder, a package can either be listed as a path, relative > to the FTP root on the mirror network, or using an alternate URL for > something stored elsewhere. > > The current set of active mirrors can always be found at > http://www.postgresql.org/mirrors.xml, so you can build URLs on the > mirror network using the protocol, host, port and path from the mirror > list, and then the relative path for the file. Or you can hit it off the mirror redirector on the website, which will then give you automatic download tracking. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
2010/1/8 Magnus Hagander <magnus@hagander.net>: > On Fri, Jan 8, 2010 at 18:44, Dave Page <dpage@pgadmin.org> wrote: >> The current set of active mirrors can always be found at >> http://www.postgresql.org/mirrors.xml, so you can build URLs on the >> mirror network using the protocol, host, port and path from the mirror >> list, and then the relative path for the file. > > Or you can hit it off the mirror redirector on the website, which will > then give you automatic download tracking. Right, but you need the mirror ID for that iirc. Maybe we should add that to the XML... -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Jan 8, 2010, at 9:24 AM, Dave Page wrote: > If that is the goal of your project then I withdraw my previous > comments, which were written on the belief that you were proposing a > generic distribution/build/installation system for PostgreSQL users. It is a generic distribution and installation system, but it just uses installer approaches provided by others. This is notunlike CPAN.pm, which doesn't include an installer itself, but things through Module::Build or ExtUtils::MakeMaker asappropriate. Completely separate domain problem, as Greg notes. > Sorry for the noise! Glad to have it cleared up! :-) Best, David
On Jan 8, 2010, at 9:38 AM, Magnus Hagander wrote: > Is there a particular reason not to use the existing mirroring network > to distribute the files? If not, then I suggest using them should be > part of the design. No, as long as PAUS can drop uploaded distributions onto the master FTP server, or else the existing mirror system can rsyncfrom PGAN's own master (I'll build all this on my own box to start with). It'll just be rsync, really, it won't whereit's mirrored or where the master index lives. Best, David
On Fri, Jan 8, 2010 at 5:55 PM, David E. Wheeler <david@kineticode.com> wrote: > On Jan 8, 2010, at 9:38 AM, Magnus Hagander wrote: > >> Is there a particular reason not to use the existing mirroring network >> to distribute the files? If not, then I suggest using them should be >> part of the design. > > No, as long as PAUS can drop uploaded distributions onto the master FTP server, or else the existing mirror system canrsync from PGAN's own master (I'll build all this on my own box to start with). It'll just be rsync, really, it won'twhere it's mirrored or where the master index lives. Either can be arranged. For StackBuilder, we created a pgFoundry project, so files can just be uploaded there by authorised users, from where they get replicated back onto the mirror network. Which leads us neatly back to the GForge URL thread :-) BTW, what's a PAUS? I missed that one... -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Jan 8, 2010, at 9:59 AM, Dave Page wrote: > Either can be arranged. For StackBuilder, we created a pgFoundry > project, so files can just be uploaded there by authorised users, from > where they get replicated back onto the mirror network. > > Which leads us neatly back to the GForge URL thread :-) > > BTW, what's a PAUS? I missed that one... That's the very first part of PGAN: • PAUS: The PostgreSQL Authors Upload Server. Users will be able to create logins and upload extension distributions. Details in the wiki. http://wiki.postgresql.org/wiki/PGAN Best, David
On Fri, Jan 8, 2010 at 18:55, David E. Wheeler <david@kineticode.com> wrote: > On Jan 8, 2010, at 9:38 AM, Magnus Hagander wrote: > >> Is there a particular reason not to use the existing mirroring network >> to distribute the files? If not, then I suggest using them should be >> part of the design. > > No, as long as PAUS can drop uploaded distributions onto the master FTP server, or else the existing mirror system canrsync from PGAN's own master (I'll build all this on my own box to start with). It'll just be rsync, really, it won'twhere it's mirrored or where the master index lives. Either can be easily arranged. Then you just need some logic in the client thingy to actually grab stuff from the mirrors. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Fri, Jan 8, 2010 at 18:48, Dave Page <dpage@pgadmin.org> wrote: > 2010/1/8 Magnus Hagander <magnus@hagander.net>: >> On Fri, Jan 8, 2010 at 18:44, Dave Page <dpage@pgadmin.org> wrote: >>> The current set of active mirrors can always be found at >>> http://www.postgresql.org/mirrors.xml, so you can build URLs on the >>> mirror network using the protocol, host, port and path from the mirror >>> list, and then the relative path for the file. >> >> Or you can hit it off the mirror redirector on the website, which will >> then give you automatic download tracking. > > Right, but you need the mirror ID for that iirc. Maybe we should add > that to the XML... That, or implement that "send me to a random mirror" feature. Or maybe the "send me to a random close mirror if available, or a random global if not" feature. :-) Either way, there's definitely room for some improvement there, but let's figure out what the exact needs are first :-) -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Jan 8, 2010, at 10:08 AM, Magnus Hagander wrote: > That, or implement that "send me to a random mirror" feature. Or > maybe the "send me to a random close mirror if available, or a random > global if not" feature. :-) > > Either way, there's definitely room for some improvement there, but > let's figure out what the exact needs are first :- I'll come back to this when I'm actually implementing it, for sure. Best, David
Magnus Hagander wrote: > On Fri, Jan 8, 2010 at 18:48, Dave Page <dpage@pgadmin.org> wrote: >> 2010/1/8 Magnus Hagander <magnus@hagander.net>: >>> On Fri, Jan 8, 2010 at 18:44, Dave Page <dpage@pgadmin.org> wrote: >>>> The current set of active mirrors can always be found at >>>> http://www.postgresql.org/mirrors.xml, so you can build URLs on the >>>> mirror network using the protocol, host, port and path from the mirror >>>> list, and then the relative path for the file. >>> Or you can hit it off the mirror redirector on the website, which will >>> then give you automatic download tracking. >> Right, but you need the mirror ID for that iirc. Maybe we should add >> that to the XML... > > That, or implement that "send me to a random mirror" feature. Or > maybe the "send me to a random close mirror if available, or a random > global if not" feature. :-) > > Either way, there's definitely room for some improvement there, but > let's figure out what the exact needs are first :-) well we could fairly trivially do that using DNS as well as in providing sets of mirrors geographically spread out (like us.mirrors.postgresql.org, eu.mirrors.postgresql.org) or whatever. Stefan
On Fri, 2010-01-08 at 15:12 +0000, Dave Page wrote: > Hey Andrew > > On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > > > Windows came late to the buildfarm. According to the CVS log, the buildfarm > > client was first checked in in Sept 2004, got initial Mingw support in Jan > > 2005 and MSVC support in March 2007, when we finally got some of the tools > > sorted out. > > Right - but the buildfarm isn't a feature being offered to end users. > Neither is this. These people are developers or DBAs after all. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Fri, 2010-01-08 at 16:33 +0000, Dave Page wrote: > On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > > On Fri, 2010-01-08 at 15:12 +0000, Dave Page wrote: > >> Hey Andrew > >> > >> On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > >> > > >> > Windows came late to the buildfarm. According to the CVS log, the buildfarm > >> > client was first checked in in Sept 2004, got initial Mingw support in Jan > >> > 2005 and MSVC support in March 2007, when we finally got some of the tools > >> > sorted out. > >> > >> Right - but the buildfarm isn't a feature being offered to end users. > >> > > > > Neither is this. These people are developers or DBAs after all. > > I know I'm a little slower than normal this week as I've had some sort > of virus, but say what? I was kinda under the impression that > developers/DBAs are end users of PostgreSQL. They make up a > significant percentage of people that I know than install, configure > and use it. My point is... if they are a developer or DBA, they aren't going to have any trouble dealing with these issues. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
robertmhaas@gmail.com (Robert Haas) writes:
> On Fri, Jan 8, 2010 at 10:12 AM, Dave Page <dpage@pgadmin.org> wrote:
>>> I have long spoken against making Windows a second class citizen. But I
>>> don't think David is going to do that (and I'll hound him if he does). But
>>> that doesn't mean it has to be fully supported from day one.
>>
>> I'm not saying it should be supported from day 1, but I think the
>> initial plan will make it very difficult to add Windows support later
>> without a great deal of rewriting/redesign. It's lack of forward
>> planning I was objecting to.
>
> I personally suspect that the client is not the most important part of
> this project. I think the value of CPAN is for searching, more than
> auto-installing. Personally, I never use the auto-install feature
> because I always want more control than you get that way. I just use
> the site to find possible modules and browse the docs, and then if I
> find something I like I check with I can pull it from the Red Hat
> repos with rpm, and if not I download it and look it over to see if it
> DWIW, and then if so I usually make a private SRPM for it and install
> from that. I'd be happy if we just had a good search-and-download
> site.
If "PGAN" leads to us having:
a) A database containing a useful set of metadata about a large set of extensions, and
b) A way for PostgreSQL developers and binary distribution makers (who *do* have GCC / XCode / MingW / Visual Studio
/... available to them) to easily: - build - test - try out - think about how to package that large
setof extensions
then we've got a Big Win of the same sort as CPAN, Ruby Gems, and PyPI.
It does NOT need to include installers for every known kind of computer;
that is a *second* problem, which actually requires a series of
solutions for:a) Fedorab) Debian (hence derivatives like Ubuntu)c) BSD Portsd) Yes, Windowse) I think Solaris has
somethingnew for packaging...
If David gets it to the point where it's easy to build and install
extensions into a PostgreSQL installation, then turning that into
packages for specific targets should be a not-insurmountable problem
that may be treated separately.
> That having been said, we should consider our filesystem layout
> carefully however to make sure that if we want to provide things like
> Windows installers in the future, we have a clean way to do that.
If the extensions get installed in a way that is "scalable" in the sense
that it's not a particularly big deal to write a script that pulls 250
extensions and installs them on a particular host for a particular PG
installation, then I'd think that the exercise has been a successful
one.
That leads, naturally enough, to an "Extension BuildFarm" :-).
I'd be somewhat surprised if the use of Windows was a material factor in
the matter.
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','acm.org').
http://www3.sympatico.ca/cbbrowne/postgresql.html
"Laugh-a while you can, Monkey Boy." -- Dr. Lizardo - Buckaroo Banzai