Thread: Planning final assault on query length limits
AFAICT, the last remaining textual length limits in the backend are a
couple of fixed-size buffers in rewriteDefine.c and array support.
In the next few days (probably this weekend) I intend to fix that code
and then remove all #define constants that have anything to do with
string length limits. (Tentative hit list is attached.)
Although the backend is in fairly good shape, there are parts of
interfaces/ and bin/ that are not. In particular:
pg_dump has a lot of uses of MAX_QUERY_SIZE. I believe Michael Ansley
is working on revising pg_dump to use expansible string buffers, so for
now I will leave that code alone (I'll just temporarily stick a
definition of MAX_QUERY_SIZE into pg_dump.h).
ecpg's lexer needs the same fixes recently applied to parser/scan.l to
eliminate a fixed-size literal buffer and allow flex's input buffer to
grow as needed. Michael Meskes usually handles ecpg --- Michael, do
you want to deal with this issue or shall I take a cut at it?
The odbc, python, and cli interfaces will all break because they contain
references to symbols I propose to remove. Since I don't use any of
these, and they aren't built by default, I can face this prospect
without flinching ;-). This is a call for whoever maintains these
modules to get busy.
ODBC will probably need some actual thought, since what it seems to be
doing with these symbols is making their values available to client
programs on request. Does ODBC's API for this function have a concept
of "no specific upper limit"?
regards, tom lane
Say goodnight to:
MAX_PARSE_BUFFER
MAX_QUERY_SIZE
ERROR_MSG_LENGTH
MAX_LINES
MAX_STRING_LENGTH
REMARK_LENGTH
ELOG_MAXLEN
MAX_BUFF_SIZE
PQ_BUFFER_SIZE
MAXPGPATH
MAX_STRING_LENGTH
YY_BUF_SIZE remove hacking in makefiles
YY_READ_BUF_SIZE no need to alter default
MaxHeapTupleSize (not used)
MaxAttributeSize (not used)
MaxAttrSize (where used for buffer sizing)
Suspicious-looking symbols found only in various interfaces/ dirs;
I don't plan to remove these but someone should:
SQL_PACKET_SIZE odbc
MAX_MESSAGE_LEN interfaces/cli, odbc
MAX_STATEMENT_LEN odbc
TEXT_FIELD_SIZE odbc
MAX_VARCHAR_SIZE odbc
DRV_VARCHAR_SIZE odbc
DRV_LONGVARCHAR_SIZE odbc
MAX_CONNECT_STRING odbc
MAX_BUFFER_SIZE interfaces/python
MAX_FIELDS odbc (does this relate to anything real?)
AFAICT, the last remaining textual length limits in the backend are a couple of fixed-size buffers in rewriteDefine.cand array support. In the next few days (probably this weekend) I intend to fix that code and then removeall #define constants that have anything to do with string length limits. (Tentative hit list is attached.) Great! Jan, does this mean that we can also lose the "rewrite string too big" problem with rules? That would be a huge win. Cheers, Brook
> AFAICT, the last remaining textual length limits in the backend are a > couple of fixed-size buffers in rewriteDefine.c and array support. > In the next few days (probably this weekend) I intend to fix that code > and then remove all #define constants that have anything to do with > string length limits. (Tentative hit list is attached.) > > Great! > > Jan, does this mean that we can also lose the "rewrite string too big" > problem with rules? > > That would be a huge win. No. We have to have long tuples. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> Jan, does this mean that we can also lose the "rewrite string too big" > problem with rules? No. We have to have long tuples. Darn. Oh well, I guess this is a major step in that direction. Cheers, Brook
Brook Milligan <brook@biology.nmsu.edu> writes:
>> Jan, does this mean that we can also lose the "rewrite string too big"
>> problem with rules?
> No. We have to have long tuples.
> Darn. Oh well, I guess this is a major step in that direction.
I'm hoping that once this is done, someone who knows the guts of the
storage managers better than I will feel motivated to work on letting
stored tuples cross block boundaries. (Paging Vadim...) That seems
to be the last piece of the puzzle.
regards, tom lane
Tom Lane wrote:
>
> Brook Milligan <brook@biology.nmsu.edu> writes:
> >> Jan, does this mean that we can also lose the "rewrite string too big"
> >> problem with rules?
>
> > No. We have to have long tuples.
>
> > Darn. Oh well, I guess this is a major step in that direction.
>
> I'm hoping that once this is done, someone who knows the guts of the
> storage managers better than I will feel motivated to work on letting
> stored tuples cross block boundaries. (Paging Vadim...) That seems
> to be the last piece of the puzzle.
You know that I'm busy with WAL...
And I already made some step in big tuples dirrection
when made memory/disk tuple presentations different -:)
typedef struct HeapTupleData
{ uint32 t_len; /* length of *t_data */ ItemPointerData t_self; /* SelfItemPointer */
HeapTupleHeadert_data; /* */ ^^^^^^^^^^^^^^^^^^^^^^ On-disk data
} HeapTupleData;
I hope that something could be added here for tuple chunks...
TupleTableSlot.ttc_buffer (and ttc_shouldFree?) is good candidate
to be moved here from TupleTableSlot.
As for smgr part - it's not hard at all.
Vadim
Vadim Mikheev <vadim@krs.ru> writes:
> Tom Lane wrote:
>> I'm hoping that once this is done, someone who knows the guts of the
>> storage managers better than I will feel motivated to work on letting
>> stored tuples cross block boundaries. (Paging Vadim...) That seems
>> to be the last piece of the puzzle.
> You know that I'm busy with WAL...
Well, Vadim doesn't want to do it, and I don't want to do it (I've
really got to spend some more time on the planner/optimizer, because
there's too much stuff half-fixed in there).
Any volunteers out there? It'd be a shame to not have this problem
area completely licked for 7.0.
regards, tom lane
> Vadim Mikheev <vadim@krs.ru> writes: > > Tom Lane wrote: > >> I'm hoping that once this is done, someone who knows the guts of the > >> storage managers better than I will feel motivated to work on letting > >> stored tuples cross block boundaries. (Paging Vadim...) That seems > >> to be the last piece of the puzzle. > > > You know that I'm busy with WAL... > > Well, Vadim doesn't want to do it, and I don't want to do it (I've > really got to spend some more time on the planner/optimizer, because > there's too much stuff half-fixed in there). > > Any volunteers out there? It'd be a shame to not have this problem > area completely licked for 7.0. Welcome to the small club, Tom. For the first 2 & 1/2 years, the only person who could tackle those big jobs was Vadim. Now you are in the club too. The problem is that there are no more. I can't imagine anyone is going to be able to jump out of the woodwork and take on a job like that. We will just have to do the best job we can, and maybe save something for 7.1. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian wrote: > > > > You know that I'm busy with WAL... > > > > Well, Vadim doesn't want to do it, and I don't want to do it (I've > > really got to spend some more time on the planner/optimizer, because > > there's too much stuff half-fixed in there). > > > > Any volunteers out there? It'd be a shame to not have this problem > > area completely licked for 7.0. > > Welcome to the small club, Tom. For the first 2 & 1/2 years, the only > person who could tackle those big jobs was Vadim. Now you are in the > club too. > > The problem is that there are no more. I can't imagine anyone is going > to be able to jump out of the woodwork and take on a job like that. We > will just have to do the best job we can, and maybe save something for > 7.1. There is Jan!... But he's busy too -:) Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks... Vadim
> > The problem is that there are no more. I can't imagine anyone is going > > to be able to jump out of the woodwork and take on a job like that. We > > will just have to do the best job we can, and maybe save something for > > 7.1. > > There is Jan!... > But he's busy too -:) > > Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks... Oops, forgot about him. Sorry Jan. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Vadim Mikheev <vadim@krs.ru> writes: > Bruce Momjian wrote: >>>> Any volunteers out there? It'd be a shame to not have this problem >>>> area completely licked for 7.0. >> >> Welcome to the small club, Tom. For the first 2 & 1/2 years, the only >> person who could tackle those big jobs was Vadim. Now you are in the >> club too. >> >> The problem is that there are no more. I can't imagine anyone is going >> to be able to jump out of the woodwork and take on a job like that. We >> will just have to do the best job we can, and maybe save something for >> 7.1. > There is Jan!... > But he's busy too -:) > Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks... Thing is, if Vadim could do it in two weeks (sounds about right), then maybe I could do it in three or four (I'd have to spend time studying parts of the backend that Vadim already knows, but I don't). It seems to me that some aspiring hacker who's already a little bit familiar with backend coding could do it in a month or two, with suitable study, and would in the process make great strides towards gurudom. This is a fairly localized task, if I'm not greatly mistaken about it. And there's plenty of time left before 7.0. So this seems like a perfect project for someone who wants to learn more about the backend and has some time to spend doing so. A year ago I didn't know a darn thing about the backend, so I'm a bit bemused to find myself being called a member of "the small club". Programming skills don't come out of nowhere, they come out of study and practice. (See http://www.tuxedo.org/~esr/faqs/loginataka.html) In short, I'd like to see the club get bigger... regards, tom lane
Tom Lane wrote: > > > Let's wait for 7.0 beta - "big tuples" seems as work for 2 weeks... > > Thing is, if Vadim could do it in two weeks (sounds about right), then > maybe I could do it in three or four (I'd have to spend time studying > parts of the backend that Vadim already knows, but I don't). It seems > to me that some aspiring hacker who's already a little bit familiar > with backend coding could do it in a month or two, with suitable study, > and would in the process make great strides towards gurudom. This is > a fairly localized task, if I'm not greatly mistaken about it. And ^^^^^^^^^^^^^^ I'm not sure. Seems that we'll have to change heap_getattr: if a column crosses page boundary then we'll have to re-construct it in memory and pfree it after using... > there's plenty of time left before 7.0. So this seems like a perfect > project for someone who wants to learn more about the backend and has > some time to spend doing so. And we always ready to help... > A year ago I didn't know a darn thing about the backend, so I'm a bit > bemused to find myself being called a member of "the small club". > Programming skills don't come out of nowhere, they come out of study > and practice. (See http://www.tuxedo.org/~esr/faqs/loginataka.html) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:))) Vadim
Vadim Mikheev <vadim@krs.ru> writes:
>> a fairly localized task, if I'm not greatly mistaken about it. And
> ^^^^^^^^^^^^^^
> I'm not sure. Seems that we'll have to change heap_getattr:
> if a column crosses page boundary then we'll have to re-construct
> it in memory and pfree it after using...
I was thinking more along the lines of reconstructing the whole tuple
in palloc'd memory and leaving heap_getattr as-is. Otherwise we have
problems with the system relations whose tuples are accessed as C
structs. You'd need to somehow guarantee that those tuples are never
split if you do it as above.
Of course, that just moves the palloc/pfree bookkeeping problem down
a level; it's still going to be tricky to avoid storage leaks.
We might be able to get some win from storing reassembled tuples in
TupleTableSlots, though.
>> there's plenty of time left before 7.0. So this seems like a perfect
>> project for someone who wants to learn more about the backend and has
>> some time to spend doing so.
> And we always ready to help...
Right. Questions can be answered.
regards, tom lane
On Wed, Oct 20, 1999 at 11:38:00AM -0400, Tom Lane wrote: > ecpg's lexer needs the same fixes recently applied to parser/scan.l to > eliminate a fixed-size literal buffer and allow flex's input buffer to I see. > grow as needed. Michael Meskes usually handles ecpg --- Michael, do > you want to deal with this issue or shall I take a cut at it? I'm currently very busy, so I guess it will take some some until I can tackle this problem. So if you have some spare time left I have no problem if you make these changes. Otherwise I will once I find time. Michael -- Michael Meskes | Go SF 49ers! Th.-Heuss-Str. 61, D-41812 Erkelenz | Go Rhein Fire! Tel.: (+49) 2431/72651 | Use Debian GNU/Linux! Email: Michael@Fam-Meskes.De | Use PostgreSQL!