Thread: Large objects.
Hey all,<br /><br />Here is simple test case of LOB usage, please, note the comments:<br /><br />#include
<libpq-fe.h><br/>#include <libpq/libpq-fs.h><br /><br />int main(int argc, char* argv[])<br />{<br />
PGconn*c = PQconnectdb("password=test");<br /><br /> PGresult* r = PQexec(c, "BEGIN");<br /> PQclear(r);<br /><br />
constunsigned int id = lo_create(c, 0);<br /><br /> int fd1 = lo_open(c, id, INV_READ | INV_WRITE);<br /> int nBytes
=lo_write(c, fd1, "D", 1);<br /> int fd1Pos = lo_lseek(c, fd1, 2147483647, SEEK_SET);<br /> fd1Pos = lo_lseek(c,
fd1,1, SEEK_CUR);<br /> nBytes = lo_write(c, fd1, "Dima", 4); // nBytes == 4 ! Should be 0, IMO.<br
/> // If not, where is my name<br />
// will be written?<br /><br /> r = PQexec(c, "COMMIT");<br />
PQclear(r);<br/><br /> r = PQexec(c, "BEGIN");<br /> PQclear(r);<br /><br /> fd1 = lo_open(c, id, INV_READ |
INV_WRITE);<br/> fd1Pos = lo_lseek(c, fd1, 0, SEEK_END); // fd1Pos == -2147483647 !<br /><br /> char buf[16];<br
/> nBytes = lo_read(c, fd1, buf, 4); // nBytes == 0 ! Correct, IMO.<br /><br /> r = PQexec(c, "COMMIT");<br />
PQclear(r);<br/><br /> return 0;<br />}<br /><br />Tell me please, why lo_write() returns me the number of bytes
"actuallywritten"<br />when current write location is out of 2GB ? IMO, in this case it should returns<br />at least
zero.<br/> lo_read() returns zero in this case, and it is correct, IMO.<br clear="all" /><br />-- <br />Regards,<br
/>Dmitriy<br/><br />
On Fri, Sep 24, 2010 at 9:13 AM, Dmitriy Igrishin <dmitigr@gmail.com> wrote: > Tell me please, why lo_write() returns me the number of bytes "actually > written" > when current write location is out of 2GB ? IMO, in this case it should > returns > at least zero. > lo_read() returns zero in this case, and it is correct, IMO. Hmm, are you sure? If the behavior of lo_read and lo_write is not symmetric, that's probably not good, but I don't see anything obvious in the code to make me think that's the case. Returning 0 for a value >= 2^31 seems problematic unless there is no possibility of a short read (or write). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company
Hey Robert,<br /><br />Yes, I am sure. I've tested it by test case in my original post.<br />Do you can compile and reproduceit please?<br clear="all" /><br />-- <br />// Dmitriy.<br /><br /><br />
On Sun, Sep 26, 2010 at 12:21 PM, Dmitriy Igrishin <dmitigr@gmail.com> wrote: > Yes, I am sure. I've tested it by test case in my original post. > Do you can compile and reproduce it please? I think the reason lo_read is returning 0 is because it's not reading anything. See attached test case, cleaned up a bit from yours and with some error checks added. According to the documentation, the maximum size of a large object is 2 GB, which may be the reason for this behavior. http://www.postgresql.org/docs/9/static/lo-intro.html -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company
Attachment
Robert Haas <robertmhaas@gmail.com> writes:
> According to the documentation, the maximum size of a large object is
> 2 GB, which may be the reason for this behavior.
In principle, since pg_largeobject stores an integer pageno, we could
support large objects of up to LOBLKSIZE * 2^31 bytes = 4TB without any
incompatible change in on-disk format. This'd require converting a lot
of the internal LO access logic to track positions as int64 not int32,
but now that we require platforms to have working int64 that's no big
drawback. The main practical problem is that the existing lo_seek and
lo_tell APIs use int32 positions. I'm not sure if there's any cleaner
way to deal with that than to add "lo_seek64" and "lo_tell64" functions,
and have the existing ones throw error if asked to deal with positions
past 2^31.
In the particular case here, I think that lo_write may actually be
writing past the 2GB boundary, while the coding in lo_read is a bit
different and stops at the 2GB "limit".
regards, tom lane
On Mon, Sep 27, 2010 at 10:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> According to the documentation, the maximum size of a large object is >> 2 GB, which may be the reason for this behavior. > > In principle, since pg_largeobject stores an integer pageno, we could > support large objects of up to LOBLKSIZE * 2^31 bytes = 4TB without any > incompatible change in on-disk format. This'd require converting a lot > of the internal LO access logic to track positions as int64 not int32, > but now that we require platforms to have working int64 that's no big > drawback. The main practical problem is that the existing lo_seek and > lo_tell APIs use int32 positions. I'm not sure if there's any cleaner > way to deal with that than to add "lo_seek64" and "lo_tell64" functions, > and have the existing ones throw error if asked to deal with positions > past 2^31. > > In the particular case here, I think that lo_write may actually be > writing past the 2GB boundary, while the coding in lo_read is a bit > different and stops at the 2GB "limit". Ouch. Letting people write data to where they can't get it back from seems double-plus ungood. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company
Hey Robert, Tom
// Dmitriy.
Tom, thank you for explanation!
-- Ouch. Letting people write data to where they can't get it back from
seems double-plus ungood.
Robert, yes, I agree with you. This is exactly what I wanted to say.
I've implemented a stream class in C++ and this circumstance makes
the code not so clean because I need to take into account the behavior
of lo_write() and 2GB limit.
I've implemented a stream class in C++ and this circumstance makes
the code not so clean because I need to take into account the behavior
of lo_write() and 2GB limit.
// Dmitriy.
On Mon, Sep 27, 2010 at 2:25 PM, Dmitriy Igrishin <dmitigr@gmail.com> wrote: > Hey Robert, Tom > > Tom, thank you for explanation! > >> Ouch. Letting people write data to where they can't get it back from >> seems double-plus ungood. >> > Robert, yes, I agree with you. This is exactly what I wanted to say. > I've implemented a stream class in C++ and this circumstance makes > the code not so clean because I need to take into account the behavior > of lo_write() and 2GB limit. On further examination, it appears we're not doing this. The reason lo_read wasn't returning any data in your earlier example is because you called it after seeking to the end of the object. If you seek to the position where the data was written, it works fine. A fairly plausible argument could be made that we shouldn't allow reading or writing past 2^31-1, but it now appears to me that the behavior is at least self-consistent. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company