Обсуждение: Should we nonblocking open FIFO files in COPY?
Hi, I have an issue that COPY from a FIFO, which has no writers, could not be canceled, because COPY invokes AllocateFile() -> fopen() -> blocking open(). ``` [postgres@s1 ~]$ mkfifo /tmp/test0 [postgres@s1 ~]$ /usr/local/pgsql/bin/psql test psql (11devel) Type "help" for help. test=# create table test(t text); CREATE TABLE test=# copy test from '/tmp/test0'; ^CCancel request sent ^CCancel request sent ^CCancel request sent ^CCancel request sent ^CCancel request sent ^CCancel request sent ... ``` Should we nonblocking open FIFO files? And a following question if we nonblocking open them, say there is a FIFO file, no one will write into it, and a utility calls `COPY FROM` it, should we just return `COPY 0` or wait writers? If we wait, then users have to interrupt or write an EOF into the FIFO after a timeout, I see some utilities do that, gptransfer for instance, just seems not right. My plan is to write a new function which nonblocking opens FIFOs just for COPY, and not waits writers, what do you think? -- Adam Lee
On Thu, Dec 21, 2017 at 10:10 PM, Adam Lee <ali@pivotal.io> wrote: > I have an issue that COPY from a FIFO, which has no writers, could not be > canceled, because COPY invokes AllocateFile() -> fopen() -> blocking open(). Hmm. What about the case where we try to open a plain file that's on an inaccessible filesystem, e.g. due to a disk failure? Allowing cancel to work just for FIFOs would be OK, I guess, but allowing it for other open() calls that hang would be better. I'm not sure if we can make it work that way, but it would be nice if we could. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, Dec 26, 2017 at 11:48:58AM -0800, Robert Haas wrote: > On Thu, Dec 21, 2017 at 10:10 PM, Adam Lee <ali@pivotal.io> wrote: > > I have an issue that COPY from a FIFO, which has no writers, could not be > > canceled, because COPY invokes AllocateFile() -> fopen() -> blocking open(). > > Hmm. What about the case where we try to open a plain file that's on > an inaccessible filesystem, e.g. due to a disk failure? Allowing > cancel to work just for FIFOs would be OK, I guess, but allowing it > for other open() calls that hang would be better. I'm not sure if we > can make it work that way, but it would be nice if we could. That is doable, just stat() and check before open(). -- Adam Lee
On Wed, Dec 27, 2017 at 10:18:03AM +0800, Adam Lee wrote: > On Tue, Dec 26, 2017 at 11:48:58AM -0800, Robert Haas wrote: > > On Thu, Dec 21, 2017 at 10:10 PM, Adam Lee <ali@pivotal.io> wrote: > > > I have an issue that COPY from a FIFO, which has no writers, could not be > > > canceled, because COPY invokes AllocateFile() -> fopen() -> blocking open(). > > > > Hmm. What about the case where we try to open a plain file that's on > > an inaccessible filesystem, e.g. due to a disk failure? Allowing > > cancel to work just for FIFOs would be OK, I guess, but allowing it > > for other open() calls that hang would be better. I'm not sure if we > > can make it work that way, but it would be nice if we could. > > That is doable, just stat() and check before open(). I think TOCTOU when I read such things.. The data folder is a trusted environment but any patches doing things like that ought to be careful. -- Michael
Вложения
On Tue, Dec 26, 2017 at 7:51 PM, Michael Paquier <michael.paquier@gmail.com> wrote: >> > Hmm. What about the case where we try to open a plain file that's on >> > an inaccessible filesystem, e.g. due to a disk failure? Allowing >> > cancel to work just for FIFOs would be OK, I guess, but allowing it >> > for other open() calls that hang would be better. I'm not sure if we >> > can make it work that way, but it would be nice if we could. >> >> That is doable, just stat() and check before open(). > > I think TOCTOU when I read such things.. The data folder is a trusted > environment but any patches doing things like that ought to be careful. Yeah. I was more wondering whether an ostensibly non-blocking open() would nevertheless block on an inaccessible file. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 2017-12-26 22:30:08 -0800, Robert Haas wrote: > On Tue, Dec 26, 2017 at 7:51 PM, Michael Paquier > <michael.paquier@gmail.com> wrote: > >> > Hmm. What about the case where we try to open a plain file that's on > >> > an inaccessible filesystem, e.g. due to a disk failure? Allowing > >> > cancel to work just for FIFOs would be OK, I guess, but allowing it > >> > for other open() calls that hang would be better. I'm not sure if we > >> > can make it work that way, but it would be nice if we could. I doubt it's realistic to make pg resilient in case of FS problems like that. Partially because the OS level handling usually is very haphazard and inconsistent, and partially because the amount of work to get there seems quite significant with only a small payoff. > >> That is doable, just stat() and check before open(). > > > > I think TOCTOU when I read such things.. The data folder is a trusted > > environment but any patches doing things like that ought to be careful. > > Yeah. I was more wondering whether an ostensibly non-blocking open() > would nevertheless block on an inaccessible file. It very likely would, depending on the type of error. Or stat() would end up being stuck somewhere in the kernel while it's retrying IO for a lengthy amount of time. Greetings, Andres Freund