Re: Patch to add hook to copydir()

Thank you both for the feedback. To give some background
on what I am trying to do. Mostly experimental work right now.

I've built an extension that takes advantage of the copy-on-write
properties of ZFS and BTRFS. A tool transforms a data directory
into a set of datasets/sub volumes. When a new database is
created and the directories are copied, the extension uses the
filesystem's ability to create a snapshot/clone of the source dataset/sub
volume.

As createdb() iterates over the template database's tablespaces
and copies the directories to the new one, the copydir() gets invoked
(from dbcommands.c). My extension hooks copydir() and instead
snapshots the source dir and clones it into the target dir (respecting the
recurse flag).

If the source dir is not a ZFS data set or BTRFS sub volume, the
standard implementation gets invoked and everything is business
as usual.

As described in the first paragraph, I've built a small tool that transforms
all base/* directories in the data directory into ZFS datasets or BTRFS
sub volumes. New tablespaces that get created later would have to be
ZFS datasets or BTRFS sub volumes as well in order for the extension
to work. As explained above, if they are not, the extension will just copy
the files.

I also hook ProcessUtility in order to clean up the ZFS datasets and
BTRFS subvolumes _after_ a database gets deleted. Postgres just
handles the deletion of the files and the extension cleans up any
dangling ZFS/BTRFS resources.

Is there anything that I am missing? My early experiments have been
very promising but my experience with Postgres internals is limited. Any
help or feedback would be much appreciated.

On Mon, Sep 2, 2019 at 11:06 PM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
> On 2019-Sep-02, Peter Eisentraut wrote:
>
> > On 2019-09-02 20:54, Swen Kooij wrote:
> > > I've been working on an extension that tightly integrates
> > > postgres with underlying filesystem . I need to customize
> > > how postgres copies directories for new databases.
> >
> > Could you share some more details, so we can assess whether that is a
> > sensible way to go about it, and what other hooks might be needed?
>
> It seems either terribly high-level, or terribly low-level, depending on
> how you look at it.  I wonder to what extent it conflicts with the table
> AM work, and with the idea of allowing FDWs to have real relfilenodes.
> I wonder if this is just one missing piece that's needed to complete
> something of a layer for which other pieces are already satisfied by
> other hooks.
>
> As is and pending further explanation, it seems a bad idea to me.  If we
> want pluggability here, maybe we need some new abstraction layer.
>
> --
> Álvaro Herrera                https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services