Обсуждение: Updating the intro for packages - improve usability, reduce new user confusion

TL;DR: It's time to update the docs to make package-based and
installer-based PostgreSQL the assumed environment our users are
working with. Or at least put it on an equal footing as a first-class
citizen, not relegate it to the dark corners of notes and appendices.
The same is true for client drivers.

I'll put my hand up to start the work, with a gradual series of small
incremental patches, but only if I can get broad agreement here that
it's the right thing to do first. Otherwise it's just wasting time.

Why?

I see a lot of confusion on Stack Overflow, chat groups, forums etc
that all stems from a common cause:

The PostgreSQL manual, including the intro/tutorial, is written with
the assumption you installed from source and are using pg_ctl, etc
directly. Very few of our users do this.

The reality is that most users are immediately faced with:

    psql: FATAL:  role "myname" does not exist

as their first PostgreSQL experience, or

psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

I think it's time to admit that most users first experience PostgreSQL
via packages or installers, and moreover that we should encourage
that. Joe "Never Used Linux Before" Bloggs and Julie "Is Terminal
where I put the commands?" Smith should not be dealing with configure,
make, macports, etc.

We shouldn't leave them in the dark to figure out they want "systemctl
enable --now postgresql-11.service" and that they need to
"/usr/pgsql-10/bin/postgresql-10-setup" first. Or are they on Debian?
Oh, then they need `pg_ctlcluster`, and the cluster is autocreated. We
let the poor confused user sort this mess out.

We, PGDG, ship these packages. We shouldn't entirely ignore them in
our own documentation. That sucks for users. PostgreSQL has a
reputation as a great DBMS that's hard to get started with. That
reputation is deserved. Some trivial changes can ease that learning
curve a lot.

So of course we get people installing Pg from packages, then
initdb'ing and pg_ctl'ing their own clusters instead of using
pg_createcluster or working with systemctl. Sigh. Then they complain
that postgres "crashed" when they restarted, because of course they
didn't integrate it with their init system. How would they? We didn't
tell them how, or that they had to. It's such a mess and I have better
things to do than deal with these unnecessary horrors when I work with
clients on real problems.

Look at what some popular systems do. Slick docs that get any idiot
straight to running the system, even if perhaps they shouldn't. I
don't want to go that far, with the sort of

   curl https://get.postgresql.org/11 | sudo bash

horror that I see many places. But we can do better than we do now
without encouraging hideously insecure and unmaintainable deployments.

(It's bad enough that we nearly totally ignore the client drivers
everyone actually uses - come on, how many of you write direct libpq
code on a day to day basis? Good luck finding PgJDBC, nPgSQL,
psycopg2, DBD::Pg or the Pg gem in our manual; hell, they most don't
even appear to be mentioned in the SOURCE TREE anywhere. psqlODBC is
mentioned - once - in "external projects".).

So I propose to restructure our intro docs to emphasise common package
based deployments, with a focus on the entrypoints at:

* https://www.postgresql.org/docs/current/tutorial-start.html
* https://www.postgresql.org/docs/current/tutorial-install.html
* https://www.postgresql.org/docs/current/installation.html

I realise we cannot cover everything, but we should:

* Move "installation from source code" from "installation.html" to a subhead
* Instead have the "installation" page list:

  - Installation from Packages on Linux
    - Fedora, RHEL, CentOS (mention PGDG and distro packages)
    - Debian and Ubuntu (mention PGDG and distro packages)
    - Other options (-> www.postgresql.org download page)
  - Installation with an installer program
    - (-> download page)
   - Installation from Ports on *BSD
   - Installation from source distribution
     - git clone (lets face it, if you use source you use git now)
     - tarball

Keep a section of the manual dedicated to common distribution and
install methods, where we can collect and maintain this info so it's
not scattered. Cross-reference it from elsewhere. So for example under

https://www.postgresql.org/docs/current/server-start.html

we'd xref the distro/OS specific part, and one of those dist/OS
specific parts would cover source installs. Right now when you read
the docs, it's all about the source install, with distro/packaged
stuff a half-considered afterthought mentioned after the manual stuff.
A user should be able to easily open the "Fedora/RHEL/CentOS" section
to get taken to the "Starting PostgreSQL on RHEL, CentOS or Fedora"
section of the distro-specific info in the manual. With, of course, a
note there saying that this applies if you installed from packages,
and xrefs for manual source installs and installer-binary installs.

We should have prominent links in the tutorial install section that link to:

* Linux: apt.postgresql.org and yum.postgresql.org
* the postgresql download page
* https://help.ubuntu.com/community/PostgreSQL for package-related
info on the Debian and Ubuntu variants
* https://fedoraproject.org/wiki/PostgreSQL for Fedora/RHEL/CentOS
package-related info

These are long lived, stable, well known links, and we're long past
the days where someone's reading the docs on a printout. Hell, most
people don't even realise we have docs packages rather than just the
website!

We should also have a short, simple "concepts" section in the tutorial
(or tweak the architecture fundamentals section) that explains a few
key points of frequent confusion:

* Explicitly mention that on most package installs you use `sudo -u
postgres psql`  to get a "postgres" superuser shell. Give the Mac and
Windows equivalents too.
* Explain that PostgreSQL is a client/server system. The psql client
connects to a server, either on the same computer or elsewhere via a
network. The server is "postgres", the usual client is "psql" but
there are others.
* Explain that whenever we write "postgres=#" we mean "run this as a
postgres superuser"
* The "psql" shell is a separate program to the "sh" or "bash" shell.
It takes SQL commands and special backslash-commands. You cannot run
SQL on the system shell, and you cannot run system commands on the
psql shell.
* There are GUIs like PgAdmin4, OmniDB, HeidiSQL, DBeaver, etc that
some people will prefer. These tools may NOT support the psql
backslash commands the user manual describes. The manual is written
with the assumption that you use psql. In particular \COPY will NOT
work on many of them.
* The "createuser", "createdb" etc commands are command-line ("Command
Prompt", Terminal, etc) commands, NOT psql commands. They have sql
equivalents like CREATE USER, CREATE DATABASE, etc.
* Give a simple recipe for how to make your own user account with your
own db and set up peer auth for it, so you can get started simply,
like

sudo -u postgres psql -c "CREATE USER ${USER} WITH CREATEDB
CREATEROLE; CREATE DATABASE $user WITH OWNER ${USER};"

It's a shame we don't have an ALTER SYSTEM equivalent for pg_hba.conf,
but most packages ship with 'peer' enabled, so it should Just Work.


Anyway. I know that was long, but I wanted to actually explain WHAT I
want to change in the docs and HOW, not just "lets change some stuff
ok?".

I'll have extremely limited time to do this, but I hope to deliver a
stream of small and simple patches to do it piece-by-piece.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

чт, 29 нояб. 2018 г. в 07:35, Craig Ringer <craig@2ndquadrant.com>:
>
> TL;DR: It's time to update the docs to make package-based and
> installer-based PostgreSQL the assumed environment our users are
> working with. Or at least put it on an equal footing as a first-class
> citizen, not relegate it to the dark corners of notes and appendices.
> The same is true for client drivers.
Could you please add the reference to the Pgfe client library to the
documentation?
The patch is provided.

чт, 29 нояб. 2018 г. в 12:25, Dmitry Igrishin <dmitigr@gmail.com>:
>
> чт, 29 нояб. 2018 г. в 07:35, Craig Ringer <craig@2ndquadrant.com>:
> >
> > TL;DR: It's time to update the docs to make package-based and
> > installer-based PostgreSQL the assumed environment our users are
> > working with. Or at least put it on an equal footing as a first-class
> > citizen, not relegate it to the dark corners of notes and appendices.
> > The same is true for client drivers.
> Could you please add the reference to the Pgfe client library to the
> documentation?
> The patch is provided.

Re: Dmitry Igrishin 2018-11-29 <CAAfz9KMcq418B55Hyv2d97Ur+6dhSo_N_u=Dx93d3bMkOPei9A@mail.gmail.com>
> > Could you please add the reference to the Pgfe client library to the
> > documentation?
> > The patch is provided.

> diff --git a/doc/src/sgml/external-projects.sgml b/doc/src/sgml/external-projects.sgml
> index 347b830..c74e391 100644
> --- a/doc/src/sgml/external-projects.sgml
> +++ b/doc/src/sgml/external-projects.sgml
> @@ -83,6 +83,13 @@
>       </row>
>  
>       <row>
> +      <entry>Pgfe</entry>
> +      <entry>C++</entry>
> +      <entry>An API inspired by libpq</entry>
> +      <entry><ulink url="https://github.com/dmitigr/pgfe"></ulink></entry>

Has that been around long enough, and is widely used, so it qualifies
for mentioning here? I've never heard of it, and it's not packaged for
Debian.

Christoph

чт, 29 нояб. 2018 г. в 12:36, Christoph Berg <myon@debian.org>:
>
> Re: Dmitry Igrishin 2018-11-29 <CAAfz9KMcq418B55Hyv2d97Ur+6dhSo_N_u=Dx93d3bMkOPei9A@mail.gmail.com>
> > > Could you please add the reference to the Pgfe client library to the
> > > documentation?
> > > The patch is provided.
>
> > diff --git a/doc/src/sgml/external-projects.sgml b/doc/src/sgml/external-projects.sgml
> > index 347b830..c74e391 100644
> > --- a/doc/src/sgml/external-projects.sgml
> > +++ b/doc/src/sgml/external-projects.sgml
> > @@ -83,6 +83,13 @@
> >       </row>
> >
> >       <row>
> > +      <entry>Pgfe</entry>
> > +      <entry>C++</entry>
> > +      <entry>An API inspired by libpq</entry>
> > +      <entry><ulink url="https://github.com/dmitigr/pgfe"></ulink></entry>
>
> Has that been around long enough, and is widely used, so it qualifies
> for mentioning here? I've never heard of it, and it's not packaged for
> Debian.
I've released Pgfe in May 2018. I don't know how widely it's used. I
think it should be just mentioned in the documentation to let the
users decide to use it or to not. It's open source and anyone can
contribute or file a bug or feature request. So, I don't see problem
here.

Could you help me please to package it for Debian? I'll happy to do it!

> On Nov 29, 2018, at 9:36 AM, Christoph Berg <myon@debian.org> wrote:
>
> Re: Dmitry Igrishin 2018-11-29 <CAAfz9KMcq418B55Hyv2d97Ur+6dhSo_N_u=Dx93d3bMkOPei9A@mail.gmail.com>
>>> Could you please add the reference to the Pgfe client library to the
>>> documentation?
>>> The patch is provided.
>
>> diff --git a/doc/src/sgml/external-projects.sgml b/doc/src/sgml/external-projects.sgml
>> index 347b830..c74e391 100644
>> --- a/doc/src/sgml/external-projects.sgml
>> +++ b/doc/src/sgml/external-projects.sgml
>> @@ -83,6 +83,13 @@
>>      </row>
>>
>>      <row>
>> +      <entry>Pgfe</entry>
>> +      <entry>C++</entry>
>> +      <entry>An API inspired by libpq</entry>
>> +      <entry><ulink url="https://github.com/dmitigr/pgfe"></ulink></entry>
>
> Has that been around long enough, and is widely used, so it qualifies
> for mentioning here? I've never heard of it, and it's not packaged for
> Debian.

No. It's been discussed at least once before that I've noticed, and IIRC
the conclusion was "add it to the wiki", which I did in May.

https://wiki.postgresql.org/wiki/Client_Libraries

Cheers,
  Steve

> On Nov 29, 2018, at 10:17 AM, Steve Atkins <steve@blighty.com> wrote:
>
>
>
>> On Nov 29, 2018, at 9:36 AM, Christoph Berg <myon@debian.org> wrote:
>>
>> Re: Dmitry Igrishin 2018-11-29 <CAAfz9KMcq418B55Hyv2d97Ur+6dhSo_N_u=Dx93d3bMkOPei9A@mail.gmail.com>
>>>> Could you please add the reference to the Pgfe client library to the
>>>> documentation?
>>>> The patch is provided.
>>
>>> diff --git a/doc/src/sgml/external-projects.sgml b/doc/src/sgml/external-projects.sgml
>>> index 347b830..c74e391 100644
>>> --- a/doc/src/sgml/external-projects.sgml
>>> +++ b/doc/src/sgml/external-projects.sgml
>>> @@ -83,6 +83,13 @@
>>>     </row>
>>>
>>>     <row>
>>> +      <entry>Pgfe</entry>
>>> +      <entry>C++</entry>
>>> +      <entry>An API inspired by libpq</entry>
>>> +      <entry><ulink url="https://github.com/dmitigr/pgfe"></ulink></entry>
>>
>> Has that been around long enough, and is widely used, so it qualifies
>> for mentioning here? I've never heard of it, and it's not packaged for
>> Debian.
>
> No. It's been discussed at least once before that I've noticed, and IIRC
> the conclusion was "add it to the wiki", which I did in May.
>
> https://wiki.postgresql.org/wiki/Client_Libraries

And while it looks like it has the potential to become a decent enough library it's
own documentation says that it's beta-quality, has an unstable API and isn't
ready to be used in production.

Cheers,
  Steve

чт, 29 нояб. 2018 г. в 13:28, Steve Atkins <steve@blighty.com>:
>
>
>
> > On Nov 29, 2018, at 10:17 AM, Steve Atkins <steve@blighty.com> wrote:
> >
> >
> >
> >> On Nov 29, 2018, at 9:36 AM, Christoph Berg <myon@debian.org> wrote:
> >>
> >> Re: Dmitry Igrishin 2018-11-29 <CAAfz9KMcq418B55Hyv2d97Ur+6dhSo_N_u=Dx93d3bMkOPei9A@mail.gmail.com>
> >>>> Could you please add the reference to the Pgfe client library to the
> >>>> documentation?
> >>>> The patch is provided.
> >>
> >>> diff --git a/doc/src/sgml/external-projects.sgml b/doc/src/sgml/external-projects.sgml
> >>> index 347b830..c74e391 100644
> >>> --- a/doc/src/sgml/external-projects.sgml
> >>> +++ b/doc/src/sgml/external-projects.sgml
> >>> @@ -83,6 +83,13 @@
> >>>     </row>
> >>>
> >>>     <row>
> >>> +      <entry>Pgfe</entry>
> >>> +      <entry>C++</entry>
> >>> +      <entry>An API inspired by libpq</entry>
> >>> +      <entry><ulink url="https://github.com/dmitigr/pgfe"></ulink></entry>
> >>
> >> Has that been around long enough, and is widely used, so it qualifies
> >> for mentioning here? I've never heard of it, and it's not packaged for
> >> Debian.
> >
> > No. It's been discussed at least once before that I've noticed, and IIRC
> > the conclusion was "add it to the wiki", which I did in May.
> >
> > https://wiki.postgresql.org/wiki/Client_Libraries
>
> And while it looks like it has the potential to become a decent enough library it's
> own documentation says that it's beta-quality, has an unstable API and isn't
> ready to be used in production.
All these problems are problems of Pgfe, rather than PostgreSQL.
Anyway, let the up to the user to decide what to use and what not. He
or she knows it better.
The problem of PostgreSQL, as this topic suggest, that things related
to PostgreSQL (including client interfaces) "relegates in the dark
corners of notes and appendices". But sometimes the situation is even
worse, because that related things is in the dark corners of Internet
and not even mentioned in the documentation at all!

On Thu, 29 Nov 2018 at 17:23, Dmitry Igrishin <dmitigr@gmail.com> wrote:
>
> чт, 29 нояб. 2018 г. в 07:35, Craig Ringer <craig@2ndquadrant.com>:
> >
> > TL;DR: It's time to update the docs to make package-based and
> > installer-based PostgreSQL the assumed environment our users are
> > working with. Or at least put it on an equal footing as a first-class
> > citizen, not relegate it to the dark corners of notes and appendices.
> > The same is true for client drivers.
> Could you please add the reference to the Pgfe client library to the
> documentation?
> The patch is provided.

Please don't hijack the thread. I don't want to devolve into a big
argument about what does and does not merit explicit listing. I can't
imagine anyone here objecting to our official, project-affiliated
client drivers PgJDBC and psqlODBC, and if that's where the line is,
whatever, that's better than some massive time wasting argument.

Can we focus on the general issue I raised, not promoting random tools please?

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Dmitry Igrishin 2018-11-29 <CAAfz9KMoGZiCBiYf1GDDnjgNpCHbYMH3aA9ZY4+Z=sFSkLyCSw@mail.gmail.com>
> Could you help me please to package it for Debian? I'll happy to do it!

The starting point would be https://www.debian.org/doc/manuals/maint-guide/

If you have any questions, I'd be glad to help.

Christoph

Hello, Craig.

You wrote:

CR> TL;DR: It's time to update the docs to make package-based and
CR> installer-based PostgreSQL the assumed environment our users are
CR> working with. Or at least put it on an equal footing as a first-class
CR> citizen, not relegate it to the dark corners of notes and appendices.
CR> The same is true for client drivers.

CR> I'll put my hand up to start the work, with a gradual series of small
CR> incremental patches, but only if I can get broad agreement here that
CR> it's the right thing to do first. Otherwise it's just wasting time.

I like your proposal. You have my +1.

CR> Why?

CR> I see a lot of confusion on Stack Overflow, chat groups, forums etc
CR> that all stems from a common cause:

CR> The PostgreSQL manual, including the intro/tutorial, is written with
CR> the assumption you installed from source and are using pg_ctl, etc
CR> directly. Very few of our users do this.

CR> The reality is that most users are immediately faced with:

CR>     psql: FATAL:  role "myname" does not exist

CR> as their first PostgreSQL experience, or

CR> psql: could not connect to server: No such file or directory
CR> Is the server running locally and accepting
CR> connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

CR> I think it's time to admit that most users first experience PostgreSQL
CR> via packages or installers, and moreover that we should encourage
CR> that. Joe "Never Used Linux Before" Bloggs and Julie "Is Terminal
CR> where I put the commands?" Smith should not be dealing with configure,
CR> make, macports, etc.

CR> We shouldn't leave them in the dark to figure out they want "systemctl
CR> enable --now postgresql-11.service" and that they need to
CR> "/usr/pgsql-10/bin/postgresql-10-setup" first. Or are they on Debian?
CR> Oh, then they need `pg_ctlcluster`, and the cluster is autocreated. We
CR> let the poor confused user sort this mess out.

CR> We, PGDG, ship these packages. We shouldn't entirely ignore them in
CR> our own documentation. That sucks for users. PostgreSQL has a
CR> reputation as a great DBMS that's hard to get started with. That
CR> reputation is deserved. Some trivial changes can ease that learning
CR> curve a lot.

CR> So of course we get people installing Pg from packages, then
CR> initdb'ing and pg_ctl'ing their own clusters instead of using
CR> pg_createcluster or working with systemctl. Sigh. Then they complain
CR> that postgres "crashed" when they restarted, because of course they
CR> didn't integrate it with their init system. How would they? We didn't
CR> tell them how, or that they had to. It's such a mess and I have better
CR> things to do than deal with these unnecessary horrors when I work with
CR> clients on real problems.

CR> Look at what some popular systems do. Slick docs that get any idiot
CR> straight to running the system, even if perhaps they shouldn't. I
CR> don't want to go that far, with the sort of

CR>    curl https://get.postgresql.org/11 | sudo bash

CR> horror that I see many places. But we can do better than we do now
CR> without encouraging hideously insecure and unmaintainable deployments.

CR> (It's bad enough that we nearly totally ignore the client drivers
CR> everyone actually uses - come on, how many of you write direct libpq
CR> code on a day to day basis? Good luck finding PgJDBC, nPgSQL,
CR> psycopg2, DBD::Pg or the Pg gem in our manual; hell, they most don't
CR> even appear to be mentioned in the SOURCE TREE anywhere. psqlODBC is
CR> mentioned - once - in "external projects".).

CR> So I propose to restructure our intro docs to emphasise common package
CR> based deployments, with a focus on the entrypoints at:

CR> * https://www.postgresql.org/docs/current/tutorial-start.html
CR> * https://www.postgresql.org/docs/current/tutorial-install.html
CR> * https://www.postgresql.org/docs/current/installation.html

CR> I realise we cannot cover everything, but we should:

CR> * Move "installation from source code" from "installation.html" to a subhead
CR> * Instead have the "installation" page list:

CR>   - Installation from Packages on Linux
CR>     - Fedora, RHEL, CentOS (mention PGDG and distro packages)
CR>     - Debian and Ubuntu (mention PGDG and distro packages)
CR>     - Other options (-> www.postgresql.org download page)
CR>   - Installation with an installer program
CR>     - (-> download page)
CR>    - Installation from Ports on *BSD
CR>    - Installation from source distribution
CR>      - git clone (lets face it, if you use source you use git now)
CR>      - tarball

CR> Keep a section of the manual dedicated to common distribution and
CR> install methods, where we can collect and maintain this info so it's
CR> not scattered. Cross-reference it from elsewhere. So for example under

CR> https://www.postgresql.org/docs/current/server-start.html

CR> we'd xref the distro/OS specific part, and one of those dist/OS
CR> specific parts would cover source installs. Right now when you read
CR> the docs, it's all about the source install, with distro/packaged
CR> stuff a half-considered afterthought mentioned after the manual stuff.
CR> A user should be able to easily open the "Fedora/RHEL/CentOS" section
CR> to get taken to the "Starting PostgreSQL on RHEL, CentOS or Fedora"
CR> section of the distro-specific info in the manual. With, of course, a
CR> note there saying that this applies if you installed from packages,
CR> and xrefs for manual source installs and installer-binary installs.

CR> We should have prominent links in the tutorial install section that link to:

CR> * Linux: apt.postgresql.org and yum.postgresql.org
CR> * the postgresql download page
CR> * https://help.ubuntu.com/community/PostgreSQL for package-related
CR> info on the Debian and Ubuntu variants
CR> * https://fedoraproject.org/wiki/PostgreSQL for Fedora/RHEL/CentOS
CR> package-related info

CR> These are long lived, stable, well known links, and we're long past
CR> the days where someone's reading the docs on a printout. Hell, most
CR> people don't even realise we have docs packages rather than just the
CR> website!

CR> We should also have a short, simple "concepts" section in the tutorial
CR> (or tweak the architecture fundamentals section) that explains a few
CR> key points of frequent confusion:

CR> * Explicitly mention that on most package installs you use `sudo -u
CR> postgres psql`  to get a "postgres" superuser shell. Give the Mac and
CR> Windows equivalents too.
CR> * Explain that PostgreSQL is a client/server system. The psql client
CR> connects to a server, either on the same computer or elsewhere via a
CR> network. The server is "postgres", the usual client is "psql" but
CR> there are others.
CR> * Explain that whenever we write "postgres=#" we mean "run this as a
CR> postgres superuser"
CR> * The "psql" shell is a separate program to the "sh" or "bash" shell.
CR> It takes SQL commands and special backslash-commands. You cannot run
CR> SQL on the system shell, and you cannot run system commands on the
CR> psql shell.
CR> * There are GUIs like PgAdmin4, OmniDB, HeidiSQL, DBeaver, etc that
CR> some people will prefer. These tools may NOT support the psql
CR> backslash commands the user manual describes. The manual is written
CR> with the assumption that you use psql. In particular \COPY will NOT
CR> work on many of them.
CR> * The "createuser", "createdb" etc commands are command-line ("Command
CR> Prompt", Terminal, etc) commands, NOT psql commands. They have sql
CR> equivalents like CREATE USER, CREATE DATABASE, etc.
CR> * Give a simple recipe for how to make your own user account with your
CR> own db and set up peer auth for it, so you can get started simply,
CR> like

CR> sudo -u postgres psql -c "CREATE USER ${USER} WITH CREATEDB
CR> CREATEROLE; CREATE DATABASE $user WITH OWNER ${USER};"

CR> It's a shame we don't have an ALTER SYSTEM equivalent for pg_hba.conf,
CR> but most packages ship with 'peer' enabled, so it should Just Work.


CR> Anyway. I know that was long, but I wanted to actually explain WHAT I
CR> want to change in the docs and HOW, not just "lets change some stuff
CR> ok?".

CR> I'll have extremely limited time to do this, but I hope to deliver a
CR> stream of small and simple patches to do it piece-by-piece.




-- 
With best wishes,
 Pavel                          mailto:pavel@gf.microolap.com