Обсуждение: Large scale reliable software system

Поиск
Список
Период
Сортировка

Large scale reliable software system

От
B M
Дата:

Dear all,

After greeting,

I taught PostgreSQL myself and developed a small scale experimental software system using PostgreSQL in the back-end.

I would like to know your advices to develop a large scale reliable software system using PostgreSQL in the back-end, through which i can share the storage with the different system users where they login to the system through the web application front-end with different passwords and usernames , save the privacy of each user data, improve overall system security and performance, achieve fast response, make backups and save the stored data from loss. The system will be hosted on a cloud.

Thank you in advance.

Re: Large scale reliable software system

От
Adrian Klaver
Дата:
On 6/26/23 16:48, B M wrote:
> Dear all,
> 
> After greeting,
> 
> I taught PostgreSQL myself and developed a small scale 
> experimentalsoftware system using PostgreSQL in the back-end.
> 
> I would like to know your advices to develop a large scale reliable 
> software system using PostgreSQL in the back-end, through which i can 
> share the storage with the different system users where they login to 
> the system through the web application front-end with different 
> passwords and usernames , save the privacy of each user data, improve 
> overall system security and performance, achieve fast response, make 
> backups and save the stored data from loss. The system will be hosted on 
> a cloud.

https://www.djangoproject.com/

> 
> Thank you in advance.
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com




Re: Large scale reliable software system

От
Guyren Howe
Дата:
This is a reasonable answer, but I want to offer a caveat.

Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.

This heuristic is nearly always mostly wrong.

Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:
On 6/26/23 16:48, B M wrote:
Dear all,

After greeting,

I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.

I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.

https://www.djangoproject.com/


Thank you in advance.


--
Adrian Klaver
adrian.klaver@aklaver.com



Re: Large scale reliable software system

От
Michael Nolan
Дата:
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.

Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand.  :-)

On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
>
> This is a reasonable answer, but I want to offer a caveat.
>
> Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development
communitythat the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby
orPython or whatever back end code. 
>
> This heuristic is nearly always mostly wrong.
>
> Guyren G Howe
> On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:
>
> On 6/26/23 16:48, B M wrote:
>
> Dear all,
>
> After greeting,
>
> I taught PostgreSQL myself and developed a small scale
> experimentalsoftware system using PostgreSQL in the back-end.
>
> I would like to know your advices to develop a large scale reliable
> software system using PostgreSQL in the back-end, through which i can
> share the storage with the different system users where they login to
> the system through the web application front-end with different
> passwords and usernames , save the privacy of each user data, improve
> overall system security and performance, achieve fast response, make
> backups and save the stored data from loss. The system will be hosted on
> a cloud.
>
>
> https://www.djangoproject.com/
>
>
> Thank you in advance.
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
>
>



Re: Large scale reliable software system

От
Guyren Howe
Дата:
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.

On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:

It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.

Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand.  :-)

On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:

This is a reasonable answer, but I want to offer a caveat.

Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.

This heuristic is nearly always mostly wrong.

Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:

On 6/26/23 16:48, B M wrote:

Dear all,

After greeting,

I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.

I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.


https://www.djangoproject.com/


Thank you in advance.


--
Adrian Klaver
adrian.klaver@aklaver.com




Re: Large scale reliable software system

От
Saurabh Agrawal
Дата:
>  Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.

So, what's your advice, and is there some book / resource to get upto speed?

Thanks!

On Tue, Jun 27, 2023 at 10:38 AM Guyren Howe <guyren@gmail.com> wrote:
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.

On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:

It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.

Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand.  :-)

On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:

This is a reasonable answer, but I want to offer a caveat.

Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.

This heuristic is nearly always mostly wrong.

Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:

On 6/26/23 16:48, B M wrote:

Dear all,

After greeting,

I taught PostgreSQL myself and developed a small scale
experimentalsoftware system using PostgreSQL in the back-end.

I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can
share the storage with the different system users where they login to
the system through the web application front-end with different
passwords and usernames , save the privacy of each user data, improve
overall system security and performance, achieve fast response, make
backups and save the stored data from loss. The system will be hosted on
a cloud.


https://www.djangoproject.com/


Thank you in advance.


--
Adrian Klaver
adrian.klaver@aklaver.com




Re: Large scale reliable software system

От
Tony Shelver
Дата:

On Tue, 27 Jun 2023 at 07:08, Guyren Howe <guyren@gmail.com> wrote:
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.

On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:

It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.

Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand.  :-)

On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:

This is a reasonable answer, but I want to offer a caveat.

Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.

This heuristic is nearly always mostly wrong.

Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:

On 6/26/23 16:48, B M wrote:



--
Adrian Klaver
adrian.klaver@aklaver.com




 The accepted front-end developer wisdom of treating the DB as a dumb data store works under conditions, for example the DB will never be accessed from a different ORM / framework, and where the performance attributes of using an ORM with 'standard' datastructures are acceptable.

The moment you need to plug in something like reporting tools, or access from a different system / API / framework / language / ORM or whatever, the approach not having rules / views / procedures / whatever built into the database falls apart.

Other things to consider are performance / load / overhead:  we have one system that involves processing through large amounts of data for reports / queries.  Shipping all that back through the ORM / db interface (ODBC / JDBC / psycopg2 / whatever for resolution / filtering on the front end application where SQL / procedures / views could do that in the DB and just ship back the required data seems counterproductive.

Tony Shelver

Re: Large scale reliable software system

От
Avin Kavish
Дата:
Well, seeing as postgres isn't designed to serve http requests or to run general purpose code that doesn't involve databases, which you can express elegantly in python, to answer OP's question - my vote is on the original answer - Django. It's got everything out of the box - authentication. file storage. etc etc.

Once you get the application running you can enhance the database as necessary.

On Tue, Jun 27, 2023 at 4:19 PM Tony Shelver <tshelver@gmail.com> wrote:

On Tue, 27 Jun 2023 at 07:08, Guyren Howe <guyren@gmail.com> wrote:
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.

On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:

It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.

Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand.  :-)

On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:

This is a reasonable answer, but I want to offer a caveat.

Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.

This heuristic is nearly always mostly wrong.

Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:

On 6/26/23 16:48, B M wrote:



--
Adrian Klaver
adrian.klaver@aklaver.com




 The accepted front-end developer wisdom of treating the DB as a dumb data store works under conditions, for example the DB will never be accessed from a different ORM / framework, and where the performance attributes of using an ORM with 'standard' datastructures are acceptable.

The moment you need to plug in something like reporting tools, or access from a different system / API / framework / language / ORM or whatever, the approach not having rules / views / procedures / whatever built into the database falls apart.

Other things to consider are performance / load / overhead:  we have one system that involves processing through large amounts of data for reports / queries.  Shipping all that back through the ORM / db interface (ODBC / JDBC / psycopg2 / whatever for resolution / filtering on the front end application where SQL / procedures / views could do that in the DB and just ship back the required data seems counterproductive.

Tony Shelver

Re: Large scale reliable software system

От
Merlin Moncure
Дата:


On Mon, Jun 26, 2023 at 6:49 PM B M <bmmasprep@gmail.com> wrote:

Dear all,

After greeting,

I taught PostgreSQL myself and developed a small scale experimental software system using PostgreSQL in the back-end.

I would like to know your advices to develop a large scale reliable software system using PostgreSQL in the back-end, through which i can share the storage with the different system users where they login to the system through the web application front-end with different passwords and usernames , save the privacy of each user data, improve overall system security and performance, achieve fast response, make backups and save the stored data from loss. The system will be hosted on a cloud.

Thank you in advance.


* your sql is code, and treat it as such, check it into git etc
* robust deployment strategy is essential to scaling team
* write a lot of tests
* become intimate with pg_stat_statements
* keep your transactions as short as possible while preserving safety
* avoid developers who advocate for keeping business logic out of the database religiously (controversial)
* try to avoid assumptions that only one technical stack interacts with your database
* do not waste time making ERDs use a tool that generates them (i like schemaspy)
* test your DR strategy before disaster strikes
* think about security model up front

merlin
 

 

Re: Large scale reliable software system

От
Adrian Klaver
Дата:
On 6/27/23 07:58, Merlin Moncure wrote:
> 
> 
> On Mon, Jun 26, 2023 at 6:49 PM B M <bmmasprep@gmail.com 
> <mailto:bmmasprep@gmail.com>> wrote:
> 
>     Dear all,
> 
>     After greeting,
> 
>     I taught PostgreSQL myself and developed a small scale
>     experimentalsoftware system using PostgreSQL in the back-end.
> 
>     I would like to know your advices to develop a large scale reliable
>     software system using PostgreSQL in the back-end, through which i
>     can share the storage with the different system users where they
>     login to the system through the web application front-end with
>     different passwords and usernames , save the privacy of each user
>     data, improve overall system security and performance, achieve fast
>     response, make backups and save the stored data from loss. The
>     system will be hosted on a cloud.
> 
>     Thank you in advance.
> 
> 
> * your sql is code, and treat it as such, check it into git etc
> * robust deployment strategy is essential to scaling team
> * write a lot of tests
> * become intimate with pg_stat_statements
> * keep your transactions as short as possible while preserving safety
> * avoid developers who advocate for keeping business logic out of the 
> database religiously (controversial)
> * try to avoid assumptions that only one technical stack interacts with 
> your database
> * do not waste time making ERDs use a tool that generates them (i like 
> schemaspy)
> * test your DR strategy before disaster strikes
> * think about security model up front


+1

> 
> merlin
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com