Re: Question about where to deploy the business logics for data processing

On 6/9/23 11:36, Nim Li wrote:
> Hello,
> 
> Thank you so so much for all the feedback so far.  :D
> 
> About this comment:
> 
>  > "... an application that requires changing the data model does not 
> seem to be well designed...don't allow model change by the business 
> logic..."
> 
> I work in a science research faculity.  When researchers start a 
> project, they don't necessary get the full picture of what they are 
> hoping to achive (yet they may get some ideas about the starting point 
> that allow them to move forward)  By the time they see 40% percent of 
> what they have done, they may start to have a different thought and move 
> towards a different direction, or in some cases, they may spin it off to 
> something different after a certain period of time  Coming with my Agile 
> Development mindset in the research area, it is common for me to see 
> users changing their requirement and expectation, with the same buckets 
> for the data.  Yes, there is quite a lot of work to keep the researchers 
> happy.  ;-)
> 
> I suppose when there is a specific end-goal to achive for a project, a 
> more specific design can be more feasible based on the goal.  But when 
> the end-goal is not necessary clear, and/or change-able, I am not 
> exactly clear how we may draw a black-and-white line to determine a 
> design is good or not (.. and for how long...)
> 

Seems to me you are looking for a two part set up:

1) A experiment play ground where ideas and processes can be tested out 
in a more free form manner. Some example software I have used or 
experimented with that can fill that role:

Pandas
https://pandas.pydata.org/

Duckdb
https://duckdb.org/

Polars
https://pola-rs.github.io/polars-book/

2) Once something that resembles a solid plan has been developed then 
move to Postgres or not.


-- 
Adrian Klaver
adrian.klaver@aklaver.com