Обсуждение: generating dynamic queries using pl/pgsql
Hi All,
Is it possible to "generate" dynamic queries using pl/pgsql?
I am trying to archive a set of tables: affiliate_event, affiliate_batch, affiliate_batch_details, affiliate_daily_batch, affiliate_daily_batch_details.
The following steps are necessary for each table:
1) select the data from the table based on the date (and other clause), and Move that data to a temporary table
2) Dump the temp table to a file
3) Delete the data from the original table
4) Delete the temporary table.
Is it possible for pl/pgsql to dynamically generate the SQL select queries needed in step 1??
What information/input would it need in that case?
These are the step 1 queries I am using:
create table temp1 as select * from affiliate_batch where tx_dt < '12/31/2003';
..
delete from temp1;
--------------------------------------------------------------------------------------------------------------------------------
create table temp1 as select bd.* from affiliate_batch_details bd join affiliate_batch b on bd.batch_id=b.id where b.tx_dt < '12/31/2003' order by bd.batch_id;
..
delete from temp1;
--------------------------------------------------------------------------------------------------------------------------------
create table temp1 as select adb.* from affiliate_daily_batch adb join affiliate_batch b on adb.monthly_batch_id=b.id where b.tx_dt < '12/31/2003' order by adb.monthly_batch_id;
..
delete from temp1;
---------------------------------------------------------------------------------------------------------------------------------
create table temp1 as select adb.* from affiliate_daily_batch_details adb where exists (select ad.id from affiliate_daily_batch ad join affiliate_batch b on ad.monthly_batch_id=b.id where adb.batch_id=ad.id and b.tx_dt < '12/31/2003' order by ad.monthly_batch_id) order by adb.batch_id;
..
delete from temp1;
---------------------------------------------------------------------------------------------------------------------------------
create table temp1 as select * from affiliate_event where dt<'12/31/2003';
..
delete from temp1;
---------------------------------------------------------------------------------------------------------------------------------
Thanks in advance!
Saranya
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
----- Original Message -----From: "sarlav kumar" <SARLAVK@YAHOO= .COM>To: pgsqlnovice Subject: [NOVICE]= generating dynamic queries using pl/pgsqlDate: Tue, 18 Jan 2005 10:34:= 11 -0800 (PST) Hi All, Is it possible to "generate" dynamic queries using pl/pgsql? I am trying to archive a set of tables: affiliate_event, affiliate_bat= ch, affiliate_batch_details, affiliate_daily_batch, affiliate_daily_batch_d= etails. The following steps are necessary for each table: 1) select the data from the table based on the date (and ot= her clause), and Move that data to a temporary table 2) Dump the temp table to a file 3) Delete the data from the original table 4) Delete the temporary table. Is it possible for pl/pgsql to dynamically generate the SQL = select queries needed in step 1?? What information/input would it need in that case? These are the step 1 queries I am using: create table temp1 as select * from affiliate_ba= tch where tx_dt < '12/31/2003'; .. delete from temp1; ----------------------------------------------------------------------= ---------------------------------------------------------- create table temp1 as select bd.* from affiliate= _batch_details bd join affiliate_batch b on bd.batch_id=3Db.id where = b.tx_dt < '12/31/2003' order by bd.batch_id; .. delete from temp1; ----------------------------------------------------------------------= ---------------------------------------------------------- create table temp1 as select adb.* from affiliat= e_daily_batch adb join affiliate_batch b on adb.monthly_batch_id=3Db.id whe= re b.tx_dt < '12/31/2003' order by adb.monthly_batch_id; .. delete from temp1; ----------------------------------------------------------------------= ----------------------------------------------------------- create table temp1 as select adb.* from affiliat= e_daily_batch_details adb where exists (select ad.id from affiliate_daily_b= atch ad join affiliate_batch b on ad.monthly_batch_id=3Db.id where adb.batc= h_id=3Dad.id and b.tx_dt < '12/31/2003' order by ad.monthly_batch_id) or= der by adb.batch_id; .. delete from temp1; ----------------------------------------------------------------------= ----------------------------------------------------------- create table temp1 as select * from affiliate_ev= ent where dt<'12/31/2003'; .. delete from temp1; ----------------------------------------------------------------------= ----------------------------------------------------------- Thanks in advance! Saranya ---------------- Yes, it's possible. <A href=3D"http://www.postgresql.org/docs/7.4/interactive/plpgsql-stat= ements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN">http://www.postgresql.org/doc= s/7.4/interactive/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN<= /A> The manual has some great information. --=20 ____________________________________________________ Get your free email from <a href=3D"http://www.kittymail.com" target=3D"_bl= ank">http://www.kittymail.com Powered by Outblaze
>Yes, it's possible.
>The manual has some great information.
I read through the manual. But I am still not clear how to actually generate the "queries" dynamically. In my case, some of the "select queries" that I want to dynamically generate have joins and some of the "select queries" are just from one table.
I am new to pl/pgsql. I came up with the following code to start with. This one just has all the queries hardcoded. I want to dynamically generate the "select query part" in the "create table temp1" statement. I am totally lost:(. It would be great if someone can help me!
Also, is it possible to dump a table within pl/pgsql?
create function try1(date,text) returns integer as '
declare
arch_date alias for $1;
filename alias for $2;
arch_date alias for $1;
filename alias for $2;
begin
create table temp1 as select * from affiliate_batch where tx_dt < arch_date;
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
drop table temp1;
create table temp1 as select bd.* from affiliate_batch_details bd join affiliate_batch b on bd.batch_id=b.id where b.tx_dt < arch_date order by bd.batch_id;
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
drop table temp1;
drop table temp1;
create table temp1 as select * from affiliate_daily_batch adb where exists (select b.id from affiliate_batch b where b.tx_dt < arch_date and adb.monthly_batch_id=b.id) order by adb.monthly_batch_id;
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
drop table temp1;
drop table temp1;
create table temp1 as select adb.* from affiliate_daily_batch_details adb join affiliate_daily_batch ad on ad.id=adb.batch_id join affiliate_batch b on ad.monthly_batch_id=b.id where b.tx_dt < arch_date order by ad.monthly_batch_id;
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
drop table temp1;
drop table temp1;
create table temp1 as select * from affiliate_event where dt< arch_date;
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
drop table temp1;
drop table temp1;
create table temp1 as select * from clickthrough where date < arch_date||''23:59:59'';
-- Here, I would like to dump the table "temp1" to 'filename' before deleting it
drop table temp1;
drop table temp1;
return 1;
end;
' language 'plpgsql';
end;
' language 'plpgsql';
Thanks a lot for all the help.
Saranya
Do you Yahoo!?
Yahoo! Search presents - Jib Jab's 'Second Term'
On Jan 20, 2005, at 12:40 PM, sarlav kumar wrote: > >Yes, it's possible. > > >http://www.postgresql.org/docs/7.4/interactive/plpgsql- > statements.html#PLPGSQL->STATEMENTS-EXECUTING-DYN > >The manual has some great information. > > I read through the manual. But I am still not clear how to actually > generate the "queries" dynamically. In my case, some of the > "select queries" that I want to dynamically generate have joins and > some of the "select queries" are just from one table. > The point of section 37.6.4 (see the manual again) is that you can build a query from a combination of text and variables. If you have pieces of your query, you need to build up the full query using the || operator (concatenation). Then you can EXECUTE it. As for "dumping" the table, look at: http://www.postgresql.org/docs/7.4/interactive/sql-copy.html You can use COPY to do this. HTH, Sean
Hi Sean,
Thanks, for the help. I got the dynamic query generation part to work.
The only thing left to do is to get the dump of the temporary table.
When I try to use COPY inside the pl/pgsql function, I get the following error:
COPY temp1 to ''aff.txt'';
WARNING: Error occurred while executing PL/pgSQL function try2
WARNING: line 38 at SQL statement
ERROR: Relative path not allowed for server side COPY command
WARNING: line 38 at SQL statement
ERROR: Relative path not allowed for server side COPY command
Then I dropped the function, and recreated the function with the following command:
COPY temp1 to ''/home/developers/ss2/aff.txt'';
WARNING: Error occurred while executing PL/pgSQL function try2
WARNING: line 38 at SQL statement
ERROR: COPY command, running in backend with effective uid 501, could not open file '/home/developers/ssivakumar/aff.txt' for writing. Errno = No such file or directory (2).
How can I get the copy command to work from within the pl/pgsql?
I even tried using \copy, it still gives me the above error.
Is there anyway to use \! pg_dump command inside pl/pgsql? I tried it, but it does not allow the character "!". Is there a way to escape "!"?
Thanks in advance,
Saranya
>http://www.postgresql.org/docs/7.4/interactive/sql-copy.html
>You can use COPY to do this.
>HTH,
>Sean
Do you Yahoo!?
Yahoo! Search presents - Jib Jab's 'Second Term'
On Jan 21, 2005, at 11:00 AM, sarlav kumar wrote: > Hi Sean, > > Thanks, for the help. I got the dynamic query generation part to work. > The only thing left to do is to get the dump of the temporary table. > > When I try to use COPY inside the pl/pgsql function, I get the > following error: > > COPY temp1 to ''aff.txt''; > > WARNING: Error occurred while executing PL/pgSQL function try2 > WARNING: line 38 at SQL statement > ERROR: Relative path not allowed for server side COPY command > > Then I dropped the function, and recreated the function with the > following command: > > COPY temp1 to ''/home/developers/ss2/aff.txt''; > > WARNING: Error occurred while executing PL/pgSQL function try2 > WARNING: line 38 at SQL statement > ERROR: COPY command, running in backend with effective uid 501, could > not open file '/home/developers/ssivakumar/aff.txt' for writing. > Errno = No such file or directory (2). > How can I get the copy command to work from within the pl/pgsql? > The tricky part about COPY is that it is executed by the SERVER! Therefore, the tables can only be written to somewhere writable by the user running the server process. If, for example, you have a user named postgres, you could set up a directory that is owned by postgres and use that for the dumps. /tmp is another place. Of course, all this has to be done on the SERVER machine; it can't be done locally to a file. I imagine that is the issue, but others can correct me if I am wrong on this. Another option is to COPY to STDOUT and then capture the output. Sean
Hi Sean,
The problem is that I dont have permission to create directories as a postgres user.
If I can get the \copy command or the \! pg_dump command to work, that would be great.
Thanks,
Saranya
Sean Davis <sdavis2@mail.nih.gov> wrote:
On Jan 21, 2005, at 11:00 AM, sarlav kumar wrote:
> Hi Sean,
>
> Thanks, for the help. I got the dynamic query generation part to work.
> The only thing left to do is to get the dump of the temporary table.
>
> When I try to use COPY inside the pl/pgsql function, I get the
> following error:
>
> COPY temp1 to ''aff.txt'';
>
> WARNING: Error occurred while executing PL/pgSQL function try2
> WARNING: line 38 at SQL statement
> ERROR: Relative path not allowed for server side COPY command
>
> Then I dropped the function, and recreated the function with the
> following command:
>
> COPY temp1 to ''/home/developers/ss2/aff.txt'';
>
> WARNING: Error occurred while executing PL/pgSQL f unction try2
> WARNING: line 38 at SQL statement
> ERROR: COPY command, running in backend with effective uid 501, could
> not open file '/home/developers/ssivakumar/aff.txt' for writing.
> Errno = No such file or directory (2).
> How can I get the copy command to work from within the pl/pgsql?
>
The tricky part about COPY is that it is executed by the SERVER!
Therefore, the tables can only be written to somewhere writable by the
user running the server process. If, for example, you have a user
named postgres, you could set up a directory that is owned by postgres
and use that for the dumps. /tmp is another place. Of course, all
this has to be done on the SERVER machine; it can't be done locally to
a file. I imagine that is the issue, but others can correct me if I am
wrong on this.
Another option is to COPY to STDOUT and then capture the output.
Sean
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Saranya, Just my personal suggestion, but I would move on to a scripting language like perl. Once you learn a bit of it, doing stuff like what you are proposing is trivial. Unfortunately, I don't have more suggestions on making psql work for you. If you want to automate the whole process, then you will benefit from learning a scripting language. Others can correct me if they don't share my sentiments. Sorry, Sean On Jan 21, 2005, at 11:44 AM, sarlav kumar wrote: > > Hi Sean, > > The problem is that I dont have permission to create directories as a > postgres user. > If I can get the \copy command or the \! pg_dump command to work, that > would be great. > > Thanks, > Saranya > > Sean Davis <sdavis2@mail.nih.gov> wrote: > > On Jan 21, 2005, at 11:00 AM, sarlav kumar wrote: > > > Hi Sean, > > > > Thanks, for the help. I got the dynamic query generation part to > work. > > The only thing left to do is to get the dump of the temporary table. > > > > When I try to use COPY inside the pl/pgsql function, I get the > > following error: > > > > COPY temp1 to ''aff.txt''; > > > > WARNING: Error occurred while executing PL/pgSQL function try2 > > WARNING: line 38 at SQL statement > > ERROR: Relative path not allowed for server side COPY command > > > > Then I dropped the function, and recreated the function with the > > following command: > > > > COPY temp1 to ''/home/developers/ss2/aff.txt''; > > > > WARNING: Error occurred while executing PL/pgSQL function try2 > > WARNING: line 38 at SQL statement > > ERROR: COPY command, running in backend with effective uid 501, > could > > not open file '/home/developers/ssivakumar/aff.txt' for writing. > > Errno = No such file or directory (2). > > How can I get the copy command to work from within the pl/pgsql? > > > > The tricky part about COPY is that it is executed by the SERVER! > Therefore, the tables can only be written to somewhere writable by the > user running the server process. If, for example, you have a user > named postgres, you could set up a directory that is owned by postgres > and use that for the dumps. /tmp is another place. Of course, all > this has to be done on the SERVER machine; it can't be done locally to > a file. I imagine that is the issue, but others can correct me if I am > wrong on this. > > Another option is to COPY to STDOUT and then capture the output. > > Sean > > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com
On Jan 21, 2005, at 11:44 AM, sarlav kumar wrote: > The problem is that I dont have permission to create directories as a > postgres user. > If I can get the \copy command or the \! pg_dump command to work, that > would be great. > The commands starting with \ are psql commands. They are only executed on the client side by psql. Stored procedures of course run on the server, and don't know anything about psql \commands. What you can do is setup a script for psql that first calls your stored procedure and then uses \copy or \!pg_dump. But you can't put these commands in the stored procedure. John DeSoi, Ph.D. http://pgedit.com/ Power Tools for PostgreSQL