Обсуждение: posgresql.log
Hi Gurus,
Looking into my postgresql.log on one of my test servers I found scary entry:
--2018-05-19 05:28:21-- http://207.148.79.161/post0514/post
Connecting to 207.148.79.161:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1606648 (1.5M) [application/octet-stream]
Saving to: ‘/var/lib/postgresql/10/main/postgresq1’
0K .......... .......... .......... .......... .......... 3% 71.0K 21s
50K .......... .......... .......... .......... .......... 6% 106K 17s
100K .......... .......... .......... .......... .......... 9% 213K 13s
150K .......... .......... .......... .......... .......... 12% 213K 11s
200K .......... .......... .......... .......... .......... 15% 16.3M 9s
250K .......... .......... .......... .......... .......... 19% 215K 8s
300K .......... .......... .......... .......... .......... 22% 15.6M 7s
350K .......... .......... .......... .......... .......... 25% 11.7M 6s
400K .......... .......... .......... .......... .......... 28% 219K 5s
450K .......... .......... .......... .......... .......... 31% 12.1M 5s
500K .......... .......... .......... .......... .......... 35% 11.7M 4s
550K .......... .......... .......... .......... .......... 38% 12.2M 3s
600K .......... .......... .......... .......... .......... 41% 12.1M 3s
650K .......... .......... .......... .......... .......... 44% 228K 3s
700K .......... .......... .......... .......... .......... 47% 12.2M 3s
750K .......... .......... .......... .......... .......... 50% 12.1M 2s
800K .......... .......... .......... .......... .......... 54% 11.7M 2s
850K .......... .......... .......... .......... .......... 57% 12.1M 2s
900K .......... .......... .......... .......... .......... 60% 11.8M 2s
950K .......... .......... .......... .......... .......... 63% 12.1M 1s
1000K .......... .......... .......... .......... .......... 66% 12.0M 1s
1050K .......... .......... .......... .......... .......... 70% 243K 1s
1100K .......... .......... .......... .......... .......... 73% 12.1M 1s
1150K .......... .......... .......... .......... .......... 76% 12.1M 1s
1200K .......... .......... .......... .......... .......... 79% 11.7M 1s
1250K .......... .......... .......... .......... .......... 82% 12.1M 1s
1300K .......... .......... .......... .......... .......... 86% 12.1M 0s
1350K .......... .......... .......... .......... .......... 89% 11.8M 0s
1400K .......... .......... .......... .......... .......... 92% 12.1M 0s
1450K .......... .......... .......... .......... .......... 95% 12.1M 0s
1500K .......... .......... .......... .......... .......... 98% 11.8M 0s
1550K .......... ........ 100% 12.5M=2.6s
2018-05-19 05:28:25 (598 KB/s) - ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]
Downloaded file is not posgresql but postgresq1(one).
It was pure pg instalation without any contrib modules addons etc, istalled on ubuntu box by apt manager using repos:
http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main
http://apt.postgresql.org/pub/repos/apt xenial-pgdg
I have never seen such entry on other my other servers…
Could you be so kind and explain me what is it? I am afraid my postgres has been hacekd.
Best Regards
Bartosz Dmytrak
On 05/21/2018 02:40 PM, Bartosz Dmytrak wrote: > Hi Gurus, > > Looking into my postgresql.log on one of my test servers I found scary > entry: Is there a Web app running on this server? The log entries below are from the Postgres logs in?: /var/log/postgresql/ > > --2018-05-19 05:28:21-- http://207.148.79.161/post0514/post > > Connecting to 207.148.79.161:80... connected. > > HTTP request sent, awaiting response... 200 OK > > Length: 1606648 (1.5M) [application/octet-stream] Hmm, the below says it downloaded 12.5M. > > Saving to: ‘/var/lib/postgresql/10/main/postgresq1’ The postgresq1 file is actually there? If so have you looked at the file: file postgresq1 to get an idea of what it is? > > 0K .......... .......... .......... .......... .......... 3% 71.0K 21s > > 50K .......... .......... .......... .......... .......... 6% > 106K 17s > > 100K .......... .......... .......... .......... .......... 9% > 213K 13s > > 150K .......... .......... .......... .......... .......... 12% > 213K 11s > > 200K .......... .......... .......... .......... .......... 15% 16.3M 9s > > 250K .......... .......... .......... .......... .......... 19% 215K 8s > > 300K .......... .......... .......... .......... .......... 22% 15.6M 7s > > 350K .......... .......... .......... .......... .......... 25% 11.7M 6s > > 400K .......... .......... .......... .......... .......... 28% 219K 5s > > 450K .......... .......... .......... .......... .......... 31% 12.1M 5s > > 500K .......... .......... .......... .......... .......... 35% 11.7M 4s > > 550K .......... .......... .......... .......... .......... 38% 12.2M 3s > > 600K .......... .......... .......... .......... .......... 41% 12.1M 3s > > 650K .......... .......... .......... .......... .......... 44% 228K 3s > > 700K .......... .......... .......... .......... .......... 47% 12.2M 3s > > 750K .......... .......... .......... .......... .......... 50% 12.1M 2s > > 800K .......... .......... .......... .......... .......... 54% 11.7M 2s > > 850K .......... .......... .......... .......... .......... 57% 12.1M 2s > > 900K .......... .......... .......... .......... .......... 60% 11.8M 2s > > 950K .......... .......... .......... .......... .......... 63% 12.1M 1s > > 1000K .......... .......... .......... .......... .......... 66% 12.0M 1s > > 1050K .......... .......... .......... .......... .......... 70% 243K 1s > > 1100K .......... .......... .......... .......... .......... 73% 12.1M 1s > > 1150K .......... .......... .......... .......... .......... 76% 12.1M 1s > > 1200K .......... .......... .......... .......... .......... 79% 11.7M 1s > > 1250K .......... .......... .......... .......... .......... 82% 12.1M 1s > > 1300K .......... .......... .......... .......... .......... 86% 12.1M 0s > > 1350K .......... .......... .......... .......... .......... 89% 11.8M 0s > > 1400K .......... .......... .......... .......... .......... 92% 12.1M 0s > > 1450K .......... .......... .......... .......... .......... 95% 12.1M 0s > > 1500K .......... .......... .......... .......... .......... 98% 11.8M 0s > > 1550K .......... ........ 100% 12.5M=2.6s > > 2018-05-19 05:28:25 (598 KB/s) - > ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648] > > Downloaded file is not posgresql but postgresq1(one). > > It was pure pg instalation without any contrib modules addons etc, > istalled on ubuntu box by apt manager using repos: > > http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main > > http://apt.postgresql.org/pub/repos/apt xenial-pgdg > > I have never seen such entry on other my other servers… > > Could you be so kind and explain me what is it? I am afraid my postgres > has been hacekd. > > Best Regards > > */Bartosz Dmytrak/* > -- Adrian Klaver adrian.klaver@aklaver.com
Hi Gurus,
Looking into my postgresql.log on one of my test servers I found scary entry:
--2018-05-19 05:28:21-- http://207.148.79.161/post0514/post
Connecting to 207.148.79.161:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1606648 (1.5M) [application/octet-stream]
Saving to: ‘/var/lib/postgresql/10/main/postgresq1’
0K .......... .......... .......... .......... .......... 3% 71.0K 21s
50K .......... .......... .......... .......... .......... 6% 106K 17s
100K .......... .......... .......... .......... .......... 9% 213K 13s
150K .......... .......... .......... .......... .......... 12% 213K 11s
200K .......... .......... .......... .......... .......... 15% 16.3M 9s
250K .......... .......... .......... .......... .......... 19% 215K 8s
300K .......... .......... .......... .......... .......... 22% 15.6M 7s
350K .......... .......... .......... .......... .......... 25% 11.7M 6s
400K .......... .......... .......... .......... .......... 28% 219K 5s
450K .......... .......... .......... .......... .......... 31% 12.1M 5s
500K .......... .......... .......... .......... .......... 35% 11.7M 4s
550K .......... .......... .......... .......... .......... 38% 12.2M 3s
600K .......... .......... .......... .......... .......... 41% 12.1M 3s
650K .......... .......... .......... .......... .......... 44% 228K 3s
700K .......... .......... .......... .......... .......... 47% 12.2M 3s
750K .......... .......... .......... .......... .......... 50% 12.1M 2s
800K .......... .......... .......... .......... .......... 54% 11.7M 2s
850K .......... .......... .......... .......... .......... 57% 12.1M 2s
900K .......... .......... .......... .......... .......... 60% 11.8M 2s
950K .......... .......... .......... .......... .......... 63% 12.1M 1s
1000K .......... .......... .......... .......... .......... 66% 12.0M 1s
1050K .......... .......... .......... .......... .......... 70% 243K 1s
1100K .......... .......... .......... .......... .......... 73% 12.1M 1s
1150K .......... .......... .......... .......... .......... 76% 12.1M 1s
1200K .......... .......... .......... .......... .......... 79% 11.7M 1s
1250K .......... .......... .......... .......... .......... 82% 12.1M 1s
1300K .......... .......... .......... .......... .......... 86% 12.1M 0s
1350K .......... .......... .......... .......... .......... 89% 11.8M 0s
1400K .......... .......... .......... .......... .......... 92% 12.1M 0s
1450K .......... .......... .......... .......... .......... 95% 12.1M 0s
1500K .......... .......... .......... .......... .......... 98% 11.8M 0s
1550K .......... ........ 100% 12.5M=2.6s
2018-05-19 05:28:25 (598 KB/s) - ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]
Downloaded file is not posgresql but postgresq1(one).
It was pure pg instalation without any contrib modules addons etc, istalled on ubuntu box by apt manager using repos:
http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main
http://apt.postgresql.org/pub/repos/apt xenial-pgdg
I have never seen such entry on other my other servers…
Could you be so kind and explain me what is it? I am afraid my postgres has been hacekd.
Best Regards
Bartosz Dmytrak
> On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote: > > > > If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a pathto other internal or critical resources. If you can image it for safekeeping and forensics, even better. +1 It's compromised. Image it if possible; save the compromise payload you know about if not. Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it. > > That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh, localattack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything relatedto that IP address. It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those reportedrecently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app willhave access to, but which a compromised postgresql can easily write to. Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all yourother postgresql servers, test and production. If there's a configuration mistake that let one server be compromisedit's may well be there on others too. > > Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc. > > IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact theirabuse department. The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interested beyondthat point unless a subpoena or search warrant were involved. > Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean. Bestbet is to backup any critical data (and check it for trustworthiness), wipe and rebuild. +1. > > Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal policies,legal requirements and agreements with third-parties you may have notification requirements and could need to engageforensics experts. Cheers, Steve
[snip]Hi Gurus,
Looking into my postgresql.log on one of my test servers I found scary entry:
--2018-05-19 05:28:21-- http://207.148.79.161/post0514/post
Connecting to 207.148.79.161:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1606648 (1.5M) [application/octet-stream]
Saving to: ‘/var/lib/postgresql/10/main/postgresq1’
0K .......... .......... .......... .......... .......... 3% 71.0K 21s
50K .......... .......... .......... .......... .......... 6% 106K 17s
100K .......... .......... .......... .......... .......... 9% 213K 13s
150K .......... .......... .......... .......... .......... 12% 213K 11s
1500K .......... .......... .......... .......... .......... 98% 11.8M 0s1550K .......... ........ 100% 12.5M=2.6s
2018-05-19 05:28:25 (598 KB/s) - ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]
Downloaded file is not posgresql but postgresq1(one).
It was pure pg instalation without any contrib modules addons etc, istalled on ubuntu box by apt manager using repos:
http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main
http://apt.postgresql.org/pub/repos/apt xenial-pgdg
I have never seen such entry on other my other servers…
Could you be so kind and explain me what is it? I am afraid my postgres has been hacekd.
This looks like what happens when the adobe flash player package downloads the closed-source binary installer. Thus, I wouldn't be surprised if the repository package isn't downloading the installation binaries from http://207.148.79.161/post0514/post.
Angular momentum makes the world go 'round.
-----Original Message----- From: Adrian Klaver [mailto:adrian.klaver@aklaver.com] Sent: Tuesday, May 22, 2018 12:03 AM To: Bartosz Dmytrak <bdmytrak@gmail.com>; pgsql-general@postgresql.org Subject: Re: posgresql.log On 05/21/2018 02:40 PM, Bartosz Dmytrak wrote: > Hi Gurus, > > Looking into my postgresql.log on one of my test servers I found scary > entry: Is there a Web app running on this server? The log entries below are from the Postgres logs in?: /var/log/postgresql/ > > --2018-05-19 05:28:21-- http://207.148.79.161/post0514/post > > Connecting to 207.148.79.161:80... connected. > > HTTP request sent, awaiting response... 200 OK > > Length: 1606648 (1.5M) [application/octet-stream] Hmm, the below says it downloaded 12.5M. > > Saving to: ‘/var/lib/postgresql/10/main/postgresq1’ The postgresq1 file is actually there? If so have you looked at the file: file postgresq1 to get an idea of what it is? > > 0K .......... .......... .......... .......... .......... 3% 71.0K > 21s > > 50K .......... .......... .......... .......... .......... 6% > 106K 17s > > 100K .......... .......... .......... .......... .......... 9% > 213K 13s > > 150K .......... .......... .......... .......... .......... 12% > 213K 11s > > 200K .......... .......... .......... .......... .......... 15% > 16.3M 9s > > 250K .......... .......... .......... .......... .......... 19% > 215K 8s > > 300K .......... .......... .......... .......... .......... 22% > 15.6M 7s > > 350K .......... .......... .......... .......... .......... 25% > 11.7M 6s > > 400K .......... .......... .......... .......... .......... 28% > 219K 5s > > 450K .......... .......... .......... .......... .......... 31% > 12.1M 5s > > 500K .......... .......... .......... .......... .......... 35% > 11.7M 4s > > 550K .......... .......... .......... .......... .......... 38% > 12.2M 3s > > 600K .......... .......... .......... .......... .......... 41% > 12.1M 3s > > 650K .......... .......... .......... .......... .......... 44% > 228K 3s > > 700K .......... .......... .......... .......... .......... 47% > 12.2M 3s > > 750K .......... .......... .......... .......... .......... 50% > 12.1M 2s > > 800K .......... .......... .......... .......... .......... 54% > 11.7M 2s > > 850K .......... .......... .......... .......... .......... 57% > 12.1M 2s > > 900K .......... .......... .......... .......... .......... 60% > 11.8M 2s > > 950K .......... .......... .......... .......... .......... 63% > 12.1M 1s > > 1000K .......... .......... .......... .......... .......... 66% > 12.0M 1s > > 1050K .......... .......... .......... .......... .......... 70% > 243K 1s > > 1100K .......... .......... .......... .......... .......... 73% > 12.1M 1s > > 1150K .......... .......... .......... .......... .......... 76% > 12.1M 1s > > 1200K .......... .......... .......... .......... .......... 79% > 11.7M 1s > > 1250K .......... .......... .......... .......... .......... 82% > 12.1M 1s > > 1300K .......... .......... .......... .......... .......... 86% > 12.1M 0s > > 1350K .......... .......... .......... .......... .......... 89% > 11.8M 0s > > 1400K .......... .......... .......... .......... .......... 92% > 12.1M 0s > > 1450K .......... .......... .......... .......... .......... 95% > 12.1M 0s > > 1500K .......... .......... .......... .......... .......... 98% > 11.8M 0s > > 1550K .......... ........ 100% 12.5M=2.6s > > 2018-05-19 05:28:25 (598 KB/s) - > ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648] > > Downloaded file is not posgresql but postgresq1(one). > > It was pure pg instalation without any contrib modules addons etc, > istalled on ubuntu box by apt manager using repos: > > http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main > > http://apt.postgresql.org/pub/repos/apt xenial-pgdg > > I have never seen such entry on other my other servers… > > Could you be so kind and explain me what is it? I am afraid my > postgres has been hacekd. > > Best Regards > > */Bartosz Dmytrak/* > -- Adrian Klaver adrian.klaver@aklaver.com HI, thanks for response, Yes - there is also webapp running on the server, but still it's rather odd to find it's logs in postgresql.log file (locatedin /var/log/postgresql, where my log exists). postgresq1 file exists in /var/lib/postgresql/10/main and it's binaryfile, I've also noticed there is a n596tx.so which is not a part of standard installation. Fortunately there is no important data on this server so, a according to other advice, I'll rebuilt it with more aggressivesecurity settings and I'll apply them to other servers too. Best regards, Bartek
-----Original Message----- From: Steve Atkins [mailto:steve@blighty.com] Sent: Tuesday, May 22, 2018 12:44 AM To: PG-General Mailing List <pgsql-general@postgresql.org> Cc: bdmytrak@gmail.com Subject: Re: posgresql.log > On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote: > > > > If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a pathto other internal or critical resources. If you can image it for safekeeping and forensics, even better. +1 It's compromised. Image it if possible; save the compromise payload you know about if not. Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it. > > That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh, localattack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything relatedto that IP address. It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those reportedrecently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app willhave access to, but which a compromised postgresql can easily write to. Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all yourother postgresql servers, test and production. If there's a configuration mistake that let one server be compromisedit's may well be there on others too. > > Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc. > > IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact theirabuse department. The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interested beyondthat point unless a subpoena or search warrant were involved. > Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean. Bestbet is to backup any critical data (and check it for trustworthiness), wipe and rebuild. +1. > > Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal policies,legal requirements and agreements with third-parties you may have notification requirements and could need to engageforensics experts. Cheers, Steve Hi Steve, Thanks a lot. That’s virtual server, so I'll do the backup. I suppose it was hacked, but fortunately there are no importantdata and I can wipe it out with no hurt. According to your advice I'll check firewall rules and block new IP ranges(many of them are blocked now). There are no default passwords, but will work on more secure approach. This serverhas to be accessible from internet, but not necessary for every IP. Thanks again for your help and advice. Hope this case will let other keep their servers more secure. Regards, Bartek
Hi Bartek It is quite significant that your postgres log file has these entries. Normally if a web application gets compromised allowing remote code execution, the attacker will be able to run scripts (often via cron and at) as the user running the web application. Typically www-data, tomcat8 etc. If the attacker has managed to execute something as the postgres user you need to check: (i) that the web application server (eg tomcat) is not running as root. I see this often enough. Then when the web app gets attacked, root is gone on the machine; (ii) the web application has minimally configured access vi pg_hba.conf. For example not as the postgres user. Just some thoughts to consider as you put your stuff back together on the new machine. Regards Bob On 22 May 2018 at 07:47, Bartosz Dmytrak <bdmytrak@gmail.com> wrote: > > > -----Original Message----- > From: Steve Atkins [mailto:steve@blighty.com] > Sent: Tuesday, May 22, 2018 12:44 AM > To: PG-General Mailing List <pgsql-general@postgresql.org> > Cc: bdmytrak@gmail.com > Subject: Re: posgresql.log > > >> On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote: >> >> >> >> If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a pathto other internal or critical resources. If you can image it for safekeeping and forensics, even better. > > +1 > > It's compromised. Image it if possible; save the compromise payload you know about if not. > > Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it. > >> >> That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh, localattack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything relatedto that IP address. > > It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those reportedrecently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app willhave access to, but which a compromised postgresql can easily write to. > > Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all yourother postgresql servers, test and production. If there's a configuration mistake that let one server be compromisedit's may well be there on others too. > >> >> Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc. >> >> IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact theirabuse department. > > The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interestedbeyond that point unless a subpoena or search warrant were involved. > >> Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean.Best bet is to backup any critical data (and check it for trustworthiness), wipe and rebuild. > > +1. > >> >> Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal policies,legal requirements and agreements with third-parties you may have notification requirements and could need to engageforensics experts. > > Cheers, > Steve > > > Hi Steve, > Thanks a lot. That’s virtual server, so I'll do the backup. I suppose it was hacked, but fortunately there are no importantdata and I can wipe it out with no hurt. According to your advice I'll check firewall rules and block new IP ranges(many of them are blocked now). There are no default passwords, but will work on more secure approach. This serverhas to be accessible from internet, but not necessary for every IP. > > Thanks again for your help and advice. Hope this case will let other keep their servers more secure. > > Regards, > Bartek > >