Обсуждение: Compromised postgresql instances
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internetwith insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries beinguploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extensionor an untrusted PL. Is anyone tracking or investigating this? Cheers, Steve
On 06/08/2018 04:34 PM, Steve Atkins wrote: > I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internetwith insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries beinguploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extensionor an untrusted PL. > > Is anyone tracking or investigating this? > Please cite actual instances of such reports. Vague queries like this help nobody. Furthermore, security concerns are best addressed to the security mailing list. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: > On 06/08/2018 04:34 PM, Steve Atkins wrote: >> I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internetwith insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries beinguploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extensionor an untrusted PL. >> >> Is anyone tracking or investigating this? > Please cite actual instances of such reports. Vague queries like this > help nobody. I imagine Steve is reacting to this report from today: https://www.postgresql.org/message-id/CANozSKLGgWDpzfua2L=OGFN=Dg3Po98UjqJJ18gBVFR1-yK5+A@mail.gmail.com I recall something similar being reported a few weeks ago, but am too lazy to trawl the archives right now. > Furthermore, security concerns are best addressed to the security > mailing list. Unless there's some evidence that these attacks are getting in through a heretofore unknown PG security vulnerability, rather than user misconfiguration (such as weak/no password), I'm not sure what the security list would have to offer. Right now it seems like Steve's move to try to gather more evidence is quite the right thing to do. regards, tom lane
> On Jun 8, 2018, at 1:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >> On 06/08/2018 04:34 PM, Steve Atkins wrote: >>> I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internetwith insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries beinguploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extensionor an untrusted PL. >>> >>> Is anyone tracking or investigating this? > >> Please cite actual instances of such reports. Vague queries like this >> help nobody. > > I imagine Steve is reacting to this report from today: > https://www.postgresql.org/message-id/CANozSKLGgWDpzfua2L=OGFN=Dg3Po98UjqJJ18gBVFR1-yK5+A@mail.gmail.com > > I recall something similar being reported a few weeks ago, https://www.postgresql.org/message-id/020901d3f14c%24512a46d0%24f37ed470%24%40gmail.com > but am > too lazy to trawl the archives right now. Yes, plus I recall a couple of discussions on IRC with similar behaviour, and a few more details about how the binaries were being uploaded. > >> Furthermore, security concerns are best addressed to the security >> mailing list. > > Unless there's some evidence that these attacks are getting in through > a heretofore unknown PG security vulnerability, rather than user > misconfiguration (such as weak/no password), I'm not sure what the > security list would have to offer. Right now it seems like Steve's move > to try to gather more evidence is quite the right thing to do. Yeah. It's not a security issue with postgresql itself, I don't believe, so not really something that has to go to the security alias. It's more of an ops issue, but I thought I'd ask here to see if anyone was already looking at it, and to raise a flag if they weren't. Cheers, Steve
On 06/08/2018 04:54 PM, Steve Atkins wrote: >> On Jun 8, 2018, at 1:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> >> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >>> On 06/08/2018 04:34 PM, Steve Atkins wrote: >>>> I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internetwith insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries beinguploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extensionor an untrusted PL. >>>> >>>> Is anyone tracking or investigating this? >>> Please cite actual instances of such reports. Vague queries like this >>> help nobody. >> I imagine Steve is reacting to this report from today: >> https://www.postgresql.org/message-id/CANozSKLGgWDpzfua2L=OGFN=Dg3Po98UjqJJ18gBVFR1-yK5+A@mail.gmail.com >> >> I recall something similar being reported a few weeks ago, > https://www.postgresql.org/message-id/020901d3f14c%24512a46d0%24f37ed470%24%40gmail.com OK, those appeared on other mailing lists I don't subscribe to, so I was missing context. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes: > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >> Please cite actual instances of such reports. Vague queries like >> this help nobody. We do also get them on the IRC channel every once in a while, not very frequently but enough to notice (maybe 2-3 so far this year?). Tom> Unless there's some evidence that these attacks are getting in Tom> through a heretofore unknown PG security vulnerability, rather Tom> than user misconfiguration (such as weak/no password), I'm not Tom> sure what the security list would have to offer. Right now it Tom> seems like Steve's move to try to gather more evidence is quite Tom> the right thing to do. Right. All the instances on IRC that I'm personally aware of have followed this pattern: either the user has used "host all all 0.0.0.0/0 trust", or they used "host all all 0.0.0.0/0 md5" where the password for the postgres user has been one where it's plausible that a simple automated dictionary attack could have guessed it (e.g. "654321" in one report). Excuses for doing this have varied (but I'm pretty sure I've heard "we put that in while trying to fix a problem and forgot to take it out" more than once - so it's worth a reminder that one should never, ever, suggest this on any help forum). The reports are similar enough and generic enough that it seems pretty certain that the scanning and subsequent compromise is all automated; some reports have included a (failed) attempt to use local root exploits to escalate from the postgres user to root access. Compromised systems have been reportedly used as DDoS traffic sources and for cryptocurrency mining, but obviously other uses can't be ruled out. I don't know of any reports of data being exfiltrated or modified, but that doesn't mean it doesn't happen. I have (private) logs of the channel going back a while, but I haven't made any attempt to track how often it happens - the above is basically all from memory. -- Andrew (irc:RhodiumToad)
On 06/08/2018 06:13 PM, Andrew Gierth wrote: >>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes: > > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: > >> Please cite actual instances of such reports. Vague queries like > >> this help nobody. > > We do also get them on the IRC channel every once in a while, not > very frequently but enough to notice (maybe 2-3 so far this year?). > > Tom> Unless there's some evidence that these attacks are getting in > Tom> through a heretofore unknown PG security vulnerability, rather > Tom> than user misconfiguration (such as weak/no password), I'm not > Tom> sure what the security list would have to offer. Right now it > Tom> seems like Steve's move to try to gather more evidence is quite > Tom> the right thing to do. > > Right. All the instances on IRC that I'm personally aware of have > followed this pattern: either the user has used "host all all 0.0.0.0/0 > trust", or they used "host all all 0.0.0.0/0 md5" where the password for > the postgres user has been one where it's plausible that a simple > automated dictionary attack could have guessed it (e.g. "654321" in one > report). > > Excuses for doing this have varied (but I'm pretty sure I've heard "we > put that in while trying to fix a problem and forgot to take it out" > more than once - so it's worth a reminder that one should never, ever, > suggest this on any help forum). > > The reports are similar enough and generic enough that it seems pretty > certain that the scanning and subsequent compromise is all automated; > some reports have included a (failed) attempt to use local root exploits > to escalate from the postgres user to root access. Compromised systems > have been reportedly used as DDoS traffic sources and for cryptocurrency > mining, but obviously other uses can't be ruled out. I don't know of any > reports of data being exfiltrated or modified, but that doesn't mean it > doesn't happen. > > I have (private) logs of the channel going back a while, but I haven't > made any attempt to track how often it happens - the above is basically > all from memory. > Well, I guess every service is going to be vulnerable to misconfiguration. The fact that they are automated (oresumably knowing how to speak the wire protocol or using a library that does) is a bit scary, but not surprising. My view is that generally DB servers should not be reachable from the internet in the first place, certainly not without a client cert, but ideally not at all. Then misconfigurations like those Andrew refers to above would not be so lethal. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
> Please cite actual instances of such reports. Vague queries like this help nobody. There were several questions on SO https://stackoverflow.com/questions/49815460 https://stackoverflow.com/questions/47499766 https://stackoverflow.com/questions/47741077 https://dba.stackexchange.com/questions/184540 And a blog post going into details on how that specific attack works. https://www.imperva.com/blog/2018/03/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/ -- Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
>>>>> "Thomas" == Thomas Kellerer <spam_eater@gmx.net> writes: Thomas> And a blog post going into details on how that specific attack works. Thomas> https://www.imperva.com/blog/2018/03/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/ *headdesk* *headdesk* *headdesk* FOR THE LOVE OF LITTLE APPLES, why, in an article as comprehensive as this, did they not list in the "quick tips" at the end, the quickest and most absolutely basic and essential tip of all, which is "don't open up your database for superuser access from the whole world" ??? To become vulnerable to this attack, you have to do ALL of these: - give your db a public IP - allow access (or forget to prevent access) to it through any firewall - configure pg to listen on the public IP - explicitly add an entry to pg_hba.conf that allows access from 0.0.0.0/0 for all users or at least the postgres user - AND have a guessable password on the postgres user or explicitly use "trust" on the above hba entry *headdesk* -- Andrew (irc:RhodiumToad)
On 06/09/2018 03:27 AM, Andrew Gierth wrote: >>>>>> "Thomas" == Thomas Kellerer <spam_eater@gmx.net> writes: > Thomas> And a blog post going into details on how that specific attack works. > > Thomas> https://www.imperva.com/blog/2018/03/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/ > > *headdesk* > > *headdesk* > > *headdesk* > > FOR THE LOVE OF LITTLE APPLES, why, in an article as comprehensive as > this, did they not list in the "quick tips" at the end, the quickest and > most absolutely basic and essential tip of all, which is "don't open up > your database for superuser access from the whole world" ??? > > To become vulnerable to this attack, you have to do ALL of these: > > - give your db a public IP > - allow access (or forget to prevent access) to it through any > firewall > - configure pg to listen on the public IP > - explicitly add an entry to pg_hba.conf that allows access from > 0.0.0.0/0 for all users or at least the postgres user > - AND have a guessable password on the postgres user or explicitly > use "trust" on the above hba entry > > *headdesk* > Against stupidity the Gods themselves contend in vain. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services