Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones

Поиск
Список
Период
Сортировка
От Brent Wood
Тема Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones
Дата
Msg-id SY4PR01MB707634C6F5C0FFD83BA0E125A1CBA@SY4PR01MB7076.ausprd01.prod.outlook.com
обсуждение исходный текст
Ответ на Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones  (Lincoln Swaine-Moore <lswainemoore@gmail.com>)
Список pgsql-general
Дерево обсуждения
Slightly off topic, but has anyone tried TimescaleDB for timeseries databases?

The issues discussed here are still there as they apply to the underlying Postgres ORDBMS.

We solve the problem (around 4 billion records of instrument sensor readings) by using UTC for the "native" timestamp, and working in that. Even though we are ½ way around the world. The local times can easily be determined & applied if desired, but by standardising on the reference time zone at the start, things have "just worked", for around 15 years now. 


Brent Wood

Principal Technician, Fisheries
NIWA
DDI:  +64 (4) 3860529
From: Lincoln Swaine-Moore <lswainemoore@gmail.com>
Sent: Thursday, October 5, 2023 08:30
To: Alban Hertroys <haramrae@gmail.com>
Cc: Marian Wendt <marian.wendt@yahoo.com>; pgsql-general <pgsql-general@lists.postgresql.org>
Subject: Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones
 
> What I do in such cases is to add an extra column with the UTC timestamp to serve as a linear scale to the local timestamps. That also helps with ordering buckets in reports and such during DST changes (especially the ones where an hour repeats).

> For hours and quarter hours I found it to be fairly convenient to base a view on a join between a date calendar and an (quarter of an) hour per UTC day table, but materialising that with some indexes may perform better (at the cost of disk space). I do materialise that currently, but our database server doesn’t have a lot of memory so I’m often not hitting the cache and performance suffers a bit (infrastructure is about to change for the better though).

That's an interesting idea, but I'm not sure I fully understand. Assuming you're aggregating data: what do you group by? For instance, at an hourly resolution, if you group by both the UTC timestamp and the local one, you might end up, say, dividing an hour-long bucket in two for time zones with half-hour-based offsets, no? 

Thanks for the detailed writeup! Definitely helpful to learn more about what people are using in production to handle this sort of thing.

--
Lincoln Swaine-Moore
Brent Wood
Principal Technician - GIS and Spatial Data Management
Programme Leader - Environmental Information Delivery
+64-4-386-0529

National Institute of Water & Atmospheric Research Ltd (NIWA)
301 Evans Bay Parade Hataitai Wellington New Zealand
Connect with NIWA: niwa.co.nz Facebook LinkedIn Twitter Instagram YouTube
To ensure compliance with legal requirements and to maintain cyber security standards, NIWA's IT systems are subject to ongoing monitoring, activity logging and auditing. This monitoring and auditing service may be provided by third parties. Such third parties can access information transmitted to, processed by and stored on NIWA's IT systems.
Note: This email is intended solely for the use of the addressee and may contain information that is confidential or subject to legal professional privilege. If you receive this email in error please immediately notify the sender and delete the email.

В списке pgsql-general по дате отправления:

Предыдущее
От: Lincoln Swaine-Moore
Дата:
Сообщение: Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones
Следующее
От: Alban Hertroys
Дата:
Сообщение: Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones