Обсуждение: Array columns vs normalized table

I work with state labor data which is reported to us in the form

    industry, year, quarter1, quarter2, quarter3, quarter4

where each quarter represents an employment count. Obviously, this can
be normalized to

    industry, year, quarter, employment

Can anyone comment on, or point to me to an article or discussion
regarding, why one would use an array column instead of normalizing
the data? That is, would there be any benefit to storing it as

    industry int, year smallint, employment int[ ]

where the last column would be a four element array with data for the
four quarters.

Thanks,
--Lee

--
Lee Hachadoorian
PhD Student, Geography
Program in Earth & Environmental Sciences
CUNY Graduate Center

Hello

2010/3/2 Lee Hachadoorian <lee.hachadoorian@gmail.com>:
> I work with state labor data which is reported to us in the form
>
>        industry, year, quarter1, quarter2, quarter3, quarter4
>
> where each quarter represents an employment count. Obviously, this can
> be normalized to
>
>        industry, year, quarter, employment
>
> Can anyone comment on, or point to me to an article or discussion
> regarding, why one would use an array column instead of normalizing
> the data? That is, would there be any benefit to storing it as

for very large timeseries you can use arrays. The storage can be more effective.

sample

industry year q1 q2 q3 q4
1, 2001, 10,11,12,13
1, 2002, 14, 15,16,17

can be transformed to tuple (industry: 1, start_year: 2002, data:
10,11,12,13,14,15,16,17, ..... )

your model isn't more safer then normalised tables and I think is
better prefer normalized tables.

the queries to arrays are difficult - so usually data from arrays are
dynamically unpacked to tables. But  time series stored as array take
significantly less space on disc. Look on
http://www.postgresql.org/docs/8.3/static/storage-page-layout.html -
you safe repeated row headers.

Regards
Pavel Stehule



I hope so nobody uses this technique. It cannot substitute normalizated tables.

>
>        industry int, year smallint, employment int[ ]

if

>
> where the last column would be a four element array with data for the
> four quarters.
>
> Thanks,
> --Lee
>
> --
> Lee Hachadoorian
> PhD Student, Geography
> Program in Earth & Environmental Sciences
> CUNY Graduate Center
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

On Tue, Mar 2, 2010 at 11:21 AM, Lee Hachadoorian
<lee.hachadoorian@gmail.com> wrote:
> I work with state labor data which is reported to us in the form
>
>        industry, year, quarter1, quarter2, quarter3, quarter4
>
> where each quarter represents an employment count. Obviously, this can
> be normalized to
>
>        industry, year, quarter, employment
>
> Can anyone comment on, or point to me to an article or discussion
> regarding, why one would use an array column instead of normalizing
> the data? That is, would there be any benefit to storing it as
>
>        industry int, year smallint, employment int[ ]
>
> where the last column would be a four element array with data for the
> four quarters.

I think you might want to step back and ask yourself why you'd want to
normalize this data at all?  Unless you are trying to operate on all 4
columns at once (comparisons or aggregation, etc.) I see no reason to.
 It will take more space and give you no new capabilities.  If you
need do need to operate on all 4 quarters simultaneously then you may
very well want to normalize, but in that case there's certainly no
advantage in going to an array type.  Personally, the only reason I'd
see for ever using an array type is when you have many very closely
related values that would cause some huge number of rows if the data
is stored normalized

--
Peter Hunsberger

Lee Hachadoorian wrote:
> I work with state labor data which is reported to us in the form
>
>     industry, year, quarter1, quarter2, quarter3, quarter4
>
> where each quarter represents an employment count. Obviously, this can
> be normalized to
>
>     industry, year, quarter, employment
>
> Can anyone comment on, or point to me to an article or discussion
> regarding, why one would use an array column instead of normalizing
> the data? That is, would there be any benefit to storing it as
>
>     industry int, year smallint, employment int[ ]
>
> where the last column would be a four element array with data for the
> four quarters.
>
> Thanks,
> --Lee
>
> --
> Lee Hachadoorian
> PhD Student, Geography
> Program in Earth & Environmental Sciences
> CUNY Graduate Center

If you want to do that, I'd recommend:
industry int,
year    smallint,
emp_q1    int,
emp_q2    int,
emp_q3    int,
emp_q4    int

That way it is more clear, easier to query, uses less space and you wont
end up with employment data for the 5th quarter or something odd like that.

Arrays are great for working with your data during the query process.
But you should generally avoid using them to store your data on disk.

Scott

Pavel, the idea of using arrays to store long time-series data sounds
good, but I take your point that normalized tables might be better and
are easier to query. I suppose the choice will be between normalizing
or using the denormalized

industry int,
year    smallint,
emp_q1  int,
emp_q2  int,
emp_q3  int,
emp_q4  int

as suggested by Peter and Scott. We're mostly actually interested in
annual numbers, but need to preserve the quarterly data for
verification and for unusual analyses. So perhaps storing denormalized
with an additional emp_annual int field, and a view that keeps the
quarterly data out of sight.

Thanks for your replies. Please feel free to comment if you think of
anything else.

Best,
--Lee

--
Lee Hachadoorian
PhD Student, Geography
Program in Earth & Environmental Sciences
CUNY Graduate Center