There's a reason “everybody” advices to move blobs out of DB, I've learned.
I get that. I really do. But the alternative has some real downsides too.
Especially around security, as I already mentioned. That's why I'd like if possible
to get input on the technical questions of my initial post.
That's not to say we wouldn't ultimately move out the big blobs outside the DB.
But given how much that would complexify the project, I do believe it is better
to do it as a second step, once the full system is up-and-running and testing at
scale has actually been performed.
We've already moved other kind of data to PostgreSQL, from SQLite DBs (thousands) this time,
and ported "as-is" the sharding done on the SQLite side to PostgreSQL (despite TOAST).
And so far, so good. With good ingestion rates. And decent runtime access to data too,
in the albeit limited testing we've had so far.
Now we need to move this other kind of data, from proprietary DB-like files this times (thousands too),
to finish our system, and be able to finally test the whole system in earnest, and at (our limited internal) scale.
So you see, I'm not completely ignoring your advise.
But for now, I'm inquiring as to the *best* way to put that data *in* PostgreSQL,
with the requirements / constraints I've listed in the first post.
It may indeed be a bad idea long term. But let's make the most of it for now.
Makes sense? Am I being unreasonable here? --DD