Generally, I'd suggest you think carefully about the nature of the jobs, and draw up a list of must-have properties (performance of course, but also things like whether jobs have to survive planned or unplanned outages, be visible across a WAN, numbers of readers and writers, delivery guarantees, etc etc) and then decide on make versus "buy". Distributed systems are hard, and hard to get right.
Am 22.03.2024 um 14:15 schrieb Fred Habash:
> We developed a home-grown queue system using Postgres, but its
> performance was largely hindered by que tables bloating and the need to
> continuously vacuum them. It did not scale whatsoever. With some
> workarounds, we ended up designing three sets of queue tables, switching
> between them based on some queue stats, vacuum the inactive set, and repeat.
> We kept this queue system for low SLA app components. For others, we
> switched to Kafka. Investing in learning and implementing purpose built
> queue systems pays off for the long term.
I wonder whether one should https://youtu.be/VEWXmdjzIpQ&t=543 not to
scale either.