Обсуждение: Download failures on apt archive
Hello, We've been trying to do direct downloads from the apt archive (e.g. [1]) and many of the files fail to download with an AccessDenied error with XML from S3 like: <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>FRN2GJREB814S5FD</RequestId> <HostId> 47zx5Bji8uWyhDlF+59x0dJhK/8+8TInBCVeQqwhIZecGE2pZfPlFZEIO2/aVtam+My7h/WY9ls= </HostId> </Error> For example [2] fails but for example [3] succeeds. This seems to be repeatable by file across browsers and with my co-workers, so I don't think it's just a client issue. Thanks, James Coleman 1: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html 2: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb 3. https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/postgresql-14_14.1-1.pgdg20.04%2B1_arm64.deb
Hi James, apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate index lists (e.g, index.html) which is why [1] doesn't work. If you are familiar with the structure of Debian repos, you can view the repo's index by browsing the appropriate files directly. For example, https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/InRelease where distribution is bullseye, focal, etc. The "InRelease" file will contain paths for various "Packages" files. You can view the list of packages by replacing "InRelease" with the appropraite package path. For example https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/main/binary-amd64/Packages Using this, we can see why [2] isn't working for you -- there is no libpq-dev in bullseye-pgdg-archive, the package is libpq5-dev. Hope this helps somewhat. --Bradford
Re: James Coleman > Hello, > > We've been trying to do direct downloads from the apt archive (e.g. > [1]) and many of the files fail to download with an AccessDenied error > with XML from S3 like: > > <Error> > <Code>AccessDenied</Code> Hmm, admittedly I've never tried to recheck if all the package links worked after the initial setup, so it's well possible something went wrong in the years since then. I'll do some QA there tomorrow. Thanks for reporting! Re: Bradford Boyle > Hi James, > > apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate > index lists (e.g, index.html) which is why [1] doesn't work. There are manually uploaded index.html files generated from the database mentioned earlier. > Using this, we can see why [2] isn't working for you -- there is no libpq-dev > in bullseye-pgdg-archive, the package is libpq5-dev. libpq-dev is correct, the 5 is only in libpq5.deb. Christoph
> There are manually uploaded index.html files generated from the > database mentioned earlier. Yep, I see that now. I should have read the code for generate-archive-lists more closely before replying. > libpq-dev is correct, the 5 is only in libpq5.deb. Right again -- I don't know what I was doing wrong with my grep commands this morning but I'll blame it on a lack of sleep and coffee. It looks like the links on [1] are incorrect -- it gives [2] as the link for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package file, I think the correct link is [3] (i.e., pool/main vs pool/14). I think the issue has to do with set_package_component in generate-pgdg-source moving lib packages to a different component when the source postgresql version is older than PG_MAIN_VERSION; it looks generate-archive-lists assumes component is always main though. --Bradford [1]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html [2]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb [3]: https://apt-archive.postgresql.org/pub/repos/apt/pool/14/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb
Re: Bradford Boyle > It looks like the links on [1] are incorrect -- it gives [2] as the link > for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package > file, I think the correct link is [3] (i.e., pool/main vs pool/14). Ah right, thanks for spotting that, I had missed that when trying to debugging the issue earlier today. I remember now that I had omitted the extra pool locations, but I missed that they are still used. So in summary, the files are all there, but the path in the referencing web page is wrong. Will fix. > I think the issue has to do with set_package_component in > generate-pgdg-source moving lib packages to a different component when > the source postgresql version is older than PG_MAIN_VERSION; it looks > generate-archive-lists assumes component is always main though. Yeah I had the idea that for the archive, I could just ignore the changed component and shovel everything into "main", but of course in that case the files would have to be uploaded to that location. Christoph
Re: To Bradford Boyle > So in summary, the files are all there, but the path in the > referencing web page is wrong. Will fix. I managed to fix the problems, the index listings in package directories like https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html will now also reference files located in other directories like pool/14/p/... James, Bradford: Thanks for spotting and debugging! Christoph
On Fri, May 19, 2023 at 11:31 AM Christoph Berg <myon@debian.org> wrote: > > Re: To Bradford Boyle > > So in summary, the files are all there, but the path in the > > referencing web page is wrong. Will fix. > > I managed to fix the problems, the index listings in package > directories like > https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html > will now also reference files located in other directories like pool/14/p/... > > James, Bradford: Thanks for spotting and debugging! Thanks for fixing this! James