Dear Users! Please post bug reports about FDM here!

can you set up bulk downloads from internet archive?

Moderators: Usher, Alex

pizzacube

can you set up bulk downloads from internet archive?

Postby pizzacube » Wed Aug 11, 2021 8:16 am

can you set up bulk downloads from the internet archive?

some large collections will not download in bulk and it gives you pages like this one example

https://archive.org/download/Entex_Adve ... 2012_04_23

that list links to download the file?

can the FDM get the files from the pages automatic so i dont have to click on each one and download them 1 at a time manually?

some of them are like 100 download links it will take forever.
User avatar
Usher
Posts: 2557
Joined: Sat Mar 20, 2010 2:37 pm
Location: Poland

Re: can you set up bulk downloads from internet archive?

Postby Usher » Wed Aug 11, 2021 4:45 pm

It depend on your needs. You can:
* Download zip archives from there.
* Download torrent which contains zip files together with images and some additional info/metadata.
* Use "Download all" from FDM context menu to download all files even if you don't need them (metadata) or they are duplicated (some folders may contain also unzipped files). With this option you can also select only needed file extensions.

For me torrent is the best choice.

Oh, and your example is a very small - only one zip and less than 1 megabyte in total.
Andrzej P. Wozniak, FDM user and forum moderator
Read FDM FAQ and the reporting rules
"How to report a bug or a problem with FDM" before posting
pizzacube

Re: can you set up bulk downloads from internet archive?

Postby pizzacube » Thu Aug 12, 2021 2:15 am

none of that is an option for the large collections

https://archive.org/download/redump.sega_saturn
Alex
FDM Team
Posts: 2784
Joined: Mon Jul 12, 2004 12:17 pm

Re: can you set up bulk downloads from internet archive?

Postby Alex » Thu Aug 12, 2021 3:40 pm

Hello,
Why is "Download all" not an option? Works fine for me. Yes, it downloads lot of useless HTML files, but it's still usable.
Alex,
FDM development team
User avatar
Usher
Posts: 2557
Joined: Sat Mar 20, 2010 2:37 pm
Location: Poland

Re: can you set up bulk downloads from internet archive?

Postby Usher » Thu Aug 12, 2021 8:49 pm

pizzacube wrote:none of that is an option for the large collections
https://archive.org/download/redump.sega_saturn
"Download all" should work OK. Do you have enough space for that download? If you are afraid of high CPU load, use moderate (medium load) mode and change (lower) traffic limits for that mode before starting download. You may even use snail mode for the start…
Andrzej P. Wozniak, FDM user and forum moderator
Read FDM FAQ and the reporting rules
"How to report a bug or a problem with FDM" before posting

Return to “General forum”

Who is online

Users browsing this forum: No registered users and 1 guest