Spider/bulk-downloader-for-reddit
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
818 Commits 2 Branches 45 Tags 14 MiB
4d3f0f9862
Commit Graph

120 Commits

Author SHA1 Message Date
Serene-Arc
c6c6002ab2 Update Erome module 2021-10-02 17:50:20 +10:00
Serene-Arc
9b23f273fc Separate function out 2021-10-02 17:50:20 +10:00
Serene-Arc
eeb2054606 Switch to yt-dlp 2021-10-02 17:50:20 +10:00
Serene
e004ccd148
Merge pull request #521 from Serene-Arc/bug_fix_518 
Fix bug with different Vidble links
 2021-09-14 13:48:26 +10:00
Serene-Arc
80baab8de7 Fix bug with different Vidble links 2021-09-14 13:47:46 +10:00
Eli Lipsitz
33312687ac imgur: download videos as mp4 instead of gif 
Some imgur URLS have the extension ".gifv" and show up as a gif,
even though they're actually supposed to be mp4 videos. Imgur
serves all videos/gifs as both .gif and .mp4. The image dict has
a key "prefer_video" to distinguish the two. This commit
overrides the .gif extension if "prefer_video" is true to ensure
we download the submission as originally intended.
 2021-09-12 17:30:25 -05:00
Ali Parlakçı
483f179ccc
Merge pull request #482 from Serene-Arc/enhancement_481 
Add ability to read IDs from files
 2021-09-12 20:07:17 +03:00
Serene-Arc
aee6f4add9 Add Vidble to download factory 2021-09-11 12:15:35 +10:00
Serene-Arc
940d646d30 Add Vidble module 2021-09-11 12:13:21 +10:00
Serene
3040a35306
Merge pull request #512 from Serene-Arc/bug_fix_510 2021-09-03 19:30:30 +10:00
Serene-Arc
87f283cc98 Fix backup config location 2021-09-03 19:24:28 +10:00
Serene-Arc
7bca303b1b Add in downloader parameters 2021-07-29 19:10:10 +10:00
Serene-Arc
dbe8733fd4 Refactor method to remove max wait time 2021-07-27 14:02:30 +10:00
Serene-Arc
3cdae99490 Implement callbacks for downloading 2021-07-27 13:39:49 +10:00
Serene-Arc
1a4ff07f78 Add ability to read IDs from files 2021-07-21 17:32:38 +10:00
Serene-Arc
77aaee96f3 Fix bug with deleted galleries 2021-07-19 18:44:54 +10:00
Serene-Arc
2f8ca766c6 Update regex 2021-07-04 11:00:02 +10:00
Serene-Arc
d03a5e556e Stop writing new value to config 2021-07-04 10:59:35 +10:00
Serene-Arc
7f1c929a08 Add fallback scope 2021-07-03 13:54:26 +10:00
Serene-Arc
d5ef991b3a Catch additional error in galleries 2021-07-02 15:11:09 +10:00
Serene-Arc
aa55a92791 Remove unused local variables 2021-07-02 14:58:56 +10:00
Serene-Arc
6efcf1ce7e Remove unused imports 2021-07-02 14:58:20 +10:00
Serene-Arc
1319eeb6da Fix error with crossposted Reddit galleries 2021-07-02 14:53:02 +10:00
Serene-Arc
8db9d0bcc4 Add test for unauthenticated instances 2021-07-02 14:29:39 +10:00
Serene-Arc
bd34c37052 Add exception for special friends subreddit 2021-07-02 14:09:44 +10:00
Serene-Arc
edfeb653a4 Record user flair in comment archive entries 2021-07-02 14:01:24 +10:00
Serene-Arc
d53b3b7274 Update gallery code to work with NSFW galleries 2021-06-25 17:52:11 +10:00
Serene-Arc
e8998da2f0 Catch some Imgur errors with weird links 2021-06-25 17:52:11 +10:00
Serene-Arc
1a52dfdcbc Add PornHub module 2021-06-25 17:47:49 +10:00
Serene-Arc
e5be624f1e Check submission URL against filter before factory 2021-06-23 14:30:39 +10:00
Ali Parlakçı
71930e06a8
Merge pull request #457 from Serene-Arc/enhancement_322 
Add option for archiver full context
 2021-06-19 13:56:04 +03:00
Serene-Arc
7c27b7bf12 Update logging message 2021-06-13 09:49:42 +10:00
Serene-Arc
c5c010bce0 Rename option 2021-06-12 10:35:31 +10:00
Serene-Arc
6eeadc8821 Add option for archiver full context 2021-06-11 15:31:11 +10:00
Serene-Arc
8ba2d0bb55 Add missing return statement 2021-06-10 18:59:22 +10:00
Serene-Arc
8be3efb6e4 Fix bug with Imgur gifs being shortened too much 
The rstrip function was used wrongly, it doesn't remove a substring but
rather removes any of the characters provided, so here it removed any I,
G, V, or F that finished the six character ID for Imgur, resulting in a
404 error for the resources in question.
 2021-06-08 13:08:39 +10:00
Serene
6dcef83666
Add ability to disable modules (#434) 
* Fix test name to match standard

* Rename file

* Add ability to disable modules

* Update README

* Fix missing comma

* Fix more missing commas. sigh...

Co-authored-by: Ali Parlakçı <parlakciali@gmail.com>
 2021-06-06 13:47:56 +03:00
Serene
434aeb8feb
Add a combined command for the archiver and downloader: clone (#433) 
* Simplify downloader function

* Add basic scraper class

* Add "scrape" command

* Rename "scrape" command to "clone"

* Add integration tests for clone command

* Update README

* Fix failing test
 2021-06-06 13:29:09 +03:00
Ali Parlakçı
a2f010c40d
Merge pull request #432 from Serene-Arc/enhancement_429 
Allow --user to be specified multiple times
 2021-06-06 13:25:00 +03:00
Ali Parlakçı
6839c65bd6
Merge pull request #396 from Serene-Arc/bug_fix_385 
Add path limit check
 2021-06-06 13:24:22 +03:00
Serene-Arc
79fba4ac4a Fix indent 2021-05-31 13:42:41 +10:00
Serene-Arc
9a1e1ebea1 Add path limit fix 2021-05-27 15:27:02 +10:00
Serene-Arc
6b78a23484 Allow --user to be specified multiple times 2021-05-27 15:22:58 +10:00
Serene-Arc
fef2fc864b Update blacklist 2021-05-25 19:33:32 +10:00
Serene-Arc
87959028e5 Add blacklist for web filetypes 2021-05-25 19:20:51 +10:00
Serene-Arc
f47688812d Rename function 2021-05-25 18:51:24 +10:00
Serene-Arc
323b2d2b03 Fix download retries logic 2021-05-25 09:56:22 +10:00
Serene-Arc
e2582ecb3e Catch error with MacOS writing per issue #407 2021-05-23 12:17:14 +10:00
Serene-Arc
47a4951279 Rename variable 2021-05-23 12:13:44 +10:00
Serene-Arc
4395dd4646 Update logging messages to include submission IDs 2021-05-22 11:53:44 +10:00