Spider/bulk-downloader-for-reddit
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
935 Commits 2 Branches 45 Tags 14 MiB
395bf9180a
Commit Graph

137 Commits

Author SHA1 Message Date
Serene
e004ccd148
Merge pull request #521 from Serene-Arc/bug_fix_518 
Fix bug with different Vidble links
 2021-09-14 13:48:26 +10:00
Serene-Arc
80baab8de7 Fix bug with different Vidble links 2021-09-14 13:47:46 +10:00
Eli Lipsitz
33312687ac imgur: download videos as mp4 instead of gif 
Some imgur URLS have the extension ".gifv" and show up as a gif,
even though they're actually supposed to be mp4 videos. Imgur
serves all videos/gifs as both .gif and .mp4. The image dict has
a key "prefer_video" to distinguish the two. This commit
overrides the .gif extension if "prefer_video" is true to ensure
we download the submission as originally intended.
 2021-09-12 17:30:25 -05:00
Ali Parlakçı
483f179ccc
Merge pull request #482 from Serene-Arc/enhancement_481 
Add ability to read IDs from files
 2021-09-12 20:07:17 +03:00
Serene-Arc
940d646d30 Add Vidble module 2021-09-11 12:13:21 +10:00
Serene-Arc
edc2db0ded Update test 2021-09-09 13:50:03 +10:00
Serene-Arc
56575dc390 Add NSFW search test 2021-09-09 13:44:01 +10:00
Serene-Arc
defd6bca77 Tweak test conditions 2021-09-09 13:44:01 +10:00
Serene-Arc
afc2a6416b Add integration test 2021-09-09 13:44:01 +10:00
Serene-Arc
dbe8733fd4 Refactor method to remove max wait time 2021-07-27 14:02:30 +10:00
Serene-Arc
3cdae99490 Implement callbacks for downloading 2021-07-27 13:39:49 +10:00
Serene-Arc
44453b1707 Update tests 2021-07-27 13:12:50 +10:00
Serene-Arc
1a4ff07f78 Add ability to read IDs from files 2021-07-21 17:32:38 +10:00
Serene-Arc
77aaee96f3 Fix bug with deleted galleries 2021-07-19 18:44:54 +10:00
Serene-Arc
8826fc5aa9 Fix outdated test 2021-07-18 14:42:20 +10:00
Serene-Arc
381e3c29fa Fix test where comments in saved list 2021-07-18 14:42:10 +10:00
Serene-Arc
6efcf1ce7e Remove unused imports 2021-07-02 14:58:20 +10:00
Serene-Arc
390ce57f46 Remove redundant parenthesis 2021-07-02 14:57:10 +10:00
Serene-Arc
1319eeb6da Fix error with crossposted Reddit galleries 2021-07-02 14:53:02 +10:00
Serene-Arc
8db9d0bcc4 Add test for unauthenticated instances 2021-07-02 14:29:39 +10:00
Serene-Arc
c4aa617737 Add test for friends subreddit 2021-07-02 14:17:13 +10:00
Serene-Arc
469a7783b8 Split integration tests 2021-07-02 14:00:48 +10:00
Serene-Arc
ffd07f38ba Fix broken subreddit test 2021-06-30 17:53:25 +10:00
Serene-Arc
528f5c567d Add additional test for Redgifs 2021-06-30 11:59:38 +10:00
Serene-Arc
d53b3b7274 Update gallery code to work with NSFW galleries 2021-06-25 17:52:11 +10:00
Serene-Arc
e8998da2f0 Catch some Imgur errors with weird links 2021-06-25 17:52:11 +10:00
Serene-Arc
1a52dfdcbc Add PornHub module 2021-06-25 17:47:49 +10:00
Serene-Arc
8b1a3d9abc Enable integration tests to be run concurrently 2021-06-24 16:38:34 +10:00
Serene-Arc
640001a7f5 Speed up test 2021-06-24 16:37:25 +10:00
Serene-Arc
1d187fcf65 Consolidate tests 2021-06-24 16:14:21 +10:00
Serene-Arc
31be3a916e Enable integration tests to be run concurrently 2021-06-24 16:14:05 +10:00
Serene-Arc
3dacaf0872 Fix renamed option in test 2021-06-24 16:12:41 +10:00
Serene-Arc
ccafebf5fe Update test 2021-06-23 14:59:26 +10:00
Serene-Arc
e5be624f1e Check submission URL against filter before factory 2021-06-23 14:30:39 +10:00
Serene-Arc
6eeadc8821 Add option for archiver full context 2021-06-11 15:31:11 +10:00
Serene-Arc
8be3efb6e4 Fix bug with Imgur gifs being shortened too much 
The rstrip function was used wrongly, it doesn't remove a substring but
rather removes any of the characters provided, so here it removed any I,
G, V, or F that finished the six character ID for Imgur, resulting in a
404 error for the resources in question.
 2021-06-08 13:08:39 +10:00
Serene
6dcef83666
Add ability to disable modules (#434) 
* Fix test name to match standard

* Rename file

* Add ability to disable modules

* Update README

* Fix missing comma

* Fix more missing commas. sigh...

Co-authored-by: Ali Parlakçı <parlakciali@gmail.com>
 2021-06-06 13:47:56 +03:00
Serene
434aeb8feb
Add a combined command for the archiver and downloader: clone (#433) 
* Simplify downloader function

* Add basic scraper class

* Add "scrape" command

* Rename "scrape" command to "clone"

* Add integration tests for clone command

* Update README

* Fix failing test
 2021-06-06 13:29:09 +03:00
Ali Parlakçı
a2f010c40d
Merge pull request #432 from Serene-Arc/enhancement_429 
Allow --user to be specified multiple times
 2021-06-06 13:25:00 +03:00
Serene-Arc
6caa02adb1 Update failing test 2021-05-27 15:58:40 +10:00
Serene-Arc
12a508c898 Update failing test 2021-05-27 15:29:43 +10:00
Serene-Arc
bf50618590 Add macOS value to test 2021-05-27 15:27:02 +10:00
Serene-Arc
c89de29f72 Update test condition 2021-05-27 15:27:02 +10:00
Serene-Arc
9a1e1ebea1 Add path limit fix 2021-05-27 15:27:02 +10:00
Serene-Arc
6b78a23484 Allow --user to be specified multiple times 2021-05-27 15:22:58 +10:00
Serene-Arc
87959028e5 Add blacklist for web filetypes 2021-05-25 19:20:51 +10:00
Serene-Arc
f47688812d Rename function 2021-05-25 18:51:24 +10:00
Serene-Arc
b74e93d2b7 Fix typo in test name 2021-05-25 18:51:00 +10:00
Serene-Arc
4c42469c0c Update failing tests 2021-05-24 12:19:15 +10:00
Ali Parlakci
cf6905db28
Reverts #384 2021-05-21 00:22:44 +03:00
Ali Parlakçı
bfa6e4da5a
Merge pull request #409 from Ailothaen/master 
Adding info to threads and comments: distinguished, spoiler, pinned, locked
 2021-05-21 00:04:53 +03:00
Ailothaen
f4c1adaa9a Added tests for new data in threads/comments (2/2) 2021-05-20 22:47:33 +02:00
Ailothaen
9d6e54148b Added tests for new data in threads/comments 2021-05-20 22:41:02 +02:00
Serene-Arc
3b28ad24b3 Fix bug with some Imgur extensions 2021-05-19 09:57:27 +10:00
Serene-Arc
122aa2839b Remove bad test case 2021-05-18 12:06:10 +10:00
Ali Parlakci
a75cc0dee9 Add file modified date test 2021-05-17 22:05:33 +10:00
Serene-Arc
fa04d61eb8 Refactor archiver tests 2021-05-17 11:50:17 +10:00
Serene-Arc
7016603763 Refactor out super class RedditConnector 2021-05-17 11:50:17 +10:00
Serene-Arc
71da1556e5 Change out of date test case 2021-05-17 11:15:50 +10:00
Ali Parlakci
200916a150 Rename --exclude-id(-file) to --skip-id(-file) 2021-05-17 10:30:55 +10:00
Ali Parlakci
f768a7d61c Rename --skip to --skip-format 2021-05-17 10:30:55 +10:00
alpbetgam
ef37712115 Fix error with old gfycat/redgifs urls 2021-05-16 19:15:36 +10:00
Ali Parlakci
17226a4f0b refactor(youtube.dl): Remove slow parameter 2021-05-16 11:06:13 +10:00
Ali Parlakci
c7a5ec4376 bug(youtube.dl): Fix crash on zero downloads #375 2021-05-16 11:06:13 +10:00
Ali Parlakçı
a2e22e894a
Fix xml archiver encoding bug (#349) 
* test_integration: add archiver tests

* archiver.py: fix encoding bug in xml archiver
 2021-05-06 16:11:48 +03:00
Serene-Arc
fc5f4a0405 Fix log access issue for archive integration tests 2021-05-06 10:06:55 +10:00
Serene-Arc
9752ef4b2a Make filename test OS-agnostic 2021-05-04 15:47:35 +10:00
Serene-Arc
c9cde54a72 Remove VReddit downloader module 2021-05-03 14:10:54 +10:00
Serene-Arc
ab96a3ba97 Remove Streamable downloader module 2021-05-03 14:10:54 +10:00
Serene-Arc
a8c2136270 Add fallback downloader 2021-05-03 14:10:54 +10:00
Serene-Arc
afa3e2548f Add customisable time formatting 2021-05-03 14:05:05 +10:00
Serene-Arc
eda12e5274 Make downloadfilter apply itself to Resources 2021-05-03 14:02:03 +10:00
Serene-Arc
711f8b0c76 Add exception for r/all in subreddit check 2021-05-02 14:00:23 +10:00
Serene-Arc
14195157de Catch errors for banned or private subreddits 2021-05-01 13:36:38 +10:00
Daniel Clowry
fe95394b3b Match import order, update docs 2021-04-30 09:22:26 +10:00
Daniel Clowry
310cc123df Add Streamable to download factory test 2021-04-30 09:22:26 +10:00
Daniel Clowry
600a85cbc8 Add Streamable tests 2021-04-30 09:22:26 +10:00
Serene-Arc
39935c58d9 Remove GifDeliveryNetwork module 2021-04-28 19:08:31 +10:00
Serene-Arc
9931839d14 Remove gifdeliverynetwork from download factory 2021-04-28 19:08:31 +10:00
Serene-Arc
17499baf61 Add informative error when testing user existence 2021-04-28 12:03:28 +10:00
Serene-Arc
e6551bb797 Return banned users as not existing 2021-04-28 12:03:28 +10:00
Serene-Arc
cb41d4749a Add option to specify logfile location 2021-04-28 12:03:28 +10:00
Serene-Arc
3bc10ce1aa Fix formatting 2021-04-28 12:03:28 +10:00
Serene-Arc
fbf8a2748e Fix formatting 2021-04-28 12:03:28 +10:00
Serene-Arc
386d5ea41c Format file 2021-04-28 12:03:28 +10:00
Serene-Arc
d960bc0b7b Use ISO format for timestamps in names 2021-04-28 12:03:28 +10:00
Ali Parlakci
1c4cfbb580 tests: move outside of the package 2021-04-21 08:29:14 +10:00