Spider/bulk-downloader-for-reddit
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
741 Commits 2 Branches 45 Tags 14 MiB
c5c010bce0
Commit Graph

741 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ali Parlakci
6c98829fd8 test_imgur.py: Remove duplicate tests 2021-04-18 16:44:51 +03:00
Serene-Arc
bf078cbaf8 Switch to rotating log files 2021-04-18 16:44:51 +03:00
Serene
702295f1ea Allow multilevel folder schemes (#251) 
* Improve some formatting

* Allow multilevel folder names
 2021-04-18 16:44:51 +03:00
Serene-Arc
dc9d02a28c Name log file with run time 2021-04-18 16:44:51 +03:00
Serene-Arc
f78856315d Switch to f-string 2021-04-18 16:44:51 +03:00
Serene-Arc
1e1dae8eaa Alter some logging messages 2021-04-18 16:44:51 +03:00
Serene
2385867afb Refactor and optimise some tests (#245) 
* Rename to follow actual directory name

* Remove unnecessary Reddit call from test

* Refactor test to reduce duplication

* Parameterise some tests

* Standardise formatting
 2021-04-18 16:44:51 +03:00
Serene-Arc
a05fa1a965 Add some to CONTRIBUTING 2021-04-18 16:44:51 +03:00
BlipRanger
c9c864b71b Update regex to be more concise, add test for m.youtube. 2021-04-18 16:44:51 +03:00
BlipRanger
a49d87e154 Add logic to handle mobile youtube (m.youtube) links. 2021-04-18 16:44:51 +03:00
Serene
32c9d6184c Archiver is smarter for comments (#242) 
* Add comment name generation to file name formatter

* Refactor to reduce duplication

* Refactor archive entry classes

* Refactor archiver class a bit

* Refactor method

* Fix comment retrieval

* Add comment-downloading to archiver

* Update test

* Update test
 2021-04-18 16:44:51 +03:00
Ali Parlakci
75d74a5362 test_download_factory: remove outdated marks 2021-04-18 16:44:51 +03:00
Ali Parlakci
ab9dea0347 test_download_factory.py: hardcode submission links 2021-04-18 16:44:51 +03:00
Serene-Arc
7d69c9a7af Fix typos 2021-04-18 16:44:51 +03:00
Serene-Arc
a170356738 Use slice to shorten name 2021-04-18 16:44:51 +03:00
Ali Parlakci
f06e8f3ac4 Revert "Use .format() instead of regular expression" 
This reverts commit 8e82252832.
 2021-04-18 16:44:51 +03:00
Ali Parlakci
0fe28deee9 file_name_formatter: added tests (#227) 2021-04-18 16:44:51 +03:00
Ali Parlakci
a32dd6d0fe Use regex to strip id 2021-04-18 16:44:51 +03:00
Ali Parlakci
7a436d0481 Use .format() instead of regular expression 2021-04-18 16:44:51 +03:00
Serene
3d2e11dc1d Add warning for non-unique file name schemes (#233) 
* Add warning for non-unique file name schemes

* Update README with warning
 2021-04-18 16:44:51 +03:00
Serene
44889d5264 Tweak regex to fix incorrect match (#237) 2021-04-18 16:44:51 +03:00
Serene-Arc
5d86c2d400 Update out-of-date tests 2021-04-18 16:44:51 +03:00
Ali Parlakci
2e879949f5 Send HTTP200 when oauth2 is successful 2021-04-18 16:44:51 +03:00
Ali Parlakçı
265505efc7 Fix missing links in self post footer (#235) 2021-04-18 16:44:51 +03:00
BlipRanger
5e6159ade3 Add created date to archive output 2021-04-18 16:44:51 +03:00
Serene-Arc
cbcbf400bb Add test for nonexistent key 2021-04-18 16:44:51 +03:00
Serene-Arc
f0d3cfefc0 Remove splice and fix quotes 2021-04-18 16:44:51 +03:00
Serene-Arc
8bd4b8b3a9 Split regex for download factory 2021-04-18 16:44:51 +03:00
Serene-Arc
8753fa0e45 Add additional test 2021-04-18 16:44:51 +03:00
Ali Parlakci
49b0fac7c7 Add necessary tests 2021-04-18 16:44:51 +03:00
Ali Parlakci
247fe3e6f7 Fix downloading .gifv instead of .mp4 (#228) 2021-04-18 16:44:51 +03:00
Ali Parlakci
58150570a3 Tests added 2021-04-18 16:44:51 +03:00
Ali Parlakci
443cf1af01 Bug fix on deleted self posts 2021-04-18 16:44:51 +03:00
Serene
f2946c0a87 Strip emojis from filenames on Windows (#222) 2021-04-18 16:44:51 +03:00
Serene-Arc
20f525bd0d Update some logging levels 2021-04-18 16:44:51 +03:00
Serene-Arc
91bebe5f68 Add some to ARCHITECTURE 2021-04-18 16:44:51 +03:00
Serene
21e8f0f8b9 Add option to exclude submission IDs (#220) 
* Add option to exclude submission IDs

* Update README

* Update logging message
 2021-04-18 16:44:51 +03:00
Serene-Arc
56347da07e Fix test creating unwanted folders 2021-04-18 16:44:51 +03:00
Serene-Arc
3cb51e638b Preserve appended ID when shortening file names 2021-04-18 16:44:51 +03:00
Serene-Arc
1cb8240eb6 Remove incorrect test marks 2021-04-18 16:44:51 +03:00
Serene-Arc
771cc711e4 Calculate existing hashes in parallel 2021-04-18 16:44:51 +03:00
Serene-Arc
79105f9f84 Fix typo 2021-04-18 16:44:51 +03:00
Serene-Arc
0007912ad5 Scrub windows paths for invalid characters 2021-04-18 16:44:51 +03:00
Serene-Arc
b3c51f49ce Copy config file to config directory first time 2021-04-18 16:44:51 +03:00
Serene-Arc
d8b741191d Fix typos 2021-04-18 16:44:51 +03:00
Serene
82860a3341 Add specific requirements (#211) 2021-04-18 16:44:51 +03:00
Serene
5b23f6688a Filter non-submissions (#212) 2021-04-18 16:44:51 +03:00
Ali Parlakçı
7174121af2 Add v2-beta to title 2021-04-18 16:44:51 +03:00
Serene-Arc
c13f2806fa Fix case where Gfycat fails after redirect 2021-04-18 16:44:51 +03:00
Serene-Arc
0d78e16b2d Make sure to log all exceptions to file 2021-04-18 16:44:51 +03:00