Commit graph

319 commits

Author SHA1 Message Date
Yen Chi Hsuan 582be35847
Update coding style after pycodestyle 2.1.0
In pycodestyle 2.1.0, E305 was introduced, which requires two blank
lines after top level declarations, too.

See https://github.com/PyCQA/pycodestyle/issues/400

See also #10689; thanks @stepshal for first mentioning this issue and
initial patches
2016-11-17 19:45:42 +08:00
Sergey M․ 690355551c
[downoader/fragment,f4m,hls] Add internal support for custom HTTP headers 2016-11-13 22:22:10 +07:00
Sergey M․ 9d64e1dcdc
[downloader/ism] Fix typo 2016-11-03 22:15:09 +07:00
Remita Amine 10380e55de [downloader/ism] fix AVC Decoder Configuration Record creation in python 3 2016-11-03 16:08:57 +01:00
Remita Amine b2758123c5 add Basic support for Smooth Streaming protocol(#8118) 2016-11-02 01:54:45 +07:00
Sergey M․ bcd6276520
[downloader/common] Remove debug output 2016-10-12 21:22:33 +07:00
Yen Chi Hsuan d54739a2e6
[downloader/http] xattr values should be bytes 2016-10-01 19:58:13 +08:00
Yen Chi Hsuan d7753d1948
[downloader/http] Use write_xattr function for --xattr-set-filesize 2016-10-01 14:47:20 +08:00
Remita Amine 6f126d903f [download/hls] Delegate downloading to ffmpeg for live streams 2016-09-24 15:39:47 +01:00
Sergey M․ 86f4d14f81
Refactor fragments interface and dash segments downloader
- Eliminate segment_urls and initialization_url
+ Introduce manifest_url (manifest may contain unfragmented data in this case url will be used for direct media URL and manifest_url for manifest itself correspondingly)
* Rewrite dashsegments downloader to use fragments data
* Improve generic mpd extraction
2016-09-17 20:35:22 +07:00
Yen Chi Hsuan 919cf1a62f
[downloader/dash] Abort if the first segment fails
Closes #10497, Closes #10542
2016-09-04 17:32:29 +08:00
Sergey M․ 4a69fa04e0
[downloader/dash] Abort download immediately after giving up on some fragment 2016-09-03 17:51:48 +07:00
Sergey M․ 2e99cd30c3
[downloader/dash:hls] Report exact fragment error on retry 2016-09-03 17:51:48 +07:00
Sergey M․ 25afc2a783
[downloader/dash:hls] Respect --fragment-retries and --skip-unavailable-fragments (Closes #10165, closes #10448) 2016-09-03 17:51:48 +07:00
Sergey M․ 9603b66012
Introduce --skip-unavailable-fragments 2016-09-03 17:51:48 +07:00
Yen Chi Hsuan 6c9b71bc08
[downloader/external] Recommend --hls-prefer-native for SOCKS users
Related: #10490
2016-08-29 19:05:38 +08:00
Remita Amine b8079a40bc [turner] fix secure m3u8 formats downloading 2016-08-28 17:51:53 +01:00
Yen Chi Hsuan 20bad91d76
[downloader/external] Clarify that ffmpeg doesn't support SOCKS
Ref: #10304
2016-08-25 22:38:06 +08:00
Remita Amine 07ea9c9b05 [downloader/hls] fill IV with zeros for IVs shorter than 16-octet 2016-08-25 08:49:44 +01:00
Remita Amine aaf44a2f47 [uplynk] Add new extractor 2016-08-13 22:53:41 +01:00
Remita Amine acfccacad5 [downloader/external:curl] Clarify why CurlFD should not capture stderr 2016-08-13 10:26:02 +01:00
Remita Amine 98e698f1ff [external/curl] respect more downloader options and display progress 2016-08-12 12:30:02 +01:00
Sergey M․ 1ad6b891b2
Add more checks for --min/max-sleep-interval arguments and use more idiomatic naming 2016-08-09 03:47:56 +07:00
singh-pratyush96 065bc35489
Add --max-sleep-interval (Closes #9930) 2016-08-09 03:32:42 +07:00
Yen Chi Hsuan 477b7a8474
[downloader/f4m] Fix for Rai live streams 2016-07-03 19:26:39 +08:00
Sergey M․ 8369a4fe76
[downloader/hls] Simplify and carry long lines 2016-06-20 21:55:17 +07:00
remitamine e154c65128 [downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader 2016-06-19 01:01:40 +01:00
Sergey M․ e69f9f5d68
[downloader/external] Decode error string before writing to stderr 2016-06-12 16:45:07 +07:00
Yen Chi Hsuan 50ce1c331c
[downloader/external] Add another env for proxies in ffmpeg/avconv
Related sources:
https://git.libav.org/?p=libav.git;a=blob;f=libavformat/http.c;h=8fe8d11e1edfdbb04a8726db2c49cfef3f572aac;hb=HEAD#l152
https://git.libav.org/?p=libav.git;a=blob;f=libavformat/tls.c;h=fab243e93e20034e88e619188c13a44a5d8ccdb9;hb=HEAD#l63
https://github.com/FFmpeg/FFmpeg/blob/f8e89d8/libavformat/http.c#L191
https://github.com/FFmpeg/FFmpeg/blob/f8e89d8/libavformat/tls.c#L92
2016-06-08 14:43:52 +08:00
Sergey M․ 633b444fd2
[downloader/hls] Correct comment on twitch vods 2016-06-05 03:31:10 +07:00
Sergey M․ 51c4d85ce7
[downloader/hls] PEP 8 2016-06-05 03:21:43 +07:00
Sergey M․ 1e236d7e23
[downloader/hls] Do not rely on EXT-X-PLAYLIST-TYPE:EVENT 2016-06-05 03:16:05 +07:00
Sergey M․ 2615fa7584
[downloader/f4m] Simply select format when it's the only one 2016-05-27 01:46:12 +06:00
Yen Chi Hsuan 1b405bb47d
[downloader/f4m] Tolerate truncate segments when testing
Replaces #9216

Fixes #9214 and test_Bloomberg partially
2016-05-12 20:02:36 +08:00
Yen Chi Hsuan edaa23f822
[compat] Rename struct_(un)pack to compat_struct_(un)pack 2016-05-10 14:51:38 +08:00
Yen Chi Hsuan dab0daeeb0
[utils,compat] Move struct_pack and struct_unpack to compat.py 2016-05-10 14:51:38 +08:00
Sergey M․ 2937590e8b
[downloader/hls] PEP 8 2016-05-09 22:16:33 +06:00
Sergey M․ e62d9c5caa
[downloader/external] Call ffmpeg with with HTTP_PROXY env variable set (#9437) 2016-05-09 22:05:12 +06:00
Sergey M․ 6104cc2985
[downloader/hls] Add event media playlists to unsupported features of hlsnative 2016-05-09 20:55:37 +06:00
Sergey M․ c15c47d19b
[downloader/hls] Remove EXT-X-MEDIA-SEQUENCE from unsupported features for hlsnative 2016-05-09 20:45:03 +06:00
Sergey M eb785b856f Merge pull request #9358 from dstftw/hls-native-to-ffmpeg-delegation
[downloader/hls] Delegate extraction to ffmpeg when unsupported features detected
2016-05-08 22:07:55 +00:00
Sergey M․ f5436c5d9e
[downloader/external] Add temp fix ffmpeg m3u8 downloads (Closes #9394) 2016-05-08 02:29:26 +06:00
Sergey M․ 0d66bd0eab
[downloader/hls] Delegate extraction to ffmpeg when unsupported features detected 2016-05-01 13:56:51 +06:00
Sergey M․ bf09af3acb
Add --hls-prefer-ffmpeg 2016-04-21 23:02:17 +06:00
remitamine d868f43c58 [ffmpeg] check for - file name in _ffmpeg_filename_argument 2016-04-16 19:45:56 +01:00
remitamine ce599d5a7e [downloader/external] enable piping for FFmpegFD(closes #2124) 2016-04-14 18:49:02 +01:00
Yen Chi Hsuan f5d8743e0a
[downloader/rtsp] Print the command 2016-04-14 15:07:31 +08:00
Sergey M․ f7df343b4a [downloader/f4m] Extract routine for removing unsupported encrypted media 2016-03-27 07:41:19 +06:00
Sergey M․ 16a8b7986b [downloader/fragment] Document fragment_retries 2016-03-19 20:54:21 +06:00
Sergey M․ 617e58d850 [downloader/{common,fragment}] Fix total retries reporting on python 2.6 2016-03-19 20:51:30 +06:00
Sergey M․ e33baba0dd [downloader/dash] Add fragment retry capability
YouTube may often return 404 HTTP error for a fragment causing the
whole download to fail. However if the same fragment is immediately
retried with the same request data this usually succeeds (1-2 attemps
is usually enough) thus allowing to download the whole file successfully.
So, we will retry all fragments that fail with 404 HTTP error for now.
2016-03-19 20:42:23 +06:00
Sergey M․ 721f26b821 [downloader/fragment] Add report_retry_fragment 2016-03-19 20:41:24 +06:00
remitamine 4230c4894d [external/downloader] fix rtmp downloading using FFmpegFD 2016-03-14 16:51:01 +01:00
remitamine 2e7e561c1d Merge pull request #8611 from remitamine/ffmpegfd
[downloader/external] Add FFmpegFD
2016-03-13 21:30:27 +01:00
remitamine d8515fd41c [downloader/external] pass configuration args to ffmpeg 2016-03-13 21:28:26 +01:00
remitamine 694c47b261 [external/downloader] don't pass -t and -ss to ffmpeg 2016-03-13 21:28:16 +01:00
remitamine 77dea16ac8 [downloader/external] check for ffmpeg availablity when it used for m3u8 download 2016-03-13 20:34:51 +01:00
remitamine 6ae27bed01 [download/external] move the check for multiple selected formats to get_suitable_downloader 2016-03-13 20:34:38 +01:00
remitamine da1973a038 [extractor/__init__] disable time range downloading 2016-03-13 16:16:26 +01:00
remitamine be24916a7f [downloader/rtsp] Add rtsp and mms downloader 2016-03-13 15:24:02 +01:00
remitamine 2cb99ebbd0 [downloader/external] add can_download mathod for checking downloader availibilty and support 2016-03-13 15:18:51 +01:00
remitamine 91ee320bfa [downloader/external] wrap available_opt in a list 2016-03-13 14:37:45 +01:00
remitamine 634415ca17 [downloader/external] skip FFmpegFD when requesting multiple formats 2016-03-13 12:23:10 +01:00
remitamine a755f82549 [ffmpeg] convert format ext to ffmpeg output formats codes 2016-03-13 12:15:29 +01:00
Sergey M․ 1b5284b13f [downloader/fragment] Make speed more smooth
At the beginning of every segment there was a drop to Unknown speed due to timeslice being too small to calculate speed.
Now last speed from the previous fragment is used.
2016-03-06 05:36:52 +06:00
Yen Chi Hsuan e9c0cdd389 [jython] Introduce compat_os_name
os.name is always 'java' on Jython
2016-03-03 19:24:24 +08:00
remitamine f34294fa0c [downloader/external:ffmpegfd] check for None value of start_time 2016-02-20 08:06:12 +01:00
remitamine 99cbe98ce8 [downloader/external] check for external downloaders availability 2016-02-20 07:58:25 +01:00
remitamine 12b84ac8c1 [downloader/external] Add FFmpegFD(fixes #622)
- replace HlsFD and RtspFD
- add basic support for downloading part of the video or audio
2016-02-19 19:29:24 +01:00
Sergey M․ 611c1dd96e [refactor] Single quotes consistency 2016-02-14 15:37:17 +06:00
Sergey M․ d800609c62 [refactor] Do not specify redundant None as second argument in dict.get() 2016-02-14 14:25:04 +06:00
Sergey M․ c78c9cd10d [downloader/dash] PEP 8 2016-02-14 14:13:09 +06:00
remitamine c43fe0268c [downloader/dash] Implement dashsegments fd in terms of fragment fd 2016-02-09 17:25:44 +01:00
Sergey M․ 56401e1e5f [downloader/hls] Do not send 'q' to ffmpeg on Windows (Closes #8300) 2016-02-06 23:24:22 +06:00
remitamine a1a22572fb [downloader/dash] make initialization_url optional 2016-02-02 18:12:32 +01:00
Sergey M․ b8c9926c0a [downloader/f4m] Do not update fragment list while test 2016-01-30 19:43:25 +06:00
Sergey M․ 2c2f1efdcd [downloader/fragment] Remove superfluous whitespace 2016-01-30 19:30:31 +06:00
Sergey M․ 09104e9930 [downloader/f4m] Add live stream flag to context
Now download progress for f4m livestreams is reported correctly
2016-01-30 19:22:15 +06:00
Sergey M․ 5fa1702ca6 [downloader/fragment] Do not report total bytes estimation and eta for live streams 2016-01-30 19:20:52 +06:00
Sergey M․ 8ab3fe81d8 [downloader/f4m] Prefer bootstrap url attribute over inline bootstrap info 2016-01-30 18:28:38 +06:00
Jaime Marquínez Ferrándiz 7d106a65ca Add --hls-use-mpegts option
When using the mpegts container hls vidoes can be played while being downloaded (useful if you are recording a live stream).
VLC and mpv play them file, but QuickTime doesn't.
2016-01-30 12:26:40 +01:00
Jaime Marquínez Ferrándiz 32d77eeb04 [downloader/common] report_retry: Don't crash when retries is infinite (fixes #8299) 2016-01-22 14:49:17 +01:00
Jaime Marquínez Ferrándiz 9d90e7de03 [downloader/hls] Ask ffmpeg to quit when interrupting youtube-dl with 'Ctrl+C' (#8252)
Otherwise the mp4 file can't be played.
2016-01-19 22:07:14 +01:00
Sergey M․ b83b782dc4 [downloader/fragment] Move helper data to context dict 2016-01-13 00:00:31 +06:00
Sergey M․ 709185a264 [downloader/fragment] More smooth calculations
`downloaded_bytes` is now updated on each fragment progress hook invocation
2016-01-12 23:18:38 +06:00
Sergey M․ 9cb1a06b6c [downloader/fragment] Remove unused code and fix zero division error 2016-01-12 22:09:38 +06:00
Jaime Marquínez Ferrándiz 3c91e41614 [downloader/fragment] Don't fail if the 'Content-Length' header is missing
In some dailymotion videos (like http://www.dailymotion.com/video/x3k0dtv from #8156) the segments URLs don't have the 'Content-Length' header and HttpFD sets the 'totat_bytes' field to None, so we also use '0' in that case (since we do different math operations with it).
2016-01-10 14:41:38 +01:00
Jaime Marquínez Ferrándiz 3f17c357d9 [downloader/hls] Don't let ffmpeg read from stdin (#8139)
If you run 'while read aurl ; do youtube-dl "${aurl}"; done < path_to_batch_file'  (batch_file contains one url per line that uses the hls downloader) each call to youtube-dl consumed some characters and 'read' would assing to 'aurl' a non valid url

(This is the same problem that was fixed for the ffmpeg postprocessors in cffcbc02de)
2016-01-04 18:35:31 +01:00
Sergey M․ 9b9c5355e4 Rename error_to_str to error_to_compat_str 2015-12-20 07:00:39 +06:00
Sergey M․ 7f8b271465 Properly convert errors to strings 2015-12-20 05:27:38 +06:00
Sergey M․ 4db43567e8 [downloader/f4m] Decode manifest before fixing 2015-12-11 20:28:44 +06:00
Sergey M․ 17b786ae73 [downloader/f4m] Fix malformed manifests (Closes #7823) 2015-12-10 22:59:50 +06:00
Yen Chi Hsuan 94e8c80473 [downloader/hls] Respect Youtubedl-* headers 2015-11-29 12:43:59 +08:00
Yen Chi Hsuan 9d0e366880 [downloader/hls] Remove Accept-encoding from headers passed to ffmpeg
Fails for Youtube Gaming live streams (#7671)
2015-11-27 21:37:45 +08:00
Sergey M? 5c2266df4b Switch codebase to use sanitized_Request instead of
compat_urllib_request.Request

[downloader/dash] Use sanitized_Request

[downloader/http] Use sanitized_Request

[atresplayer] Use sanitized_Request

[bambuser] Use sanitized_Request

[bliptv] Use sanitized_Request

[brightcove] Use sanitized_Request

[cbs] Use sanitized_Request

[ceskatelevize] Use sanitized_Request

[collegerama] Use sanitized_Request

[extractor/common] Use sanitized_Request

[crunchyroll] Use sanitized_Request

[dailymotion] Use sanitized_Request

[dcn] Use sanitized_Request

[dramafever] Use sanitized_Request

[dumpert] Use sanitized_Request

[eitb] Use sanitized_Request

[escapist] Use sanitized_Request

[everyonesmixtape] Use sanitized_Request

[extremetube] Use sanitized_Request

[facebook] Use sanitized_Request

[fc2] Use sanitized_Request

[flickr] Use sanitized_Request

[4tube] Use sanitized_Request

[gdcvault] Use sanitized_Request

[extractor/generic] Use sanitized_Request

[hearthisat] Use sanitized_Request

[hotnewhiphop] Use sanitized_Request

[hypem] Use sanitized_Request

[iprima] Use sanitized_Request

[ivi] Use sanitized_Request

[keezmovies] Use sanitized_Request

[letv] Use sanitized_Request

[lynda] Use sanitized_Request

[metacafe] Use sanitized_Request

[minhateca] Use sanitized_Request

[miomio] Use sanitized_Request

[meovideo] Use sanitized_Request

[mofosex] Use sanitized_Request

[moniker] Use sanitized_Request

[mooshare] Use sanitized_Request

[movieclips] Use sanitized_Request

[mtv] Use sanitized_Request

[myvideo] Use sanitized_Request

[neteasemusic] Use sanitized_Request

[nfb] Use sanitized_Request

[niconico] Use sanitized_Request

[noco] Use sanitized_Request

[nosvideo] Use sanitized_Request

[novamov] Use sanitized_Request

[nowness] Use sanitized_Request

[nuvid] Use sanitized_Request

[played] Use sanitized_Request

[pluralsight] Use sanitized_Request

[pornhub] Use sanitized_Request

[pornotube] Use sanitized_Request

[primesharetv] Use sanitized_Request

[promptfile] Use sanitized_Request

[qqmusic] Use sanitized_Request

[rtve] Use sanitized_Request

[safari] Use sanitized_Request

[sandia] Use sanitized_Request

[shared] Use sanitized_Request

[sharesix] Use sanitized_Request

[sina] Use sanitized_Request

[smotri] Use sanitized_Request

[sohu] Use sanitized_Request

[spankwire] Use sanitized_Request

[sportdeutschland] Use sanitized_Request

[streamcloud] Use sanitized_Request

[streamcz] Use sanitized_Request

[tapely] Use sanitized_Request

[tube8] Use sanitized_Request

[tubitv] Use sanitized_Request

[twitch] Use sanitized_Request

[twitter] Use sanitized_Request

[udemy] Use sanitized_Request

[vbox7] Use sanitized_Request

[veoh] Use sanitized_Request

[vessel] Use sanitized_Request

[vevo] Use sanitized_Request

[viddler] Use sanitized_Request

[videomega] Use sanitized_Request

[viewvster] Use sanitized_Request

[viki] Use sanitized_Request

[vk] Use sanitized_Request

[vodlocker] Use sanitized_Request

[voicerepublic] Use sanitized_Request

[wistia] Use sanitized_Request

[xfileshare] Use sanitized_Request

[xtube] Use sanitized_Request

[xvideos] Use sanitized_Request

[yandexmusic] Use sanitized_Request

[youku] Use sanitized_Request

[youporn] Use sanitized_Request

[youtube] Use sanitized_Request

[patreon] Use sanitized_Request

[extractor/common] Remove unused import

[nfb] PEP 8
2015-11-23 21:56:23 +06:00
ashutosh-mishra 17cc153435 Typo fix, found while going through the code. 2015-11-20 22:51:46 +05:30
Jaime Marquínez Ferrándiz 36e6f62cd0 Use a wrapper around xml.etree.ElementTree.fromstring in python 2.x (#7178)
Attributes aren't unicode objects, so they couldn't be directly used in info_dict fields (for example '--write-description' doesn't work with bytes).
2015-10-25 20:13:16 +01:00
Sergey M․ 8cd9614abf [downloader/f4m] More accurate fragment URL construction 2015-10-24 21:02:31 +06:00
remitamine 324ac0a243 [downloader/f4m] get the redirected f4m_url and handle url query string properly 2015-10-24 20:05:46 +06:00
Yen Chi Hsuan 985e4fdc07 [downloader/hls] Add headers only for http(s) URLs
ffmpeg 2.8.1 raises an error with -headers and non-http input files.
2015-10-18 13:44:21 +08:00