164 changed files with 3670 additions and 6117 deletions
--- a/.github/ISSUE_TEMPLATE/1_broken_site.md
+++ b/.github/ISSUE_TEMPLATE/1_broken_site.md
@ -18,7 +18,7 @@ title: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.09.20. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.11.28. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
 - Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
@ -26,7 +26,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 - [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running youtube-dl version **2020.09.20**
+- [ ] I've verified that I'm running youtube-dl version **2019.11.28**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@ -41,7 +41,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2020.09.20
+ [debug] youtube-dl version 2019.11.28
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.github/ISSUE_TEMPLATE/2_site_support_request.md
+++ b/.github/ISSUE_TEMPLATE/2_site_support_request.md
@ -19,7 +19,7 @@ labels: 'site-support-request'
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.09.20. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.11.28. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 - [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running youtube-dl version **2020.09.20**
+- [ ] I've verified that I'm running youtube-dl version **2019.11.28**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
--- a/.github/ISSUE_TEMPLATE/3_site_feature_request.md
+++ b/.github/ISSUE_TEMPLATE/3_site_feature_request.md
@ -18,13 +18,13 @@ title: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.09.20. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.11.28. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
 - Finally, put x into all relevant boxes (like this [x])
 -->
 - [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running youtube-dl version **2020.09.20**
+- [ ] I've verified that I'm running youtube-dl version **2019.11.28**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
--- a/.github/ISSUE_TEMPLATE/4_bug_report.md
+++ b/.github/ISSUE_TEMPLATE/4_bug_report.md
@ -18,7 +18,7 @@ title: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.09.20. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.11.28. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
 - Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 - [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running youtube-dl version **2020.09.20**
+- [ ] I've verified that I'm running youtube-dl version **2019.11.28**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@ -43,7 +43,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2020.09.20
+ [debug] youtube-dl version 2019.11.28
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.github/ISSUE_TEMPLATE/5_feature_request.md
+++ b/.github/ISSUE_TEMPLATE/5_feature_request.md
@ -19,13 +19,13 @@ labels: 'request'
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.09.20. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.11.28. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
 - Finally, put x into all relevant boxes (like this [x])
 -->
 - [ ] I'm reporting a feature request
- [ ] I've verified that I'm running youtube-dl version **2020.09.20**
+- [ ] I've verified that I'm running youtube-dl version **2019.11.28**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
--- a/.travis.yml
+++ b/.travis.yml
@ -13,7 +13,7 @@ dist: trusty
 env:
  - YTDL_TEST_SET=core
  - YTDL_TEST_SET=download
-jobs:
+matrix:
  include:
    - python: 3.7
      dist: xenial
@ -35,11 +35,6 @@ jobs:
      env: YTDL_TEST_SET=download
    - env: JYTHON=true; YTDL_TEST_SET=core
    - env: JYTHON=true; YTDL_TEST_SET=download
    - name: flake8
      python: 3.8
      dist: xenial
      install: pip install flake8
      script: flake8 .
  fast_finish: true
  allow_failures:
    - env: YTDL_TEST_SET=download
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@ -153,7 +153,7 @@ After you have ensured this site is distributing its content legally, you can fo
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in.
 7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](http://flake8.pycqa.org/en/latest/index.html#quickstart):
        $ flake8 youtube_dl/extractor/yourextractor.py
--- a/422
+++ b/422
@ -1,421 +1,3 @@
 version 2020.09.20
 Core
 * [extractor/common] Relax interaction count extraction in _json_ld
 + [extractor/common] Extract author as uploader for VideoObject in _json_ld
 * [downloader/hls] Fix incorrect end byte in Range HTTP header for
  media segments with EXT-X-BYTERANGE (#14748, #24512)
 * [extractor/common] Handle ssl.CertificateError in _request_webpage (#26601)
 * [downloader/http] Improve timeout detection when reading block of data
  (#10935)
 * [downloader/http] Retry download when urlopen times out (#10935, #26603)
 Extractors
 * [redtube] Extend URL regular expression (#26506)
 * [twitch] Refactor
 * [twitch:stream] Switch to GraphQL and fix reruns (#26535)
 + [telequebec] Add support for brightcove videos (#25833)
 * [pornhub] Extract metadata from JSON-LD (#26614)
 * [pornhub] Fix view count extraction (#26621, #26614)
 version 2020.09.14
 Core
 + [postprocessor/embedthumbnail] Add support for non jpg/png thumbnails
  (#25687, #25717)
 Extractors
 * [rtlnl] Extend URL regular expression (#26549, #25821)
 * [youtube] Fix empty description extraction (#26575, #26006)
 * [srgssr] Extend URL regular expression (#26555, #26556, #26578)
 * [googledrive] Use redirect URLs for source format (#18877, #23919, #24689,
  #26565)
 * [svtplay] Fix id extraction (#26576)
 * [redbulltv] Improve support for rebull.com TV localized URLs (#22063)
 + [redbulltv] Add support for new redbull.com TV URLs (#22037, #22063)
 * [soundcloud:pagedplaylist] Reduce pagination limit (#26557)
 version 2020.09.06
 Core
 + [utils] Recognize wav mimetype (#26463)
 Extractors
 * [nrktv:episode] Improve video id extraction (#25594, #26369, #26409)
 * [youtube] Fix age gate content detection (#26100, #26152, #26311, #26384)
 * [youtube:user] Extend URL regular expression (#26443)
 * [xhamster] Improve initials regular expression (#26526, #26353)
 * [svtplay] Fix video id extraction (#26425, #26428, #26438)
 * [twitch] Rework extractors (#12297, #20414, #20604, #21811, #21812, #22979,
  #24263, #25010, #25553, #25606)
    * Switch to GraphQL
    + Add support for collections
    + Add support for clips and collections playlists
 * [biqle] Improve video ext extraction
 * [xhamster] Fix extraction (#26157, #26254)
 * [xhamster] Extend URL regular expression (#25789, #25804, #25927))
 version 2020.07.28
 Extractors
 * [youtube] Fix sigfunc name extraction (#26134, #26135, #26136, #26137)
 * [youtube] Improve description extraction (#25937, #25980)
 * [wistia] Restrict embed regular expression (#25969)
 * [youtube] Prevent excess HTTP 301 (#25786)
 + [youtube:playlists] Extend URL regular expression (#25810)
 + [bellmedia] Add support for cp24.com clip URLs (#25764)
 * [brightcove] Improve embed detection (#25674)
 version 2020.06.16.1
 Extractors
 * [youtube] Force old layout (#25682, #25683, #25680, #25686)
 * [youtube] Fix categories and improve tags extraction
 version 2020.06.16
 Extractors
 * [youtube] Fix uploader id and uploader URL extraction
 * [youtube] Improve view count extraction
 * [youtube] Fix upload date extraction (#25677)
 * [youtube] Fix thumbnails extraction (#25676)
 * [youtube] Fix playlist and feed extraction (#25675)
 + [facebook] Add support for single-video ID links
 + [youtube] Extract chapters from JSON (#24819)
 + [kaltura] Add support for multiple embeds on a webpage (#25523)
 version 2020.06.06
 Extractors
 * [tele5] Bypass geo restriction
 + [jwplatform] Add support for bypass geo restriction
 * [tele5] Prefer jwplatform over nexx (#25533)
 * [twitch:stream] Expect 400 and 410 HTTP errors from API
 * [twitch:stream] Fix extraction (#25528)
 * [twitch] Fix thumbnails extraction (#25531)
 + [twitch] Pass v5 Accept HTTP header (#25531)
 * [brightcove] Fix subtitles extraction (#25540)
 + [malltv] Add support for sk.mall.tv (#25445)
 * [periscope] Fix untitled broadcasts (#25482)
 * [jwplatform] Improve embeds extraction (#25467)
 version 2020.05.29
 Core
 * [postprocessor/ffmpeg] Embed series metadata with --add-metadata
 * [utils] Fix file permissions in write_json_file (#12471, #25122)
 Extractors
 * [ard:beta] Extend URL regular expression (#25405)
 + [youtube] Add support for more invidious instances (#25417)
 * [giantbomb] Extend URL regular expression (#25222)
 * [ard] Improve URL regular expression (#25134, #25198)
 * [redtube] Improve formats extraction and extract m3u8 formats (#25311,
  #25321)
 * [indavideo] Switch to HTTPS for API request (#25191)
 * [redtube] Improve title extraction (#25208)
 * [vimeo] Improve format extraction and sorting (#25285)
 * [soundcloud] Reduce API playlist page limit (#25274)
 + [youtube] Add support for yewtu.be (#25226)
 * [mailru] Fix extraction (#24530, #25239)
 * [bellator] Fix mgid extraction (#25195)
 version 2020.05.08
 Core
 * [downloader/http] Request last data block of exact remaining size
 * [downloader/http] Finish downloading once received data length matches
  expected
 * [extractor/common] Use compat_cookiejar_Cookie for _set_cookie to always
  ensure cookie name and value are bytestrings on python 2 (#23256, #24776)
 + [compat] Introduce compat_cookiejar_Cookie
 * [utils] Improve cookie files support
    + Add support for UTF-8 in cookie files
    * Skip malformed cookie file entries instead of crashing (invalid entry
      length, invalid expires at)
 Extractors
 * [youtube] Improve signature cipher extraction (#25187, #25188)
 * [iprima] Improve extraction (#25138)
 * [uol] Fix extraction (#22007)
 + [orf] Add support for more radio stations (#24938, #24968)
 * [dailymotion] Fix typo
 - [puhutv] Remove no longer available HTTP formats (#25124)
 version 2020.05.03
 Core
 + [extractor/common] Extract multiple JSON-LD entries
 * [options] Clarify doc on --exec command (#19087, #24883)
 * [extractor/common] Skip malformed ISM manifest XMLs while extracting
  ISM formats (#24667)
 Extractors
 * [crunchyroll] Fix and improve extraction (#25096, #25060)
 * [youtube] Improve player id extraction
 * [youtube] Use redirected video id if any (#25063)
 * [yahoo] Fix GYAO Player extraction and relax URL regular expression
  (#24178, #24778)
 * [tvplay] Fix Viafree extraction (#15189, #24473, #24789)
 * [tenplay] Relax URL regular expression (#25001)
 + [prosiebensat1] Extract series metadata
 * [prosiebensat1] Improve extraction and remove 7tv.de support (#24948)
 - [prosiebensat1] Remove 7tv.de support (#24948)
 * [youtube] Fix DRM videos detection (#24736)
 * [thisoldhouse] Fix video id extraction (#24548, #24549)
 + [soundcloud] Extract AAC format (#19173, #24708)
 * [youtube] Skip broken multifeed videos (#24711)
 * [nova:embed] Fix extraction (#24700)
 * [motherless] Fix extraction (#24699)
 * [twitch:clips] Extend URL regular expression (#24290, #24642)
 * [tv4] Fix ISM formats extraction (#24667)
 * [tele5] Fix extraction (#24553)
 + [mofosex] Add support for generic embeds (#24633)
 + [youporn] Add support for generic embeds
 + [spankwire] Add support for generic embeds (#24633)
 * [spankwire] Fix extraction (#18924, #20648)
 version 2020.03.24
 Core
 - [utils] Revert support for cookie files with spaces used instead of tabs
 Extractors
 * [teachable] Update upskillcourses and gns3 domains
 * [generic] Look for teachable embeds before wistia
 + [teachable] Extract chapter metadata (#24421)
 + [bilibili] Add support for player.bilibili.com (#24402)
 + [bilibili] Add support for new URL schema with BV ids (#24439, #24442)
 * [limelight] Remove disabled API requests (#24255)
 * [soundcloud] Fix download URL extraction (#24394)
 + [cbc:watch] Add support for authentication (#19160)
 * [hellporno] Fix extraction (#24399)
 * [xtube] Fix formats extraction (#24348)
 * [ndr] Fix extraction (#24326)
 * [nhk] Update m3u8 URL and use native HLS downloader (#24329)
 - [nhk] Remove obsolete rtmp formats (#24329)
 * [nhk] Relax URL regular expression (#24329)
 - [vimeo] Revert fix showcase password protected video extraction (#24224)
 version 2020.03.08
 Core
 + [utils] Add support for cookie files with spaces used instead of tabs
 Extractors
 + [pornhub] Add support for pornhubpremium.com (#24288)
 - [youtube] Remove outdated code and unnecessary requests
 * [youtube] Improve extraction in 429 HTTP error conditions (#24283)
 * [nhk] Update API version (#24270)
 version 2020.03.06
 Extractors
 * [youtube] Fix age-gated videos support without login (#24248)
 * [vimeo] Fix showcase password protected video extraction (#24224)
 * [pornhub] Improve title extraction (#24184)
 * [peertube] Improve extraction (#23657)
 + [servus] Add support for new URL schema (#23475, #23583, #24142)
 * [vimeo] Fix subtitles URLs (#24209)
 version 2020.03.01
 Core
 * [YoutubeDL] Force redirect URL to unicode on python 2
 - [options] Remove duplicate short option -v for --version (#24162)
 Extractors
 * [xhamster] Fix extraction (#24205)
 * [franceculture] Fix extraction (#24204)
 + [telecinco] Add support for article opening videos
 * [telecinco] Fix extraction (#24195)
 * [xtube] Fix metadata extraction (#21073, #22455)
 * [youjizz] Fix extraction (#24181)
 - Remove no longer needed compat_str around geturl
 * [pornhd] Fix extraction (#24128)
 + [teachable] Add support for multiple videos per lecture (#24101)
 + [wistia] Add support for multiple generic embeds (#8347, 11385)
 * [imdb] Fix extraction (#23443)
 * [tv2dk:bornholm:play] Fix extraction (#24076)
 version 2020.02.16
 Core
 * [YoutubeDL] Fix playlist entry indexing with --playlist-items (#10591,
  #10622)
 * [update] Fix updating via symlinks (#23991)
 + [compat] Introduce compat_realpath (#23991)
 Extractors
 + [npr] Add support for streams (#24042)
 + [24video] Add support for porn.24video.net (#23779, #23784)
 - [jpopsuki] Remove extractor (#23858)
 * [nova] Improve extraction (#23690)
 * [nova:embed] Improve (#23690)
 * [nova:embed] Fix extraction (#23672)
 + [abc:iview] Add support for 720p (#22907, #22921)
 * [nytimes] Improve format sorting (#24010)
 + [toggle] Add support for mewatch.sg (#23895, #23930)
 * [thisoldhouse] Fix extraction (#23951)
 + [popcorntimes] Add support for popcorntimes.tv (#23949)
 * [sportdeutschland] Update to new API
 * [twitch:stream] Lowercase channel id for stream request (#23917)
 * [tv5mondeplus] Fix extraction (#23907, #23911)
 * [tva] Relax URL regular expression (#23903)
 * [vimeo] Fix album extraction (#23864)
 * [viewlift] Improve extraction
    * Fix extraction (#23851)
    + Add support for authentication
    + Add support for more domains
 * [svt] Fix series extraction (#22297)
 * [svt] Fix article extraction (#22897, #22919)
 * [soundcloud] Imporve private playlist/set tracks extraction (#3707)
 version 2020.01.24
 Extractors
 * [youtube] Fix sigfunc name extraction (#23819)
 * [stretchinternet] Fix extraction (#4319)
 * [voicerepublic] Fix extraction
 * [azmedien] Fix extraction (#23783)
 * [businessinsider] Fix jwplatform id extraction (#22929, #22954)
 + [24video] Add support for 24video.vip (#23753)
 * [ivi:compilation] Fix entries extraction (#23770)
 * [ard] Improve extraction (#23761)
    * Simplify extraction
    + Extract age limit and series
    * Bypass geo-restriction
 + [nbc] Add support for nbc multi network URLs (#23049)
 * [americastestkitchen] Fix extraction
 * [zype] Improve extraction
    + Extract subtitles (#21258)
    + Support URLs with alternative keys/tokens (#21258)
    + Extract more metadata
 * [orf:tvthek] Improve geo restricted videos detection (#23741)
 * [soundcloud] Restore previews extraction (#23739)
 version 2020.01.15
 Extractors
 * [yourporn] Fix extraction (#21645, #22255, #23459)
 + [canvas] Add support for new API endpoint (#17680, #18629)
 * [ndr:base:embed] Improve thumbnails extraction (#23731)
 + [vodplatform] Add support for embed.kwikmotion.com domain
 + [twitter] Add support for promo_video_website cards (#23711)
 * [orf:radio] Clean description and improve extraction
 * [orf:fm4] Fix extraction (#23599)
 * [safari] Fix kaltura session extraction (#23679, #23670)
 * [lego] Fix extraction and extract subtitle (#23687)
 * [cloudflarestream] Improve extraction
    + Add support for bytehighway.net domain
    + Add support for signed URLs
    + Extract thumbnail
 * [naver] Improve extraction
    * Improve geo-restriction handling
    + Extract automatic captions
    + Extract uploader metadata
    + Extract VLive HLS formats
    * Improve metadata extraction
 - [pandatv] Remove extractor (#23630)
 * [dctp] Fix format extraction (#23656)
 + [scrippsnetworks] Add support for www.discovery.com videos
 * [discovery] Fix anonymous token extraction (#23650)
 * [nrktv:seriebase] Fix extraction (#23625, #23537)
 * [wistia] Improve format extraction and extract subtitles (#22590)
 * [vice] Improve extraction (#23631)
 * [redtube] Detect private videos (#23518)
 version 2020.01.01
 Extractors
 * [brightcove] Invalidate policy key cache on failing requests
 * [pornhub] Improve locked videos detection (#22449, #22780)
 + [pornhub] Add support for m3u8 formats
 * [pornhub] Fix extraction (#22749, #23082)
 * [brightcove] Update policy key on failing requests
 * [spankbang] Improve removed video detection (#23423)
 * [spankbang] Fix extraction (#23307, #23423, #23444)
 * [soundcloud] Automatically update client id on failing requests
 * [prosiebensat1] Improve geo restriction handling (#23571)
 * [brightcove] Cache brightcove player policy keys
 * [teachable] Fail with error message if no video URL found
 * [teachable] Improve locked lessons detection (#23528)
 + [scrippsnetworks] Add support for Scripps Networks sites (#19857, #22981)
 * [mitele] Fix extraction (#21354, #23456)
 * [soundcloud] Update client id (#23516)
 * [mailru] Relax URL regular expressions (#23509)
 version 2019.12.25
 Core
 * [utils] Improve str_to_int
 + [downloader/hls] Add ability to override AES decryption key URL (#17521)
 Extractors
 * [mediaset] Fix parse formats (#23508)
 + [tv2dk:bornholm:play] Add support for play.tv2bornholm.dk (#23291)
 + [slideslive] Add support for url and vimeo service names (#23414)
 * [slideslive] Fix extraction (#23413)
 * [twitch:clips] Fix extraction (#23375)
 + [soundcloud] Add support for token protected embeds (#18954)
 * [vk] Improve extraction
    * Fix User Videos extraction (#23356)
    * Extract all videos for lists with more than 1000 videos (#23356)
    + Add support for video albums (#14327, #14492)
 - [kontrtube] Remove extractor
 - [videopremium] Remove extractor
 - [musicplayon] Remove extractor (#9225)
 + [ufctv] Add support for ufcfightpass.imgdge.com and
  ufcfightpass.imggaming.com (#23343)
 + [twitch] Extract m3u8 formats frame rate (#23333)
 + [imggaming] Add support for playlists and extract subtitles
 + [ufcarabia] Add support for UFC Arabia (#23312)
 * [ufctv] Fix extraction
 * [yahoo] Fix gyao brightcove player id (#23303)
 * [vzaar] Override AES decryption key URL (#17521)
 + [vzaar] Add support for AES HLS manifests (#17521, #23299)
 * [nrl] Fix extraction
 * [teachingchannel] Fix extraction
 * [nintendo] Fix extraction and partially add support for Nintendo Direct
  videos (#4592)
 + [ooyala] Add better fallback values for domain and streams variables
 + [youtube] Add support youtubekids.com (#23272)
 * [tv2] Detect DRM protection
 + [tv2] Add support for katsomo.fi and mtv.fi (#10543)
 * [tv2] Fix tv2.no article extraction
 * [msn] Improve extraction
    + Add support for YouTube and NBCSports embeds
    + Add support for articles with multiple videos
    * Improve AOL embed support
    * Improve format extraction
 * [abcotvs] Relax URL regular expression and improve metadata extraction
  (#18014)
 * [channel9] Reduce response size
 * [adobetv] Improve extaction
    * Use OnDemandPagedList for list extractors
    * Reduce show extraction requests
    * Extract original video format and subtitles
    + Add support for adobe tv embeds
 version 2019.11.28
 Core
@ -1001,7 +583,7 @@ Extractors
 version 2019.04.17
 Extractors
-* [openload] Randomize User-Agent (#20688)
+* [openload] Randomize User-Agent (closes #20688)
 + [openload] Add support for oladblock domains (#20471)
 * [adn] Fix subtitle extraction (#12724)
 + [aol] Add support for localized websites
@ -1566,7 +1148,7 @@ Extractors
 + [youtube] Extract channel meta fields (#9676, #12939)
 * [porntube] Fix extraction (#17541)
 * [asiancrush] Fix extraction (#15630)
-+ [twitch:clips] Extend URL regular expression (#17559)
+ [twitch:clips] Extend URL regular expression (closes #17559)
 + [vzaar] Add support for HLS
 * [tube8] Fix metadata extraction (#17520)
 * [eporner] Extract JSON-LD (#17519)
--- a/README.md
+++ b/README.md
@ -434,9 +434,9 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
                                     either the path to the binary or its
                                     containing directory.
    --exec CMD                       Execute a command on the file after
-                                     downloading and post-processing, similar to
+                                     downloading, similar to find's -exec
-                                     find's -exec syntax. Example: --exec 'adb
+                                     syntax. Example: --exec 'adb push {}
-                                     push {} /sdcard/Music/ && rm {}'
+                                     /sdcard/Music/ && rm {}'
    --convert-subs FORMAT            Convert the subtitles to other format
                                     (currently supported: srt|ass|vtt|lrc)
@ -545,7 +545,7 @@ The basic usage is not to set any template arguments when downloading a single f
 - `extractor` (string): Name of the extractor
 - `extractor_key` (string): Key name of the extractor
 - `epoch` (numeric): Unix epoch when creating the file
- - `autonumber` (numeric): Number that will be increased with each download, starting at `--autonumber-start`
+ - `autonumber` (numeric): Five-digit number that will be increased with each download, starting at zero
 - `playlist` (string): Name or id of the playlist that contains the video
 - `playlist_index` (numeric): Index of the video in the playlist padded with leading zeros according to the total length of the playlist
 - `playlist_id` (string): Playlist identifier
@ -835,9 +835,7 @@ In February 2015, the new YouTube player contained a character sequence in a str
 ### HTTP Error 429: Too Many Requests or 402: Payment Required
-These two error codes indicate that the service is blocking your IP address because of overuse. Usually this is a soft block meaning that you can gain access again after solving CAPTCHA. Just open a browser and solve a CAPTCHA the service suggests you and after that [pass cookies](#how-do-i-pass-cookies-to-youtube-dl) to youtube-dl. Note that if your machine has multiple external IPs then you should also pass exactly the same IP you've used for solving CAPTCHA with [`--source-address`](#network-options). Also you may need to pass a `User-Agent` HTTP header of your browser with [`--user-agent`](#workarounds).
+These two error codes indicate that the service is blocking your IP address because of overuse. Contact the service and ask them to unblock your IP address, or - if you have acquired a whitelisted IP address already - use the [`--proxy` or `--source-address` options](#network-options) to select another IP address.
 If this is not the case (no CAPTCHA suggested to solve by the service) then you can contact the service and ask them to unblock your IP address, or - if you have acquired a whitelisted IP address already - use the [`--proxy` or `--source-address` options](#network-options) to select another IP address.
 ### SyntaxError: Non-ASCII character
@ -1032,7 +1030,7 @@ After you have ensured this site is distributing its content legally, you can fo
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in.
 7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](http://flake8.pycqa.org/en/latest/index.html#quickstart):
        $ flake8 youtube_dl/extractor/yourextractor.py
--- a/devscripts/create-github-release.py
+++ b/devscripts/create-github-release.py
@ -1,6 +1,7 @@
 #!/usr/bin/env python
 from __future__ import unicode_literals
 import base64
 import io
 import json
 import mimetypes
@ -14,6 +15,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 from youtube_dl.compat import (
    compat_basestring,
    compat_input,
    compat_getpass,
    compat_print,
    compat_urllib_request,
@ -38,20 +40,28 @@ class GitHubReleaser(object):
        try:
            info = netrc.netrc().authenticators(self._NETRC_MACHINE)
            if info is not None:
-                self._token = info[2]
+                self._username = info[0]
                self._password = info[2]
                compat_print('Using GitHub credentials found in .netrc...')
                return
            else:
                compat_print('No GitHub credentials found in .netrc')
        except (IOError, netrc.NetrcParseError):
            compat_print('Unable to parse .netrc')
-        self._token = compat_getpass(
+        self._username = compat_input(
-            'Type your GitHub PAT (personal access token) and press [Return]: ')
+            'Type your GitHub username or email address and press [Return]: ')
        self._password = compat_getpass(
            'Type your GitHub password and press [Return]: ')
    def _call(self, req):
        if isinstance(req, compat_basestring):
            req = sanitized_Request(req)
-        req.add_header('Authorization', 'token %s' % self._token)
+        # Authorizing manually since GitHub does not response with 401 with
        # WWW-Authenticate header set (see
        # https://developer.github.com/v3/#basic-authentication)
        b64 = base64.b64encode(
            ('%s:%s' % (self._username, self._password)).encode('utf-8')).decode('ascii')
        req.add_header('Authorization', 'Basic %s' % b64)
        response = self._opener.open(req).read().decode('utf-8')
        return json.loads(response)
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@ -28,11 +28,10 @@
 - **acast:channel**
 - **ADN**: Anime Digital Network
 - **AdobeConnect**
- - **adobetv**
+ - **AdobeTV**
- - **adobetv:channel**
+ - **AdobeTVChannel**
- - **adobetv:embed**
+ - **AdobeTVShow**
- - **adobetv:show**
+ - **AdobeTVVideo**
 - **adobetv:video**
 - **AdultSwim**
 - **aenetworks**: A+E Networks: A&E, Lifetime, History.com, FYI Network and History Vault
 - **afreecatv**: afreecatv.com
@ -98,7 +97,6 @@
 - **BiliBili**
 - **BilibiliAudio**
 - **BilibiliAudioAlbum**
 - **BiliBiliPlayer**
 - **BioBioChileTV**
 - **BIQLE**
 - **BitChute**
@ -390,6 +388,7 @@
 - **JeuxVideo**
 - **Joj**
 - **Jove**
 - **jpopsuki.tv**
 - **JWPlatform**
 - **Kakao**
 - **Kaltura**
@ -397,7 +396,6 @@
 - **Kankan**
 - **Karaoketv**
 - **KarriereVideos**
 - **Katsomo**
 - **KeezMovies**
 - **Ketnet**
 - **KhanAcademy**
@ -405,6 +403,7 @@
 - **KinjaEmbed**
 - **KinoPoisk**
 - **KonserthusetPlay**
 - **kontrtube**: KontrTube.ru - Труба зовёт
 - **KrasView**: Красвью
 - **Ku6**
 - **KUSI**
@ -497,7 +496,6 @@
 - **MNetTV**
 - **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
 - **Mofosex**
 - **MofosexEmbed**
 - **Mojvideo**
 - **Morningstar**: morningstar.com
 - **Motherless**
@ -515,6 +513,7 @@
 - **mtvjapan**
 - **mtvservices:embedded**
 - **MuenchenTV**: münchen.tv
 - **MusicPlayOn**
 - **mva**: Microsoft Virtual Academy videos
 - **mva:course**: Microsoft Virtual Academy courses
 - **Mwave**
@ -620,25 +619,16 @@
 - **Ooyala**
 - **OoyalaExternal**
 - **OraTV**
 - **orf:burgenland**: Radio Burgenland
 - **orf:fm4**: radio FM4
 - **orf:fm4:story**: fm4.orf.at stories
 - **orf:iptv**: iptv.ORF.at
 - **orf:kaernten**: Radio Kärnten
 - **orf:noe**: Radio Niederösterreich
 - **orf:oberoesterreich**: Radio Oberösterreich
 - **orf:oe1**: Radio Österreich 1
 - **orf:oe3**: Radio Österreich 3
 - **orf:salzburg**: Radio Salzburg
 - **orf:steiermark**: Radio Steiermark
 - **orf:tirol**: Radio Tirol
 - **orf:tvthek**: ORF TVthek
 - **orf:vorarlberg**: Radio Vorarlberg
 - **orf:wien**: Radio Wien
 - **OsnatelTV**
 - **OutsideTV**
 - **PacktPub**
 - **PacktPubCourse**
 - **PandaTV**: 熊猫TV
 - **pandora.tv**: 판도라TV
 - **ParamountNetwork**
 - **parliamentlive.tv**: UK parliament videos
@ -674,7 +664,6 @@
 - **Pokemon**
 - **PolskieRadio**
 - **PolskieRadioCategory**
 - **Popcorntimes**
 - **PopcornTV**
 - **PornCom**
 - **PornerBros**
@ -717,8 +706,6 @@
 - **RayWenderlichCourse**
 - **RBMARadio**
 - **RDS**: RDS.ca
 - **RedBull**
 - **RedBullEmbed**
 - **RedBullTV**
 - **RedBullTVRrnContent**
 - **Reddit**
@ -774,7 +761,6 @@
 - **screen.yahoo:search**: Yahoo screen search
 - **Screencast**
 - **ScreencastOMatic**
 - **ScrippsNetworks**
 - **scrippsnetworks:watch**
 - **SCTE**
 - **SCTECourse**
@ -927,7 +913,6 @@
 - **tv2.hu**
 - **TV2Article**
 - **TV2DK**
 - **TV2DKBornholmPlay**
 - **TV4**: tv4.se and tv4play.se
 - **TV5MondePlus**: TV5MONDE+
 - **TVA**
@ -952,13 +937,16 @@
 - **TVPlayHome**
 - **Tweakers**
 - **TwitCasting**
 - **twitch:chapter**
 - **twitch:clips**
 - **twitch:profile**
 - **twitch:stream**
 - **twitch:video**
 - **twitch:videos:all**
 - **twitch:videos:highlights**
 - **twitch:videos:past-broadcasts**
 - **twitch:videos:uploads**
 - **twitch:vod**
 - **TwitchCollection**
 - **TwitchVideos**
 - **TwitchVideosClips**
 - **TwitchVideosCollections**
 - **twitter**
 - **twitter:amplify**
 - **twitter:broadcast**
@ -966,7 +954,6 @@
 - **udemy**
 - **udemy:course**
 - **UDNEmbed**: 聯合影音
 - **UFCArabia**
 - **UFCTV**
 - **UKTVPlay**
 - **umg:de**: Universal Music Deutschland
@ -1006,6 +993,7 @@
 - **videomore**
 - **videomore:season**
 - **videomore:video**
 - **VideoPremium**
 - **VideoPress**
 - **Vidio**
 - **VidLii**
@ -1015,8 +1003,8 @@
 - **Vidzi**
 - **vier**: vier.be and vijf.be
 - **vier:videos**
- - **viewlift**
+ - **ViewLift**
- - **viewlift:embed**
+ - **ViewLiftEmbed**
 - **Viidea**
 - **viki**
 - **viki:channel**
--- a/test/test_YoutubeDL.py
+++ b/test/test_YoutubeDL.py
@ -816,15 +816,11 @@ class TestYoutubeDL(unittest.TestCase):
            'webpage_url': 'http://example.com',
        }
        def get_downloaded_info_dicts(params):
            ydl = YDL(params)
            # make a deep copy because the dictionary and nested entries
            # can be modified
            ydl.process_ie_result(copy.deepcopy(playlist))
            return ydl.downloaded_info_dicts
        def get_ids(params):
-            return [int(v['id']) for v in get_downloaded_info_dicts(params)]
+            ydl = YDL(params)
            # make a copy because the dictionary can be modified
            ydl.process_ie_result(playlist.copy())
            return [int(v['id']) for v in ydl.downloaded_info_dicts]
        result = get_ids({})
        self.assertEqual(result, [1, 2, 3, 4])
@ -856,22 +852,6 @@ class TestYoutubeDL(unittest.TestCase):
        result = get_ids({'playlist_items': '2-4,3-4,3'})
        self.assertEqual(result, [2, 3, 4])
        # Tests for https://github.com/ytdl-org/youtube-dl/issues/10591
        # @{
        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
        self.assertEqual(result[0]['playlist_index'], 2)
        self.assertEqual(result[1]['playlist_index'], 3)
        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
        self.assertEqual(result[0]['playlist_index'], 2)
        self.assertEqual(result[1]['playlist_index'], 3)
        self.assertEqual(result[2]['playlist_index'], 4)
        result = get_downloaded_info_dicts({'playlist_items': '4,2'})
        self.assertEqual(result[0]['playlist_index'], 4)
        self.assertEqual(result[1]['playlist_index'], 2)
        # @}
    def test_urlopen_no_file_protocol(self):
        # see https://github.com/ytdl-org/youtube-dl/issues/8227
        ydl = YDL()
--- a/test/test_YoutubeDLCookieJar.py
+++ b/test/test_YoutubeDLCookieJar.py
@ -39,13 +39,6 @@ class TestYoutubeDLCookieJar(unittest.TestCase):
        assert_cookie_has_value('HTTPONLY_COOKIE')
        assert_cookie_has_value('JS_ACCESSIBLE_COOKIE')
    def test_malformed_cookies(self):
        cookiejar = YoutubeDLCookieJar('./test/testdata/cookies/malformed_cookies.txt')
        cookiejar.load(ignore_discard=True, ignore_expires=True)
        # Cookies should be empty since all malformed cookie file entries
        # will be ignored
        self.assertFalse(cookiejar._cookies)
 if __name__ == '__main__':
    unittest.main()
--- a/test/test_subtitles.py
+++ b/test/test_subtitles.py
@ -26,6 +26,7 @@ from youtube_dl.extractor import (
    ThePlatformIE,
    ThePlatformFeedIE,
    RTVEALaCartaIE,
    FunnyOrDieIE,
    DemocracynowIE,
 )
@ -321,6 +322,18 @@ class TestRtveSubtitles(BaseTestSubtitles):
        self.assertEqual(md5(subtitles['es']), '69e70cae2d40574fb7316f31d6eb7fca')
 class TestFunnyOrDieSubtitles(BaseTestSubtitles):
    url = 'http://www.funnyordie.com/videos/224829ff6d/judd-apatow-will-direct-your-vine'
    IE = FunnyOrDieIE
    def test_allsubtitles(self):
        self.DL.params['writesubtitles'] = True
        self.DL.params['allsubtitles'] = True
        subtitles = self.getSubtitles()
        self.assertEqual(set(subtitles.keys()), set(['en']))
        self.assertEqual(md5(subtitles['en']), 'c5593c193eacd353596c11c2d4f9ecc4')
 class TestDemocracynowSubtitles(BaseTestSubtitles):
    url = 'http://www.democracynow.org/shows/2015/7/3'
    IE = DemocracynowIE
--- a/test/test_utils.py
+++ b/test/test_utils.py
@ -499,12 +499,6 @@ class TestUtil(unittest.TestCase):
    def test_str_to_int(self):
        self.assertEqual(str_to_int('123,456'), 123456)
        self.assertEqual(str_to_int('123.456'), 123456)
        self.assertEqual(str_to_int(523), 523)
        # Python 3 has no long
        if sys.version_info < (3, 0):
            eval('self.assertEqual(str_to_int(123456L), 123456)')
        self.assertEqual(str_to_int('noninteger'), None)
        self.assertEqual(str_to_int([]), None)
    def test_url_basename(self):
        self.assertEqual(url_basename('http://foo.de/'), '')
@ -803,8 +797,6 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(mimetype2ext('text/vtt'), 'vtt')
        self.assertEqual(mimetype2ext('text/vtt;charset=utf-8'), 'vtt')
        self.assertEqual(mimetype2ext('text/html; charset=utf-8'), 'html')
        self.assertEqual(mimetype2ext('audio/x-wav'), 'wav')
        self.assertEqual(mimetype2ext('audio/x-wav;codec=pcm'), 'wav')
    def test_month_by_name(self):
        self.assertEqual(month_by_name(None), None)
@ -994,12 +986,6 @@ class TestUtil(unittest.TestCase):
        on = js_to_json('{42:4.2e1}')
        self.assertEqual(json.loads(on), {'42': 42.0})
        on = js_to_json('{ "0x40": "0x40" }')
        self.assertEqual(json.loads(on), {'0x40': '0x40'})
        on = js_to_json('{ "040": "040" }')
        self.assertEqual(json.loads(on), {'040': '040'})
    def test_js_to_json_malformed(self):
        self.assertEqual(js_to_json('42a1'), '42"a1"')
        self.assertEqual(js_to_json('42a-1'), '42"a"-1')
--- a/test/test_youtube_chapters.py
+++ b/test/test_youtube_chapters.py
@ -267,7 +267,7 @@ class TestYoutubeChapters(unittest.TestCase):
        for description, duration, expected_chapters in self._TEST_CASES:
            ie = YoutubeIE()
            expect_value(
-                self, ie._extract_chapters_from_description(description, duration),
+                self, ie._extract_chapters(description, duration),
                expected_chapters, None)
--- a/test/test_youtube_signature.py
+++ b/test/test_youtube_signature.py
@ -74,28 +74,6 @@ _TESTS = [
 ]
 class TestPlayerInfo(unittest.TestCase):
    def test_youtube_extract_player_info(self):
        PLAYER_URLS = (
            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/en_US/base.js', '64dddad9'),
            # obsolete
            ('https://www.youtube.com/yts/jsbin/player_ias-vfle4-e03/en_US/base.js', 'vfle4-e03'),
            ('https://www.youtube.com/yts/jsbin/player_ias-vfl49f_g4/en_US/base.js', 'vfl49f_g4'),
            ('https://www.youtube.com/yts/jsbin/player_ias-vflCPQUIL/en_US/base.js', 'vflCPQUIL'),
            ('https://www.youtube.com/yts/jsbin/player-vflzQZbt7/en_US/base.js', 'vflzQZbt7'),
            ('https://www.youtube.com/yts/jsbin/player-en_US-vflaxXRn1/base.js', 'vflaxXRn1'),
            ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js', 'vflXGBaUN'),
            ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js', 'vflKjOTVq'),
            ('http://s.ytimg.com/yt/swfbin/watch_as3-vflrEm9Nq.swf', 'vflrEm9Nq'),
            ('https://s.ytimg.com/yts/swfbin/player-vflenCdZL/watch_as3.swf', 'vflenCdZL'),
        )
        for player_url, expected_player_id in PLAYER_URLS:
            expected_player_type = player_url.split('.')[-1]
            player_type, player_id = YoutubeIE._extract_player_info(player_url)
            self.assertEqual(player_type, expected_player_type)
            self.assertEqual(player_id, expected_player_id)
 class TestSignature(unittest.TestCase):
    def setUp(self):
        TEST_DIR = os.path.dirname(os.path.abspath(__file__))
--- a/test/testdata/cookies/malformed_cookies.txt
+++ b/test/testdata/cookies/malformed_cookies.txt
@ -1,9 +0,0 @@
 # Netscape HTTP Cookie File
 # http://curl.haxx.se/rfc/cookie_spec.html
 # This is a generated file!  Do not edit.
 # Cookie file entry with invalid number of fields - 6 instead of 7
 www.foobar.foobar	FALSE	/	FALSE	0	COOKIE
 # Cookie file entry with invalid expires at
 www.foobar.foobar	FALSE	/	FALSE	1.7976931348623157e+308	COOKIE	VALUE
--- a/youtube_dl/YoutubeDL.py
+++ b/youtube_dl/YoutubeDL.py
@ -92,7 +92,6 @@ from .utils import (
    YoutubeDLCookieJar,
    YoutubeDLCookieProcessor,
    YoutubeDLHandler,
    YoutubeDLRedirectHandler,
 )
 from .cache import Cache
 from .extractor import get_info_extractor, gen_extractor_classes, _LAZY_LOADER
@ -991,7 +990,7 @@ class YoutubeDL(object):
                    'playlist_title': ie_result.get('title'),
                    'playlist_uploader': ie_result.get('uploader'),
                    'playlist_uploader_id': ie_result.get('uploader_id'),
-                    'playlist_index': playlistitems[i - 1] if playlistitems else i + playliststart,
+                    'playlist_index': i + playliststart,
                    'extractor': ie_result['extractor'],
                    'webpage_url': ie_result['webpage_url'],
                    'webpage_url_basename': url_basename(ie_result['webpage_url']),
@ -2344,7 +2343,6 @@ class YoutubeDL(object):
        debuglevel = 1 if self.params.get('debug_printtraffic') else 0
        https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel)
        ydlh = YoutubeDLHandler(self.params, debuglevel=debuglevel)
        redirect_handler = YoutubeDLRedirectHandler()
        data_handler = compat_urllib_request_DataHandler()
        # When passing our own FileHandler instance, build_opener won't add the
@ -2358,7 +2356,7 @@ class YoutubeDL(object):
        file_handler.file_open = file_open
        opener = compat_urllib_request.build_opener(
-            proxy_handler, https_handler, cookie_processor, ydlh, redirect_handler, data_handler, file_handler)
+            proxy_handler, https_handler, cookie_processor, ydlh, data_handler, file_handler)
        # Delete the default user-agent header, which would otherwise apply in
        # cases where our custom HTTP handler doesn't come into play
--- a/youtube_dl/compat.py
+++ b/youtube_dl/compat.py
@ -57,17 +57,6 @@ try:
 except ImportError:  # Python 2
    import cookielib as compat_cookiejar
 if sys.version_info[0] == 2:
    class compat_cookiejar_Cookie(compat_cookiejar.Cookie):
        def __init__(self, version, name, value, *args, **kwargs):
            if isinstance(name, compat_str):
                name = name.encode()
            if isinstance(value, compat_str):
                value = value.encode()
            compat_cookiejar.Cookie.__init__(self, version, name, value, *args, **kwargs)
 else:
    compat_cookiejar_Cookie = compat_cookiejar.Cookie
 try:
    import http.cookies as compat_cookies
 except ImportError:  # Python 2
@ -2765,17 +2754,6 @@ else:
        compat_expanduser = os.path.expanduser
 if compat_os_name == 'nt' and sys.version_info < (3, 8):
    # os.path.realpath on Windows does not follow symbolic links
    # prior to Python 3.8 (see https://bugs.python.org/issue9949)
    def compat_realpath(path):
        while os.path.islink(path):
            path = os.path.abspath(os.readlink(path))
        return path
 else:
    compat_realpath = os.path.realpath
 if sys.version_info < (3, 0):
    def compat_print(s):
        from .utils import preferredencoding
@ -2998,7 +2976,6 @@ __all__ = [
    'compat_basestring',
    'compat_chr',
    'compat_cookiejar',
    'compat_cookiejar_Cookie',
    'compat_cookies',
    'compat_ctypes_WINFUNCTYPE',
    'compat_etree_Element',
@ -3021,7 +2998,6 @@ __all__ = [
    'compat_os_name',
    'compat_parse_qs',
    'compat_print',
    'compat_realpath',
    'compat_setenv',
    'compat_shlex_quote',
    'compat_shlex_split',
--- a/youtube_dl/downloader/hls.py
+++ b/youtube_dl/downloader/hls.py
@ -64,7 +64,7 @@ class HlsFD(FragmentFD):
        s = urlh.read().decode('utf-8', 'ignore')
        if not self.can_download(s, info_dict):
-            if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
+            if info_dict.get('extra_param_to_segment_url'):
                self.report_error('pycrypto not found. Please install it.')
                return False
            self.report_warning(
@ -141,7 +141,7 @@ class HlsFD(FragmentFD):
                    count = 0
                    headers = info_dict.get('http_headers', {})
                    if byte_range:
-                        headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
+                        headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'])
                    while count <= fragment_retries:
                        try:
                            success, frag_content = self._download_fragment(
@ -169,7 +169,7 @@ class HlsFD(FragmentFD):
                    if decrypt_info['METHOD'] == 'AES-128':
                        iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                            self._prepare_url(info_dict, decrypt_info['URI'])).read()
                        frag_content = AES.new(
                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
                    self._append_fragment(ctx, frag_content)
--- a/youtube_dl/downloader/http.py
+++ b/youtube_dl/downloader/http.py
@ -106,12 +106,7 @@ class HttpFD(FileDownloader):
                set_range(request, range_start, range_end)
            # Establish connection
            try:
-                try:
+                ctx.data = self.ydl.urlopen(request)
                    ctx.data = self.ydl.urlopen(request)
                except (compat_urllib_error.URLError, ) as err:
                    if isinstance(err.reason, socket.timeout):
                        raise RetryDownload(err)
                    raise err
                # When trying to resume, Content-Range HTTP header of response has to be checked
                # to match the value of requested Range HTTP header. This is due to a webservers
                # that don't support resuming and serve a whole file with no Content-Range
@ -223,27 +218,24 @@ class HttpFD(FileDownloader):
            def retry(e):
                to_stdout = ctx.tmpfilename == '-'
-                if ctx.stream is not None:
+                if not to_stdout:
-                    if not to_stdout:
+                    ctx.stream.close()
-                        ctx.stream.close()
+                ctx.stream = None
                    ctx.stream = None
                ctx.resume_len = byte_counter if to_stdout else os.path.getsize(encodeFilename(ctx.tmpfilename))
                raise RetryDownload(e)
            while True:
                try:
                    # Download and write
-                    data_block = ctx.data.read(block_size if data_len is None else min(block_size, data_len - byte_counter))
+                    data_block = ctx.data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
                # socket.timeout is a subclass of socket.error but may not have
                # errno set
                except socket.timeout as e:
                    retry(e)
                except socket.error as e:
-                    # SSLError on python 2 (inherits socket.error) may have
+                    if e.errno not in (errno.ECONNRESET, errno.ETIMEDOUT):
-                    # no errno set but this error message
+                        raise
-                    if e.errno in (errno.ECONNRESET, errno.ETIMEDOUT) or getattr(e, 'message', None) == 'The read operation timed out':
+                    retry(e)
                        retry(e)
                    raise
                byte_counter += len(data_block)
@ -307,7 +299,7 @@ class HttpFD(FileDownloader):
                    'elapsed': now - ctx.start_time,
                })
-                if data_len is not None and byte_counter == data_len:
+                if is_test and byte_counter == data_len:
                    break
            if not is_test and ctx.chunk_size and ctx.data_len is not None and byte_counter < ctx.data_len:
--- a/youtube_dl/extractor/abc.py
+++ b/youtube_dl/extractor/abc.py
@ -110,17 +110,17 @@ class ABCIViewIE(InfoExtractor):
    # ABC iview programs are normally available for 14 days only.
    _TESTS = [{
-        'url': 'https://iview.abc.net.au/show/gruen/series/11/video/LE1927H001S00',
+        'url': 'https://iview.abc.net.au/show/ben-and-hollys-little-kingdom/series/0/video/ZX9371A050S00',
-        'md5': '67715ce3c78426b11ba167d875ac6abf',
+        'md5': 'cde42d728b3b7c2b32b1b94b4a548afc',
        'info_dict': {
-            'id': 'LE1927H001S00',
+            'id': 'ZX9371A050S00',
            'ext': 'mp4',
-            'title': "Series 11 Ep 1",
+            'title': "Gaston's Birthday",
-            'series': "Gruen",
+            'series': "Ben And Holly's Little Kingdom",
-            'description': 'md5:52cc744ad35045baf6aded2ce7287f67',
+            'description': 'md5:f9de914d02f226968f598ac76f105bcf',
-            'upload_date': '20190925',
+            'upload_date': '20180604',
-            'uploader_id': 'abc1',
+            'uploader_id': 'abc4kids',
-            'timestamp': 1569445289,
+            'timestamp': 1528140219,
        },
        'params': {
            'skip_download': True,
@ -148,7 +148,7 @@ class ABCIViewIE(InfoExtractor):
                'hdnea': token,
            })
-        for sd in ('720', 'sd', 'sd-low'):
+        for sd in ('sd', 'sd-low'):
            sd_url = try_get(
                stream, lambda x: x['streams']['hls'][sd], compat_str)
            if not sd_url:
--- a/youtube_dl/extractor/abcotvs.py
+++ b/youtube_dl/extractor/abcotvs.py
@ -4,30 +4,29 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    dict_get,
    int_or_none,
-    try_get,
+    parse_iso8601,
 )
 class ABCOTVSIE(InfoExtractor):
    IE_NAME = 'abcotvs'
    IE_DESC = 'ABC Owned Television Stations'
-    _VALID_URL = r'https?://(?P<site>abc(?:7(?:news|ny|chicago)?|11|13|30)|6abc)\.com(?:(?:/[^/]+)*/(?P<display_id>[^/]+))?/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:abc(?:7(?:news|ny|chicago)?|11|13|30)|6abc)\.com(?:/[^/]+/(?P<display_id>[^/]+))?/(?P<id>\d+)'
    _TESTS = [
        {
            'url': 'http://abc7news.com/entertainment/east-bay-museum-celebrates-vintage-synthesizers/472581/',
            'info_dict': {
-                'id': '472548',
+                'id': '472581',
                'display_id': 'east-bay-museum-celebrates-vintage-synthesizers',
                'ext': 'mp4',
-                'title': 'East Bay museum celebrates synthesized music',
+                'title': 'East Bay museum celebrates vintage synthesizers',
                'description': 'md5:24ed2bd527096ec2a5c67b9d5a9005f3',
                'thumbnail': r're:^https?://.*\.jpg$',
-                'timestamp': 1421118520,
+                'timestamp': 1421123075,
                'upload_date': '20150113',
                'uploader': 'Jonathan Bloom',
            },
            'params': {
                # m3u8 download
@ -38,63 +37,39 @@ class ABCOTVSIE(InfoExtractor):
            'url': 'http://abc7news.com/472581',
            'only_matching': True,
        },
        {
            'url': 'https://6abc.com/man-75-killed-after-being-struck-by-vehicle-in-chester/5725182/',
            'only_matching': True,
        },
    ]
    _SITE_MAP = {
        '6abc': 'wpvi',
        'abc11': 'wtvd',
        'abc13': 'ktrk',
        'abc30': 'kfsn',
        'abc7': 'kabc',
        'abc7chicago': 'wls',
        'abc7news': 'kgo',
        'abc7ny': 'wabc',
    }
    def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
-        display_id = display_id or video_id
+        video_id = mobj.group('id')
-        station = self._SITE_MAP[site]
+        display_id = mobj.group('display_id') or video_id
-        data = self._download_json(
+        webpage = self._download_webpage(url, display_id)
            'https://api.abcotvs.com/v2/content', display_id, query={
                'id': video_id,
                'key': 'otv.web.%s.story' % station,
                'station': station,
            })['data']
        video = try_get(data, lambda x: x['featuredMedia']['video'], dict) or data
        video_id = compat_str(dict_get(video, ('id', 'publishedKey'), video_id))
        title = video.get('title') or video['linkText']
-        formats = []
+        m3u8 = self._html_search_meta(
-        m3u8_url = video.get('m3u8')
+            'contentURL', webpage, 'm3u8 url', fatal=True).split('?')[0]
-        if m3u8_url:
+
-            formats = self._extract_m3u8_formats(
+        formats = self._extract_m3u8_formats(m3u8, display_id, 'mp4')
                video['m3u8'].split('?')[0], display_id, 'mp4', m3u8_id='hls', fatal=False)
        mp4_url = video.get('mp4')
        if mp4_url:
            formats.append({
                'abr': 128,
                'format_id': 'https',
                'height': 360,
                'url': mp4_url,
                'width': 640,
            })
        self._sort_formats(formats)
-        image = video.get('image') or {}
+        title = self._og_search_title(webpage).strip()
        description = self._og_search_description(webpage).strip()
        thumbnail = self._og_search_thumbnail(webpage)
        timestamp = parse_iso8601(self._search_regex(
            r'<div class="meta">\s*<time class="timeago" datetime="([^"]+)">',
            webpage, 'upload date', fatal=False))
        uploader = self._search_regex(
            r'rel="author">([^<]+)</a>',
            webpage, 'uploader', default=None)
        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
-            'description': dict_get(video, ('description', 'caption'), try_get(video, lambda x: x['meta']['description'])),
+            'description': description,
-            'thumbnail': dict_get(image, ('source', 'dynamicSource')),
+            'thumbnail': thumbnail,
-            'timestamp': int_or_none(video.get('date')),
+            'timestamp': timestamp,
-            'duration': int_or_none(video.get('length')),
+            'uploader': uploader,
            'formats': formats,
        }
--- a/youtube_dl/extractor/adobetv.py
+++ b/youtube_dl/extractor/adobetv.py
@ -1,119 +1,25 @@
 from __future__ import unicode_literals
 import functools
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    float_or_none,
    int_or_none,
    ISO639Utils,
    OnDemandPagedList,
    parse_duration,
    str_or_none,
    str_to_int,
    unified_strdate,
    str_to_int,
    int_or_none,
    float_or_none,
    ISO639Utils,
    determine_ext,
 )
 class AdobeTVBaseIE(InfoExtractor):
-    def _call_api(self, path, video_id, query, note=None):
+    _API_BASE_URL = 'http://tv.adobe.com/api/v4/'
        return self._download_json(
            'http://tv.adobe.com/api/v4/' + path,
            video_id, note, query=query)['data']
    def _parse_subtitles(self, video_data, url_key):
        subtitles = {}
        for translation in video_data.get('translations', []):
            vtt_path = translation.get(url_key)
            if not vtt_path:
                continue
            lang = translation.get('language_w3c') or ISO639Utils.long2short(translation['language_medium'])
            subtitles.setdefault(lang, []).append({
                'ext': 'vtt',
                'url': vtt_path,
            })
        return subtitles
    def _parse_video_data(self, video_data):
        video_id = compat_str(video_data['id'])
        title = video_data['title']
        s3_extracted = False
        formats = []
        for source in video_data.get('videos', []):
            source_url = source.get('url')
            if not source_url:
                continue
            f = {
                'format_id': source.get('quality_level'),
                'fps': int_or_none(source.get('frame_rate')),
                'height': int_or_none(source.get('height')),
                'tbr': int_or_none(source.get('video_data_rate')),
                'width': int_or_none(source.get('width')),
                'url': source_url,
            }
            original_filename = source.get('original_filename')
            if original_filename:
                if not (f.get('height') and f.get('width')):
                    mobj = re.search(r'_(\d+)x(\d+)', original_filename)
                    if mobj:
                        f.update({
                            'height': int(mobj.group(2)),
                            'width': int(mobj.group(1)),
                        })
                if original_filename.startswith('s3://') and not s3_extracted:
                    formats.append({
                        'format_id': 'original',
                        'preference': 1,
                        'url': original_filename.replace('s3://', 'https://s3.amazonaws.com/'),
                    })
                    s3_extracted = True
            formats.append(f)
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': title,
            'description': video_data.get('description'),
            'thumbnail': video_data.get('thumbnail'),
            'upload_date': unified_strdate(video_data.get('start_date')),
            'duration': parse_duration(video_data.get('duration')),
            'view_count': str_to_int(video_data.get('playcount')),
            'formats': formats,
            'subtitles': self._parse_subtitles(video_data, 'vtt'),
        }
 class AdobeTVEmbedIE(AdobeTVBaseIE):
    IE_NAME = 'adobetv:embed'
    _VALID_URL = r'https?://tv\.adobe\.com/embed/\d+/(?P<id>\d+)'
    _TEST = {
        'url': 'https://tv.adobe.com/embed/22/4153',
        'md5': 'c8c0461bf04d54574fc2b4d07ac6783a',
        'info_dict': {
            'id': '4153',
            'ext': 'flv',
            'title': 'Creating Graphics Optimized for BlackBerry',
            'description': 'md5:eac6e8dced38bdaae51cd94447927459',
            'thumbnail': r're:https?://.*\.jpg$',
            'upload_date': '20091109',
            'duration': 377,
            'view_count': int,
        },
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        video_data = self._call_api(
            'episode/' + video_id, video_id, {'disclosure': 'standard'})[0]
        return self._parse_video_data(video_data)
 class AdobeTVIE(AdobeTVBaseIE):
    IE_NAME = 'adobetv'
    _VALID_URL = r'https?://tv\.adobe\.com/(?:(?P<language>fr|de|es|jp)/)?watch/(?P<show_urlname>[^/]+)/(?P<id>[^/]+)'
    _TEST = {
@ -136,33 +42,45 @@ class AdobeTVIE(AdobeTVBaseIE):
        if not language:
            language = 'en'
-        video_data = self._call_api(
+        video_data = self._download_json(
-            'episode/get', urlname, {
+            self._API_BASE_URL + 'episode/get/?language=%s&show_urlname=%s&urlname=%s&disclosure=standard' % (language, show_urlname, urlname),
-                'disclosure': 'standard',
+            urlname)['data'][0]
-                'language': language,
+
-                'show_urlname': show_urlname,
+        formats = [{
-                'urlname': urlname,
+            'url': source['url'],
-            })[0]
+            'format_id': source.get('quality_level') or source['url'].split('-')[-1].split('.')[0] or None,
-        return self._parse_video_data(video_data)
+            'width': int_or_none(source.get('width')),
            'height': int_or_none(source.get('height')),
            'tbr': int_or_none(source.get('video_data_rate')),
        } for source in video_data['videos']]
        self._sort_formats(formats)
        return {
            'id': compat_str(video_data['id']),
            'title': video_data['title'],
            'description': video_data.get('description'),
            'thumbnail': video_data.get('thumbnail'),
            'upload_date': unified_strdate(video_data.get('start_date')),
            'duration': parse_duration(video_data.get('duration')),
            'view_count': str_to_int(video_data.get('playcount')),
            'formats': formats,
        }
 class AdobeTVPlaylistBaseIE(AdobeTVBaseIE):
-    _PAGE_SIZE = 25
+    def _parse_page_data(self, page_data):
        return [self.url_result(self._get_element_url(element_data)) for element_data in page_data]
-    def _fetch_page(self, display_id, query, page):
+    def _extract_playlist_entries(self, url, display_id):
-        page += 1
+        page = self._download_json(url, display_id)
-        query['page'] = page
+        entries = self._parse_page_data(page['data'])
-        for element_data in self._call_api(
+        for page_num in range(2, page['paging']['pages'] + 1):
-                self._RESOURCE, display_id, query, 'Download Page %d' % page):
+            entries.extend(self._parse_page_data(
-            yield self._process_data(element_data)
+                self._download_json(url + '&page=%d' % page_num, display_id)['data']))
-
+        return entries
    def _extract_playlist_entries(self, display_id, query):
        return OnDemandPagedList(functools.partial(
            self._fetch_page, display_id, query), self._PAGE_SIZE)
 class AdobeTVShowIE(AdobeTVPlaylistBaseIE):
    IE_NAME = 'adobetv:show'
    _VALID_URL = r'https?://tv\.adobe\.com/(?:(?P<language>fr|de|es|jp)/)?show/(?P<id>[^/]+)'
    _TEST = {
@ -174,31 +92,26 @@ class AdobeTVShowIE(AdobeTVPlaylistBaseIE):
        },
        'playlist_mincount': 136,
    }
-    _RESOURCE = 'episode'
+
-    _process_data = AdobeTVBaseIE._parse_video_data
+    def _get_element_url(self, element_data):
        return element_data['urls'][0]
    def _real_extract(self, url):
        language, show_urlname = re.match(self._VALID_URL, url).groups()
        if not language:
            language = 'en'
-        query = {
+        query = 'language=%s&show_urlname=%s' % (language, show_urlname)
            'disclosure': 'standard',
            'language': language,
            'show_urlname': show_urlname,
        }
-        show_data = self._call_api(
+        show_data = self._download_json(self._API_BASE_URL + 'show/get/?%s' % query, show_urlname)['data'][0]
            'show/get', show_urlname, query)[0]
        return self.playlist_result(
-            self._extract_playlist_entries(show_urlname, query),
+            self._extract_playlist_entries(self._API_BASE_URL + 'episode/?%s' % query, show_urlname),
-            str_or_none(show_data.get('id')),
+            compat_str(show_data['id']),
-            show_data.get('show_name'),
+            show_data['show_name'],
-            show_data.get('show_description'))
+            show_data['show_description'])
 class AdobeTVChannelIE(AdobeTVPlaylistBaseIE):
    IE_NAME = 'adobetv:channel'
    _VALID_URL = r'https?://tv\.adobe\.com/(?:(?P<language>fr|de|es|jp)/)?channel/(?P<id>[^/]+)(?:/(?P<category_urlname>[^/]+))?'
    _TEST = {
@ -208,30 +121,24 @@ class AdobeTVChannelIE(AdobeTVPlaylistBaseIE):
        },
        'playlist_mincount': 96,
    }
    _RESOURCE = 'show'
-    def _process_data(self, show_data):
+    def _get_element_url(self, element_data):
-        return self.url_result(
+        return element_data['url']
            show_data['url'], 'AdobeTVShow', str_or_none(show_data.get('id')))
    def _real_extract(self, url):
        language, channel_urlname, category_urlname = re.match(self._VALID_URL, url).groups()
        if not language:
            language = 'en'
-        query = {
+        query = 'language=%s&channel_urlname=%s' % (language, channel_urlname)
            'channel_urlname': channel_urlname,
            'language': language,
        }
        if category_urlname:
-            query['category_urlname'] = category_urlname
+            query += '&category_urlname=%s' % category_urlname
        return self.playlist_result(
-            self._extract_playlist_entries(channel_urlname, query),
+            self._extract_playlist_entries(self._API_BASE_URL + 'show/?%s' % query, channel_urlname),
            channel_urlname)
-class AdobeTVVideoIE(AdobeTVBaseIE):
+class AdobeTVVideoIE(InfoExtractor):
    IE_NAME = 'adobetv:video'
    _VALID_URL = r'https?://video\.tv\.adobe\.com/v/(?P<id>\d+)'
    _TEST = {
@ -253,36 +160,38 @@ class AdobeTVVideoIE(AdobeTVBaseIE):
        video_data = self._parse_json(self._search_regex(
            r'var\s+bridge\s*=\s*([^;]+);', webpage, 'bridged data'), video_id)
        title = video_data['title']
-        formats = []
+        formats = [{
-        sources = video_data.get('sources') or []
+            'format_id': '%s-%s' % (determine_ext(source['src']), source.get('height')),
-        for source in sources:
+            'url': source['src'],
-            source_src = source.get('src')
+            'width': int_or_none(source.get('width')),
-            if not source_src:
+            'height': int_or_none(source.get('height')),
-                continue
+            'tbr': int_or_none(source.get('bitrate')),
-            formats.append({
+        } for source in video_data['sources']]
                'filesize': int_or_none(source.get('kilobytes') or None, invscale=1000),
                'format_id': '-'.join(filter(None, [source.get('format'), source.get('label')])),
                'height': int_or_none(source.get('height') or None),
                'tbr': int_or_none(source.get('bitrate') or None),
                'width': int_or_none(source.get('width') or None),
                'url': source_src,
            })
        self._sort_formats(formats)
        # For both metadata and downloaded files the duration varies among
        # formats. I just pick the max one
        duration = max(filter(None, [
            float_or_none(source.get('duration'), scale=1000)
-            for source in sources]))
+            for source in video_data['sources']]))
        subtitles = {}
        for translation in video_data.get('translations', []):
            lang_id = translation.get('language_w3c') or ISO639Utils.long2short(translation['language_medium'])
            if lang_id not in subtitles:
                subtitles[lang_id] = []
            subtitles[lang_id].append({
                'url': translation['vttPath'],
                'ext': 'vtt',
            })
        return {
            'id': video_id,
            'formats': formats,
-            'title': title,
+            'title': video_data['title'],
            'description': video_data.get('description'),
-            'thumbnail': video_data.get('video', {}).get('poster'),
+            'thumbnail': video_data['video'].get('poster'),
            'duration': duration,
-            'subtitles': self._parse_subtitles(video_data, 'vttPath'),
+            'subtitles': subtitles,
        }
--- a/youtube_dl/extractor/afreecatv.py
+++ b/youtube_dl/extractor/afreecatv.py
@ -275,7 +275,7 @@ class AfreecaTVIE(InfoExtractor):
        video_element = video_xml.findall(compat_xpath('./track/video'))[-1]
        if video_element is None or video_element.text is None:
            raise ExtractorError(
-                'Video %s does not exist' % video_id, expected=True)
+                'Video %s video does not exist' % video_id, expected=True)
        video_url = video_element.text.strip()
--- a/youtube_dl/extractor/americastestkitchen.py
+++ b/youtube_dl/extractor/americastestkitchen.py
@ -5,7 +5,6 @@ from .common import InfoExtractor
 from ..utils import (
    clean_html,
    int_or_none,
    js_to_json,
    try_get,
    unified_strdate,
 )
@ -14,21 +13,22 @@ from ..utils import (
 class AmericasTestKitchenIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?americastestkitchen\.com/(?:episode|videos)/(?P<id>\d+)'
    _TESTS = [{
-        'url': 'https://www.americastestkitchen.com/episode/582-weeknight-japanese-suppers',
+        'url': 'https://www.americastestkitchen.com/episode/548-summer-dinner-party',
        'md5': 'b861c3e365ac38ad319cfd509c30577f',
        'info_dict': {
-            'id': '5b400b9ee338f922cb06450c',
+            'id': '1_5g5zua6e',
-            'title': 'Weeknight Japanese Suppers',
+            'title': 'Summer Dinner Party',
            'ext': 'mp4',
-            'description': 'md5:3d0c1a44bb3b27607ce82652db25b4a8',
+            'description': 'md5:858d986e73a4826979b6a5d9f8f6a1ec',
-            'thumbnail': r're:^https?://',
+            'thumbnail': r're:^https?://.*\.jpg',
-            'timestamp': 1523664000,
+            'timestamp': 1497285541,
-            'upload_date': '20180414',
+            'upload_date': '20170612',
-            'release_date': '20180414',
+            'uploader_id': 'roger.metcalf@americastestkitchen.com',
            'release_date': '20170617',
            'series': "America's Test Kitchen",
-            'season_number': 18,
+            'season_number': 17,
-            'episode': 'Weeknight Japanese Suppers',
+            'episode': 'Summer Dinner Party',
-            'episode_number': 15,
+            'episode_number': 24,
        },
        'params': {
            'skip_download': True,
@ -47,7 +47,7 @@ class AmericasTestKitchenIE(InfoExtractor):
            self._search_regex(
                r'window\.__INITIAL_STATE__\s*=\s*({.+?})\s*;\s*</script>',
                webpage, 'initial context'),
-            video_id, js_to_json)
+            video_id)
        ep_data = try_get(
            video_data,
@ -55,7 +55,17 @@ class AmericasTestKitchenIE(InfoExtractor):
             lambda x: x['videoDetail']['content']['data']), dict)
        ep_meta = ep_data.get('full_video', {})
-        zype_id = ep_data.get('zype_id') or ep_meta['zype_id']
+        zype_id = ep_meta.get('zype_id')
        if zype_id:
            embed_url = 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % zype_id
            ie_key = 'Zype'
        else:
            partner_id = self._search_regex(
                r'src=["\'](?:https?:)?//(?:[^/]+\.)kaltura\.com/(?:[^/]+/)*(?:p|partner_id)/(\d+)',
                webpage, 'kaltura partner id')
            external_id = ep_data.get('external_id') or ep_meta['external_id']
            embed_url = 'kaltura:%s:%s' % (partner_id, external_id)
            ie_key = 'Kaltura'
        title = ep_data.get('title') or ep_meta.get('title')
        description = clean_html(ep_meta.get('episode_description') or ep_data.get(
@ -69,8 +79,8 @@ class AmericasTestKitchenIE(InfoExtractor):
        return {
            '_type': 'url_transparent',
-            'url': 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % zype_id,
+            'url': embed_url,
-            'ie_key': 'Zype',
+            'ie_key': ie_key,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
--- a/youtube_dl/extractor/ard.py
+++ b/youtube_dl/extractor/ard.py
@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import json
 import re
 from .common import InfoExtractor
@ -23,101 +22,7 @@ from ..utils import (
 from ..compat import compat_etree_fromstring
-class ARDMediathekBaseIE(InfoExtractor):
+class ARDMediathekIE(InfoExtractor):
    _GEO_COUNTRIES = ['DE']
    def _extract_media_info(self, media_info_url, webpage, video_id):
        media_info = self._download_json(
            media_info_url, video_id, 'Downloading media JSON')
        return self._parse_media_info(media_info, video_id, '"fsk"' in webpage)
    def _parse_media_info(self, media_info, video_id, fsk):
        formats = self._extract_formats(media_info, video_id)
        if not formats:
            if fsk:
                raise ExtractorError(
                    'This video is only available after 20:00', expected=True)
            elif media_info.get('_geoblocked'):
                self.raise_geo_restricted(
                    'This video is not available due to geoblocking',
                    countries=self._GEO_COUNTRIES)
        self._sort_formats(formats)
        subtitles = {}
        subtitle_url = media_info.get('_subtitleUrl')
        if subtitle_url:
            subtitles['de'] = [{
                'ext': 'ttml',
                'url': subtitle_url,
            }]
        return {
            'id': video_id,
            'duration': int_or_none(media_info.get('_duration')),
            'thumbnail': media_info.get('_previewImage'),
            'is_live': media_info.get('_isLive') is True,
            'formats': formats,
            'subtitles': subtitles,
        }
    def _extract_formats(self, media_info, video_id):
        type_ = media_info.get('_type')
        media_array = media_info.get('_mediaArray', [])
        formats = []
        for num, media in enumerate(media_array):
            for stream in media.get('_mediaStreamArray', []):
                stream_urls = stream.get('_stream')
                if not stream_urls:
                    continue
                if not isinstance(stream_urls, list):
                    stream_urls = [stream_urls]
                quality = stream.get('_quality')
                server = stream.get('_server')
                for stream_url in stream_urls:
                    if not url_or_none(stream_url):
                        continue
                    ext = determine_ext(stream_url)
                    if quality != 'auto' and ext in ('f4m', 'm3u8'):
                        continue
                    if ext == 'f4m':
                        formats.extend(self._extract_f4m_formats(
                            update_url_query(stream_url, {
                                'hdcore': '3.1.1',
                                'plugin': 'aasp-3.1.1.69.124'
                            }), video_id, f4m_id='hds', fatal=False))
                    elif ext == 'm3u8':
                        formats.extend(self._extract_m3u8_formats(
                            stream_url, video_id, 'mp4', 'm3u8_native',
                            m3u8_id='hls', fatal=False))
                    else:
                        if server and server.startswith('rtmp'):
                            f = {
                                'url': server,
                                'play_path': stream_url,
                                'format_id': 'a%s-rtmp-%s' % (num, quality),
                            }
                        else:
                            f = {
                                'url': stream_url,
                                'format_id': 'a%s-%s-%s' % (num, ext, quality)
                            }
                        m = re.search(
                            r'_(?P<width>\d+)x(?P<height>\d+)\.mp4$',
                            stream_url)
                        if m:
                            f.update({
                                'width': int(m.group('width')),
                                'height': int(m.group('height')),
                            })
                        if type_ == 'audio':
                            f['vcodec'] = 'none'
                        formats.append(f)
        return formats
 class ARDMediathekIE(ARDMediathekBaseIE):
    IE_NAME = 'ARD:mediathek'
    _VALID_URL = r'^https?://(?:(?:(?:www|classic)\.)?ardmediathek\.de|mediathek\.(?:daserste|rbb-online)\.de|one\.ard\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
@ -158,6 +63,94 @@ class ARDMediathekIE(ARDMediathekBaseIE):
    def suitable(cls, url):
        return False if ARDBetaMediathekIE.suitable(url) else super(ARDMediathekIE, cls).suitable(url)
    def _extract_media_info(self, media_info_url, webpage, video_id):
        media_info = self._download_json(
            media_info_url, video_id, 'Downloading media JSON')
        formats = self._extract_formats(media_info, video_id)
        if not formats:
            if '"fsk"' in webpage:
                raise ExtractorError(
                    'This video is only available after 20:00', expected=True)
            elif media_info.get('_geoblocked'):
                raise ExtractorError('This video is not available due to geo restriction', expected=True)
        self._sort_formats(formats)
        duration = int_or_none(media_info.get('_duration'))
        thumbnail = media_info.get('_previewImage')
        is_live = media_info.get('_isLive') is True
        subtitles = {}
        subtitle_url = media_info.get('_subtitleUrl')
        if subtitle_url:
            subtitles['de'] = [{
                'ext': 'ttml',
                'url': subtitle_url,
            }]
        return {
            'id': video_id,
            'duration': duration,
            'thumbnail': thumbnail,
            'is_live': is_live,
            'formats': formats,
            'subtitles': subtitles,
        }
    def _extract_formats(self, media_info, video_id):
        type_ = media_info.get('_type')
        media_array = media_info.get('_mediaArray', [])
        formats = []
        for num, media in enumerate(media_array):
            for stream in media.get('_mediaStreamArray', []):
                stream_urls = stream.get('_stream')
                if not stream_urls:
                    continue
                if not isinstance(stream_urls, list):
                    stream_urls = [stream_urls]
                quality = stream.get('_quality')
                server = stream.get('_server')
                for stream_url in stream_urls:
                    if not url_or_none(stream_url):
                        continue
                    ext = determine_ext(stream_url)
                    if quality != 'auto' and ext in ('f4m', 'm3u8'):
                        continue
                    if ext == 'f4m':
                        formats.extend(self._extract_f4m_formats(
                            update_url_query(stream_url, {
                                'hdcore': '3.1.1',
                                'plugin': 'aasp-3.1.1.69.124'
                            }),
                            video_id, f4m_id='hds', fatal=False))
                    elif ext == 'm3u8':
                        formats.extend(self._extract_m3u8_formats(
                            stream_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
                    else:
                        if server and server.startswith('rtmp'):
                            f = {
                                'url': server,
                                'play_path': stream_url,
                                'format_id': 'a%s-rtmp-%s' % (num, quality),
                            }
                        else:
                            f = {
                                'url': stream_url,
                                'format_id': 'a%s-%s-%s' % (num, ext, quality)
                            }
                        m = re.search(r'_(?P<width>\d+)x(?P<height>\d+)\.mp4$', stream_url)
                        if m:
                            f.update({
                                'width': int(m.group('width')),
                                'height': int(m.group('height')),
                            })
                        if type_ == 'audio':
                            f['vcodec'] = 'none'
                        formats.append(f)
        return formats
    def _real_extract(self, url):
        # determine video id from url
        m = re.match(self._VALID_URL, url)
@ -249,7 +242,7 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 class ARDIE(InfoExtractor):
-    _VALID_URL = r'(?P<mainurl>https?://(www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?P<id>[0-9]+))\.html'
+    _VALID_URL = r'(?P<mainurl>https?://(www\.)?daserste\.de/[^?#]+/videos/(?P<display_id>[^/?#]+)-(?P<id>[0-9]+))\.html'
    _TESTS = [{
        # available till 14.02.2019
        'url': 'http://www.daserste.de/information/talk/maischberger/videos/das-groko-drama-zerlegen-sich-die-volksparteien-video-102.html',
@ -263,9 +256,6 @@ class ARDIE(InfoExtractor):
            'upload_date': '20180214',
            'thumbnail': r're:^https?://.*\.jpg$',
        },
    }, {
        'url': 'https://www.daserste.de/information/reportage-dokumentation/erlebnis-erde/videosextern/woelfe-und-herdenschutzhunde-ungleiche-brueder-102.html',
        'only_matching': True,
    }, {
        'url': 'http://www.daserste.de/information/reportage-dokumentation/dokus/videos/die-story-im-ersten-mission-unter-falscher-flagge-100.html',
        'only_matching': True,
@ -312,31 +302,21 @@ class ARDIE(InfoExtractor):
        }
-class ARDBetaMediathekIE(ARDMediathekBaseIE):
+class ARDBetaMediathekIE(InfoExtractor):
-    _VALID_URL = r'https://(?:(?:beta|www)\.)?ardmediathek\.de/(?P<client>[^/]+)/(?:player|live|video)/(?P<display_id>(?:[^/]+/)*)(?P<video_id>[a-zA-Z0-9]+)'
+    _VALID_URL = r'https://(?:beta|www)\.ardmediathek\.de/[^/]+/(?:player|live)/(?P<video_id>[a-zA-Z0-9]+)(?:/(?P<display_id>[^/?#]+))?'
    _TESTS = [{
-        'url': 'https://ardmediathek.de/ard/video/die-robuste-roswita/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
+        'url': 'https://beta.ardmediathek.de/ard/player/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE/die-robuste-roswita',
-        'md5': 'dfdc87d2e7e09d073d5a80770a9ce88f',
+        'md5': '2d02d996156ea3c397cfc5036b5d7f8f',
        'info_dict': {
            'display_id': 'die-robuste-roswita',
-            'id': '70153354',
+            'id': 'Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
-            'title': 'Die robuste Roswita',
+            'title': 'Tatort: Die robuste Roswita',
            'description': r're:^Der Mord.*trüber ist als die Ilm.',
            'duration': 5316,
-            'thumbnail': 'https://img.ardmediathek.de/standard/00/70/15/33/90/-1852531467/16x9/960?mandant=ard',
+            'thumbnail': 'https://img.ardmediathek.de/standard/00/55/43/59/34/-1774185891/16x9/960?mandant=ard',
-            'timestamp': 1577047500,
+            'upload_date': '20180826',
            'upload_date': '20191222',
            'ext': 'mp4',
        },
    }, {
        'url': 'https://beta.ardmediathek.de/ard/video/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
        'only_matching': True,
    }, {
        'url': 'https://ardmediathek.de/ard/video/saartalk/saartalk-gesellschaftsgift-haltung-gegen-hass/sr-fernsehen/Y3JpZDovL3NyLW9ubGluZS5kZS9TVF84MTY4MA/',
        'only_matching': True,
    }, {
        'url': 'https://www.ardmediathek.de/ard/video/trailer/private-eyes-s01-e01/one/Y3JpZDovL3dkci5kZS9CZWl0cmFnLTE1MTgwYzczLWNiMTEtNGNkMS1iMjUyLTg5MGYzOWQxZmQ1YQ/',
        'only_matching': True,
    }, {
        'url': 'https://www.ardmediathek.de/ard/player/Y3JpZDovL3N3ci5kZS9hZXgvbzEwNzE5MTU/',
        'only_matching': True,
@ -348,75 +328,73 @@ class ARDBetaMediathekIE(ARDMediathekBaseIE):
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('video_id')
-        display_id = mobj.group('display_id')
+        display_id = mobj.group('display_id') or video_id
        if display_id:
            display_id = display_id.rstrip('/')
        if not display_id:
            display_id = video_id
-        player_page = self._download_json(
+        webpage = self._download_webpage(url, display_id)
-            'https://api.ardmediathek.de/public-gateway',
+        data_json = self._search_regex(r'window\.__APOLLO_STATE__\s*=\s*(\{.*);\n', webpage, 'json')
-            display_id, data=json.dumps({
+        data = self._parse_json(data_json, display_id)
-                'query': '''{
+
-  playerPage(client:"%s", clipId: "%s") {
+        res = {
-    blockedByFsk
+            'id': video_id,
    broadcastedOn
    maturityContentRating
    mediaCollection {
      _duration
      _geoblocked
      _isLive
      _mediaArray {
        _mediaStreamArray {
          _quality
          _server
          _stream
        }
      }
      _previewImage
      _subtitleUrl
      _type
    }
    show {
      title
    }
    synopsis
    title
    tracking {
      atiCustomVars {
        contentId
      }
    }
  }
 }''' % (mobj.group('client'), video_id),
            }).encode(), headers={
                'Content-Type': 'application/json'
            })['data']['playerPage']
        title = player_page['title']
        content_id = str_or_none(try_get(
            player_page, lambda x: x['tracking']['atiCustomVars']['contentId']))
        media_collection = player_page.get('mediaCollection') or {}
        if not media_collection and content_id:
            media_collection = self._download_json(
                'https://www.ardmediathek.de/play/media/' + content_id,
                content_id, fatal=False) or {}
        info = self._parse_media_info(
            media_collection, content_id or video_id,
            player_page.get('blockedByFsk'))
        age_limit = None
        description = player_page.get('synopsis')
        maturity_content_rating = player_page.get('maturityContentRating')
        if maturity_content_rating:
            age_limit = int_or_none(maturity_content_rating.lstrip('FSK'))
        if not age_limit and description:
            age_limit = int_or_none(self._search_regex(
                r'\(FSK\s*(\d+)\)\s*$', description, 'age limit', default=None))
        info.update({
            'age_limit': age_limit,
            'display_id': display_id,
-            'title': title,
+        }
-            'description': description,
+        formats = []
-            'timestamp': unified_timestamp(player_page.get('broadcastedOn')),
+        subtitles = {}
-            'series': try_get(player_page, lambda x: x['show']['title']),
+        geoblocked = False
        for widget in data.values():
            if widget.get('_geoblocked') is True:
                geoblocked = True
            if '_duration' in widget:
                res['duration'] = int_or_none(widget['_duration'])
            if 'clipTitle' in widget:
                res['title'] = widget['clipTitle']
            if '_previewImage' in widget:
                res['thumbnail'] = widget['_previewImage']
            if 'broadcastedOn' in widget:
                res['timestamp'] = unified_timestamp(widget['broadcastedOn'])
            if 'synopsis' in widget:
                res['description'] = widget['synopsis']
            subtitle_url = url_or_none(widget.get('_subtitleUrl'))
            if subtitle_url:
                subtitles.setdefault('de', []).append({
                    'ext': 'ttml',
                    'url': subtitle_url,
                })
            if '_quality' in widget:
                format_url = url_or_none(try_get(
                    widget, lambda x: x['_stream']['json'][0]))
                if not format_url:
                    continue
                ext = determine_ext(format_url)
                if ext == 'f4m':
                    formats.extend(self._extract_f4m_formats(
                        format_url + '?hdcore=3.11.0',
                        video_id, f4m_id='hds', fatal=False))
                elif ext == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
                        format_url, video_id, 'mp4', m3u8_id='hls',
                        fatal=False))
                else:
                    # HTTP formats are not available when geoblocked is True,
                    # other formats are fine though
                    if geoblocked:
                        continue
                    quality = str_or_none(widget.get('_quality'))
                    formats.append({
                        'format_id': ('http-' + quality) if quality else 'http',
                        'url': format_url,
                        'preference': 10,  # Plain HTTP, that's nice
                    })
        if not formats and geoblocked:
            self.raise_geo_restricted(
                msg='This video is not available due to geoblocking',
                countries=['DE'])
        self._sort_formats(formats)
        res.update({
            'subtitles': subtitles,
            'formats': formats,
        })
-        return info
+
        return res
--- a/youtube_dl/extractor/azmedien.py
+++ b/youtube_dl/extractor/azmedien.py
@ -47,19 +47,39 @@ class AZMedienIE(InfoExtractor):
        'url': 'https://www.telebaern.tv/telebaern-news/montag-1-oktober-2018-ganze-sendung-133531189#video=0_7xjo9lf1',
        'only_matching': True
    }]
-    _API_TEMPL = 'https://www.%s/api/pub/gql/%s/NewsArticleTeaser/cb9f2f81ed22e9b47f4ca64ea3cc5a5d13e88d1d'
+
    _PARTNER_ID = '1719221'
    def _real_extract(self, url):
-        host, display_id, article_id, entry_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
        host = mobj.group('host')
        video_id = mobj.group('id')
        entry_id = mobj.group('kaltura_id')
        if not entry_id:
-            entry_id = self._download_json(
+            api_url = 'https://www.%s/api/pub/gql/%s' % (host, host.split('.')[0])
-                self._API_TEMPL % (host, host.split('.')[0]), display_id, query={
+            payload = {
-                    'variables': json.dumps({
+                'query': '''query VideoContext($articleId: ID!) {
-                        'contextId': 'NewsArticle:' + article_id,
+                    article: node(id: $articleId) {
-                    }),
+                      ... on Article {
-                })['data']['context']['mainAsset']['video']['kaltura']['kalturaId']
+                        mainAssetRelation {
                          asset {
                            ... on VideoAsset {
                              kalturaId
                            }
                          }
                        }
                      }
                    }
                  }''',
                'variables': {'articleId': 'Article:%s' % mobj.group('article_id')},
            }
            json_data = self._download_json(
                api_url, video_id, headers={
                    'Content-Type': 'application/json',
                },
                data=json.dumps(payload).encode())
            entry_id = json_data['data']['article']['mainAssetRelation']['asset']['kalturaId']
        return self.url_result(
            'kaltura:%s:%s' % (self._PARTNER_ID, entry_id),
--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@ -528,7 +528,7 @@ class BBCCoUkIE(InfoExtractor):
            def get_programme_id(item):
                def get_from_attributes(item):
-                    for p in ('identifier', 'group'):
+                    for p in('identifier', 'group'):
                        value = item.get(p)
                        if value and re.match(r'^[pb][\da-z]{7}$', value):
                            return value
--- a/youtube_dl/extractor/bellmedia.py
+++ b/youtube_dl/extractor/bellmedia.py
@ -25,8 +25,8 @@ class BellMediaIE(InfoExtractor):
                etalk|
                marilyn
            )\.ca|
-            (?:much|cp24)\.com
+            much\.com
-        )/.*?(?:\b(?:vid(?:eoid)?|clipId)=|-vid|~|%7E|/(?:episode)?)(?P<id>[0-9]{6,})'''
+        )/.*?(?:\bvid(?:eoid)?=|-vid|~|%7E|/(?:episode)?)(?P<id>[0-9]{6,})'''
    _TESTS = [{
        'url': 'https://www.bnnbloomberg.ca/video/david-cockfield-s-top-picks~1403070',
        'md5': '36d3ef559cfe8af8efe15922cd3ce950',
@ -62,9 +62,6 @@ class BellMediaIE(InfoExtractor):
    }, {
        'url': 'http://www.etalk.ca/video?videoid=663455',
        'only_matching': True,
    }, {
        'url': 'https://www.cp24.com/video?clipId=1982548',
        'only_matching': True,
    }]
    _DOMAINS = {
        'thecomedynetwork': 'comedy',
--- a/youtube_dl/extractor/bilibili.py
+++ b/youtube_dl/extractor/bilibili.py
@ -24,18 +24,7 @@ from ..utils import (
 class BiliBiliIE(InfoExtractor):
-    _VALID_URL = r'''(?x)
+    _VALID_URL = r'https?://(?:www\.|bangumi\.|)bilibili\.(?:tv|com)/(?:video/av|anime/(?P<anime_id>\d+)/play#)(?P<id>\d+)'
                    https?://
                        (?:(?:www|bangumi)\.)?
                        bilibili\.(?:tv|com)/
                        (?:
                            (?:
                                video/[aA][vV]|
                                anime/(?P<anime_id>\d+)/play\#
                            )(?P<id_bv>\d+)|
                            video/[bB][vV](?P<id>[^/?#&]+)
                        )
                    '''
    _TESTS = [{
        'url': 'http://www.bilibili.tv/video/av1074402/',
@ -103,10 +92,6 @@ class BiliBiliIE(InfoExtractor):
                'skip_download': True,  # Test metadata only
            },
        }]
    }, {
        # new BV video id format
        'url': 'https://www.bilibili.com/video/BV1JE411F741',
        'only_matching': True,
    }]
    _APP_KEY = 'iVGUTjsxvpLeuDCf'
@ -124,7 +109,7 @@ class BiliBiliIE(InfoExtractor):
        url, smuggled_data = unsmuggle_url(url, {})
        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id') or mobj.group('id_bv')
+        video_id = mobj.group('id')
        anime_id = mobj.group('anime_id')
        webpage = self._download_webpage(url, video_id)
@ -434,17 +419,3 @@ class BilibiliAudioAlbumIE(BilibiliAudioBaseIE):
                    entries, am_id, album_title, album_data.get('intro'))
        return self.playlist_result(entries, am_id)
 class BiliBiliPlayerIE(InfoExtractor):
    _VALID_URL = r'https?://player\.bilibili\.com/player\.html\?.*?\baid=(?P<id>\d+)'
    _TEST = {
        'url': 'http://player.bilibili.com/player.html?aid=92494333&cid=157926707&page=1',
        'only_matching': True,
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        return self.url_result(
            'http://www.bilibili.tv/video/av%s/' % video_id,
            ie=BiliBiliIE.ie_key(), video_id=video_id)
--- a/youtube_dl/extractor/biqle.py
+++ b/youtube_dl/extractor/biqle.py
@ -3,11 +3,10 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from .vk import VKIE
-from ..compat import (
+from ..utils import (
-    compat_b64decode,
+    HEADRequest,
-    compat_urllib_parse_unquote,
+    int_or_none,
 )
 from ..utils import int_or_none
 class BIQLEIE(InfoExtractor):
@ -48,16 +47,9 @@ class BIQLEIE(InfoExtractor):
        if VKIE.suitable(embed_url):
            return self.url_result(embed_url, VKIE.ie_key(), video_id)
-        embed_page = self._download_webpage(
+        self._request_webpage(
-            embed_url, video_id, headers={'Referer': url})
+            HEADRequest(embed_url), video_id, headers={'Referer': url})
-        video_ext = self._get_cookies(embed_url).get('video_ext')
+        video_id, sig, _, access_token = self._get_cookies(embed_url)['video_ext'].value.split('%3A')
        if video_ext:
            video_ext = compat_urllib_parse_unquote(video_ext.value)
        if not video_ext:
            video_ext = compat_b64decode(self._search_regex(
                r'video_ext\s*:\s*[\'"]([A-Za-z0-9+/=]+)',
                embed_page, 'video_ext')).decode()
        video_id, sig, _, access_token = video_ext.split(':')
        item = self._download_json(
            'https://api.vk.com/method/video.get', video_id,
            headers={'User-Agent': 'okhttp/3.4.1'}, query={
--- a/youtube_dl/extractor/brightcove.py
+++ b/youtube_dl/extractor/brightcove.py
@ -5,34 +5,32 @@ import base64
 import re
 import struct
 from .adobepass import AdobePassIE
 from .common import InfoExtractor
 from .adobepass import AdobePassIE
 from ..compat import (
    compat_etree_fromstring,
    compat_HTTPError,
    compat_parse_qs,
    compat_urllib_parse_urlparse,
    compat_urlparse,
    compat_xml_parse_error,
    compat_HTTPError,
 )
 from ..utils import (
    clean_html,
    extract_attributes,
    ExtractorError,
    extract_attributes,
    find_xpath_attr,
    fix_xml_ampersands,
    float_or_none,
    int_or_none,
    js_to_json,
-    mimetype2ext,
+    int_or_none,
    parse_iso8601,
    smuggle_url,
    str_or_none,
    unescapeHTML,
    unsmuggle_url,
    UnsupportedError,
    update_url_query,
-    url_or_none,
+    clean_html,
    mimetype2ext,
    UnsupportedError,
 )
@ -426,7 +424,7 @@ class BrightcoveNewIE(AdobePassIE):
        # [2] looks like:
        for video, script_tag, account_id, player_id, embed in re.findall(
                r'''(?isx)
-                    (<video(?:-js)?\s+[^>]*\bdata-video-id\s*=\s*['"]?[^>]+>)
+                    (<video\s+[^>]*\bdata-video-id\s*=\s*['"]?[^>]+>)
                    (?:.*?
                        (<script[^>]+
                            src=["\'](?:https?:)?//players\.brightcove\.net/
@ -555,16 +553,10 @@ class BrightcoveNewIE(AdobePassIE):
        subtitles = {}
        for text_track in json_data.get('text_tracks', []):
-            if text_track.get('kind') != 'captions':
+            if text_track.get('src'):
-                continue
+                subtitles.setdefault(text_track.get('srclang'), []).append({
-            text_track_url = url_or_none(text_track.get('src'))
+                    'url': text_track['src'],
-            if not text_track_url:
+                })
                continue
            lang = (str_or_none(text_track.get('srclang'))
                    or str_or_none(text_track.get('label')) or 'en').lower()
            subtitles.setdefault(lang, []).append({
                'url': text_track_url,
            })
        is_live = False
        duration = float_or_none(json_data.get('duration'), 1000)
@ -594,63 +586,45 @@ class BrightcoveNewIE(AdobePassIE):
        account_id, player_id, embed, content_type, video_id = re.match(self._VALID_URL, url).groups()
-        policy_key_id = '%s_%s' % (account_id, player_id)
+        webpage = self._download_webpage(
-        policy_key = self._downloader.cache.load('brightcove', policy_key_id)
+            'http://players.brightcove.net/%s/%s_%s/index.min.js'
-        policy_key_extracted = False
+            % (account_id, player_id, embed), video_id)
        store_pk = lambda x: self._downloader.cache.store('brightcove', policy_key_id, x)
-        def extract_policy_key():
+        policy_key = None
            webpage = self._download_webpage(
                'http://players.brightcove.net/%s/%s_%s/index.min.js'
                % (account_id, player_id, embed), video_id)
-            policy_key = None
+        catalog = self._search_regex(
-
+            r'catalog\(({.+?})\);', webpage, 'catalog', default=None)
-            catalog = self._search_regex(
+        if catalog:
-                r'catalog\(({.+?})\);', webpage, 'catalog', default=None)
+            catalog = self._parse_json(
                js_to_json(catalog), video_id, fatal=False)
            if catalog:
-                catalog = self._parse_json(
+                policy_key = catalog.get('policyKey')
                    js_to_json(catalog), video_id, fatal=False)
                if catalog:
                    policy_key = catalog.get('policyKey')
-            if not policy_key:
+        if not policy_key:
-                policy_key = self._search_regex(
+            policy_key = self._search_regex(
-                    r'policyKey\s*:\s*(["\'])(?P<pk>.+?)\1',
+                r'policyKey\s*:\s*(["\'])(?P<pk>.+?)\1',
-                    webpage, 'policy key', group='pk')
+                webpage, 'policy key', group='pk')
            store_pk(policy_key)
            return policy_key
        api_url = 'https://edge.api.brightcove.com/playback/v1/accounts/%s/%ss/%s' % (account_id, content_type, video_id)
-        headers = {}
+        headers = {
            'Accept': 'application/json;pk=%s' % policy_key,
        }
        referrer = smuggled_data.get('referrer')
        if referrer:
            headers.update({
                'Referer': referrer,
                'Origin': re.search(r'https?://[^/]+', referrer).group(0),
            })
-
+        try:
-        for _ in range(2):
+            json_data = self._download_json(api_url, video_id, headers=headers)
-            if not policy_key:
+        except ExtractorError as e:
-                policy_key = extract_policy_key()
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                policy_key_extracted = True
+                json_data = self._parse_json(e.cause.read().decode(), video_id)[0]
-            headers['Accept'] = 'application/json;pk=%s' % policy_key
+                message = json_data.get('message') or json_data['error_code']
-            try:
+                if json_data.get('error_subcode') == 'CLIENT_GEO':
-                json_data = self._download_json(api_url, video_id, headers=headers)
+                    self.raise_geo_restricted(msg=message)
-                break
+                raise ExtractorError(message, expected=True)
-            except ExtractorError as e:
+            raise
                if isinstance(e.cause, compat_HTTPError) and e.cause.code in (401, 403):
                    json_data = self._parse_json(e.cause.read().decode(), video_id)[0]
                    message = json_data.get('message') or json_data['error_code']
                    if json_data.get('error_subcode') == 'CLIENT_GEO':
                        self.raise_geo_restricted(msg=message)
                    elif json_data.get('error_code') == 'INVALID_POLICY_KEY' and not policy_key_extracted:
                        policy_key = None
                        store_pk(None)
                        continue
                    raise ExtractorError(message, expected=True)
                raise
        errors = json_data.get('errors')
        if errors and errors[0].get('error_subcode') == 'TVE_AUTH':
--- a/youtube_dl/extractor/businessinsider.py
+++ b/youtube_dl/extractor/businessinsider.py
@ -9,26 +9,21 @@ class BusinessInsiderIE(InfoExtractor):
    _VALID_URL = r'https?://(?:[^/]+\.)?businessinsider\.(?:com|nl)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'http://uk.businessinsider.com/how-much-radiation-youre-exposed-to-in-everyday-life-2016-6',
-        'md5': 'ffed3e1e12a6f950aa2f7d83851b497a',
+        'md5': 'ca237a53a8eb20b6dc5bd60564d4ab3e',
        'info_dict': {
-            'id': 'cjGDb0X9',
+            'id': 'hZRllCfw',
            'ext': 'mp4',
-            'title': "Bananas give you more radiation exposure than living next to a nuclear power plant",
+            'title': "Here's how much radiation you're exposed to in everyday life",
-            'description': 'md5:0175a3baf200dd8fa658f94cade841b3',
+            'description': 'md5:9a0d6e2c279948aadaa5e84d6d9b99bd',
-            'upload_date': '20160611',
+            'upload_date': '20170709',
-            'timestamp': 1465675620,
+            'timestamp': 1499606400,
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://www.businessinsider.nl/5-scientifically-proven-things-make-you-less-attractive-2017-7/',
-        'md5': '43f438dbc6da0b89f5ac42f68529d84a',
+        'only_matching': True,
        'info_dict': {
            'id': '5zJwd4FK',
            'ext': 'mp4',
            'title': 'Deze dingen zorgen ervoor dat je minder snel een date scoort',
            'description': 'md5:2af8975825d38a4fed24717bbe51db49',
            'upload_date': '20170705',
            'timestamp': 1499270528,
        },
    }, {
        'url': 'http://www.businessinsider.com/excel-index-match-vlookup-video-how-to-2015-2?IR=T',
        'only_matching': True,
@ -40,8 +35,7 @@ class BusinessInsiderIE(InfoExtractor):
        jwplatform_id = self._search_regex(
            (r'data-media-id=["\']([a-zA-Z0-9]{8})',
             r'id=["\']jwplayer_([a-zA-Z0-9]{8})',
-             r'id["\']?\s*:\s*["\']?([a-zA-Z0-9]{8})',
+             r'id["\']?\s*:\s*["\']?([a-zA-Z0-9]{8})'),
             r'(?:jwplatform\.com/players/|jwplayer_)([a-zA-Z0-9]{8})'),
            webpage, 'jwplatform id')
        return self.url_result(
            'jwplatform:%s' % jwplatform_id, ie=JWPlatformIE.ie_key(),
--- a/youtube_dl/extractor/canvas.py
+++ b/youtube_dl/extractor/canvas.py
@ -13,8 +13,6 @@ from ..utils import (
    int_or_none,
    merge_dicts,
    parse_iso8601,
    str_or_none,
    url_or_none,
 )
@ -22,15 +20,15 @@ class CanvasIE(InfoExtractor):
    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza)/assets/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://mediazone.vrt.be/api/v1/ketnet/assets/md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
-        'md5': '68993eda72ef62386a15ea2cf3c93107',
+        'md5': '90139b746a0a9bd7bb631283f6e2a64e',
        'info_dict': {
            'id': 'md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
            'display_id': 'md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
-            'ext': 'mp4',
+            'ext': 'flv',
            'title': 'Nachtwacht: De Greystook',
-            'description': 'Nachtwacht: De Greystook',
+            'description': 'md5:1db3f5dc4c7109c821261e7512975be7',
            'thumbnail': r're:^https?://.*\.jpg$',
-            'duration': 1468.04,
+            'duration': 1468.03,
        },
        'expected_warnings': ['is not a supported codec', 'Unknown MIME type'],
    }, {
@ -41,45 +39,23 @@ class CanvasIE(InfoExtractor):
        'HLS': 'm3u8_native',
        'HLS_AES': 'm3u8',
    }
    _REST_API_BASE = 'https://media-services-public.vrt.be/vualto-video-aggregator-web/rest/external/v1'
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        site_id, video_id = mobj.group('site_id'), mobj.group('id')
        # Old API endpoint, serves more formats but may fail for some videos
        data = self._download_json(
            'https://mediazone.vrt.be/api/v1/%s/assets/%s'
-            % (site_id, video_id), video_id, 'Downloading asset JSON',
+            % (site_id, video_id), video_id)
            'Unable to download asset JSON', fatal=False)
        # New API endpoint
        if not data:
            token = self._download_json(
                '%s/tokens' % self._REST_API_BASE, video_id,
                'Downloading token', data=b'',
                headers={'Content-Type': 'application/json'})['vrtPlayerToken']
            data = self._download_json(
                '%s/videos/%s' % (self._REST_API_BASE, video_id),
                video_id, 'Downloading video JSON', fatal=False, query={
                    'vrtPlayerToken': token,
                    'client': '%s@PROD' % site_id,
                }, expected_status=400)
            message = data.get('message')
            if message and not data.get('title'):
                if data.get('code') == 'AUTHENTICATION_REQUIRED':
                    self.raise_login_required(message)
                raise ExtractorError(message, expected=True)
        title = data['title']
        description = data.get('description')
        formats = []
        for target in data['targetUrls']:
-            format_url, format_type = url_or_none(target.get('url')), str_or_none(target.get('type'))
+            format_url, format_type = target.get('url'), target.get('type')
            if not format_url or not format_type:
                continue
            format_type = format_type.upper()
            if format_type in self._HLS_ENTRY_PROTOCOLS_MAP:
                formats.extend(self._extract_m3u8_formats(
                    format_url, video_id, 'mp4', self._HLS_ENTRY_PROTOCOLS_MAP[format_type],
@ -158,20 +134,20 @@ class CanvasEenIE(InfoExtractor):
        },
        'skip': 'Pagina niet gevonden',
    }, {
-        'url': 'https://www.een.be/thuis/emma-pakt-thilly-aan',
+        'url': 'https://www.een.be/sorry-voor-alles/herbekijk-sorry-voor-alles',
        'info_dict': {
-            'id': 'md-ast-3a24ced2-64d7-44fb-b4ed-ed1aafbf90b8',
+            'id': 'mz-ast-11a587f8-b921-4266-82e2-0bce3e80d07f',
-            'display_id': 'emma-pakt-thilly-aan',
+            'display_id': 'herbekijk-sorry-voor-alles',
            'ext': 'mp4',
-            'title': 'Emma pakt Thilly aan',
+            'title': 'Herbekijk Sorry voor alles',
-            'description': 'md5:c5c9b572388a99b2690030afa3f3bad7',
+            'description': 'md5:8bb2805df8164e5eb95d6a7a29dc0dd3',
            'thumbnail': r're:^https?://.*\.jpg$',
-            'duration': 118.24,
+            'duration': 3788.06,
        },
        'params': {
            'skip_download': True,
        },
-        'expected_warnings': ['is not a supported codec'],
+        'skip': 'Episode no longer available',
    }, {
        'url': 'https://www.canvas.be/check-point/najaar-2016/de-politie-uw-vriend',
        'only_matching': True,
@ -207,44 +183,19 @@ class VrtNUIE(GigyaBaseIE):
    IE_DESC = 'VrtNU.be'
    _VALID_URL = r'https?://(?:www\.)?vrt\.be/(?P<site_id>vrtnu)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TESTS = [{
        # Available via old API endpoint
        'url': 'https://www.vrt.be/vrtnu/a-z/postbus-x/1/postbus-x-s1a1/',
        'info_dict': {
            'id': 'pbs-pub-2e2d8c27-df26-45c9-9dc6-90c78153044d$vid-90c932b1-e21d-4fb8-99b1-db7b49cf74de',
-            'ext': 'mp4',
+            'ext': 'flv',
            'title': 'De zwarte weduwe',
-            'description': 'md5:db1227b0f318c849ba5eab1fef895ee4',
+            'description': 'md5:d90c21dced7db869a85db89a623998d4',
            'duration': 1457.04,
            'thumbnail': r're:^https?://.*\.jpg$',
-            'season': 'Season 1',
+            'season': '1',
            'season_number': 1,
            'episode_number': 1,
        },
-        'skip': 'This video is only available for registered users',
+        'skip': 'This video is only available for registered users'
        'params': {
            'username': '<snip>',
            'password': '<snip>',
        },
        'expected_warnings': ['is not a supported codec'],
    }, {
        # Only available via new API endpoint
        'url': 'https://www.vrt.be/vrtnu/a-z/kamp-waes/1/kamp-waes-s1a5/',
        'info_dict': {
            'id': 'pbs-pub-0763b56c-64fb-4d38-b95b-af60bf433c71$vid-ad36a73c-4735-4f1f-b2c0-a38e6e6aa7e1',
            'ext': 'mp4',
            'title': 'Aflevering 5',
            'description': 'Wie valt door de mand tijdens een missie?',
            'duration': 2967.06,
            'season': 'Season 1',
            'season_number': 1,
            'episode_number': 5,
        },
        'skip': 'This video is only available for registered users',
        'params': {
            'username': '<snip>',
            'password': '<snip>',
        },
        'expected_warnings': ['Unable to download asset JSON', 'is not a supported codec', 'Unknown MIME type'],
    }]
    _NETRC_MACHINE = 'vrtnu'
    _APIKEY = '3_0Z2HujMtiWq_pkAjgnS2Md2E11a1AwZjYiBETtwNE-EoEHDINgtnvcAOpNgmrVGy'
--- a/youtube_dl/extractor/cbc.py
+++ b/youtube_dl/extractor/cbc.py
@ -1,10 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import hashlib
 import json
 import re
 from xml.sax.saxutils import escape
 from .common import InfoExtractor
 from ..compat import (
@ -218,29 +216,6 @@ class CBCWatchBaseIE(InfoExtractor):
        'clearleap': 'http://www.clearleap.com/namespace/clearleap/1.0/',
    }
    _GEO_COUNTRIES = ['CA']
    _LOGIN_URL = 'https://api.loginradius.com/identity/v2/auth/login'
    _TOKEN_URL = 'https://cloud-api.loginradius.com/sso/jwt/api/token'
    _API_KEY = '3f4beddd-2061-49b0-ae80-6f1f2ed65b37'
    _NETRC_MACHINE = 'cbcwatch'
    def _signature(self, email, password):
        data = json.dumps({
            'email': email,
            'password': password,
        }).encode()
        headers = {'content-type': 'application/json'}
        query = {'apikey': self._API_KEY}
        resp = self._download_json(self._LOGIN_URL, None, data=data, headers=headers, query=query)
        access_token = resp['access_token']
        # token
        query = {
            'access_token': access_token,
            'apikey': self._API_KEY,
            'jwtapp': 'jwt',
        }
        resp = self._download_json(self._TOKEN_URL, None, headers=headers, query=query)
        return resp['signature']
    def _call_api(self, path, video_id):
        url = path if path.startswith('http') else self._API_BASE_URL + path
@ -264,8 +239,7 @@ class CBCWatchBaseIE(InfoExtractor):
    def _real_initialize(self):
        if self._valid_device_token():
            return
-        device = self._downloader.cache.load(
+        device = self._downloader.cache.load('cbcwatch', 'device') or {}
            'cbcwatch', self._cache_device_key()) or {}
        self._device_id, self._device_token = device.get('id'), device.get('token')
        if self._valid_device_token():
            return
@ -274,30 +248,16 @@ class CBCWatchBaseIE(InfoExtractor):
    def _valid_device_token(self):
        return self._device_id and self._device_token
    def _cache_device_key(self):
        email, _ = self._get_login_info()
        return '%s_device' % hashlib.sha256(email.encode()).hexdigest() if email else 'device'
    def _register_device(self):
        self._device_id = self._device_token = None
        result = self._download_xml(
            self._API_BASE_URL + 'device/register',
            None, 'Acquiring device token',
            data=b'<device><type>web</type></device>')
        self._device_id = xpath_text(result, 'deviceId', fatal=True)
-        email, password = self._get_login_info()
+        self._device_token = xpath_text(result, 'deviceToken', fatal=True)
        if email and password:
            signature = self._signature(email, password)
            data = '<login><token>{0}</token><device><deviceId>{1}</deviceId><type>web</type></device></login>'.format(
                escape(signature), escape(self._device_id)).encode()
            url = self._API_BASE_URL + 'device/login'
            result = self._download_xml(
                url, None, data=data,
                headers={'content-type': 'application/xml'})
            self._device_token = xpath_text(result, 'token', fatal=True)
        else:
            self._device_token = xpath_text(result, 'deviceToken', fatal=True)
        self._downloader.cache.store(
-            'cbcwatch', self._cache_device_key(), {
+            'cbcwatch', 'device', {
                'id': self._device_id,
                'token': self._device_token,
            })
--- a/youtube_dl/extractor/channel9.py
+++ b/youtube_dl/extractor/channel9.py
@ -32,7 +32,7 @@ class Channel9IE(InfoExtractor):
            'upload_date': '20130828',
            'session_code': 'KOS002',
            'session_room': 'Arena 1A',
-            'session_speakers': 'count:5',
+            'session_speakers': ['Andrew Coates', 'Brady Gaster', 'Mads Kristensen', 'Ed Blankenship', 'Patrick Klug'],
        },
    }, {
        'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
@ -64,15 +64,15 @@ class Channel9IE(InfoExtractor):
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://channel9.msdn.com/Events/DEVintersection/DEVintersection-2016/RSS',
        'info_dict': {
            'id': 'Events/DEVintersection/DEVintersection-2016',
            'title': 'DEVintersection 2016 Orlando Sessions',
        },
        'playlist_mincount': 14,
    }, {
        'url': 'https://channel9.msdn.com/Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b/RSS',
        'info_dict': {
            'id': 'Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b',
            'title': 'Channel 9',
        },
        'playlist_mincount': 100,
    }, {
        'url': 'https://channel9.msdn.com/Events/DEVintersection/DEVintersection-2016/RSS',
        'only_matching': True,
    }, {
        'url': 'https://channel9.msdn.com/Events/Speakers/scott-hanselman/RSS?UrlSafeName=scott-hanselman',
@ -112,11 +112,11 @@ class Channel9IE(InfoExtractor):
                episode_data), content_path)
            content_id = episode_data['contentId']
            is_session = '/Sessions(' in episode_data['api']
-            content_url = 'https://channel9.msdn.com/odata' + episode_data['api'] + '?$select=Captions,CommentCount,MediaLengthInSeconds,PublishedDate,Rating,RatingCount,Title,VideoMP4High,VideoMP4Low,VideoMP4Medium,VideoPlayerPreviewImage,VideoWMV,VideoWMVHQ,Views,'
+            content_url = 'https://channel9.msdn.com/odata' + episode_data['api']
            if is_session:
-                content_url += 'Code,Description,Room,Slides,Speakers,ZipFile&$expand=Speakers'
+                content_url += '?$expand=Speakers'
            else:
-                content_url += 'Authors,Body&$expand=Authors'
+                content_url += '?$expand=Authors'
            content_data = self._download_json(content_url, content_id)
            title = content_data['Title']
@ -210,7 +210,7 @@ class Channel9IE(InfoExtractor):
                'id': content_id,
                'title': title,
                'description': clean_html(content_data.get('Description') or content_data.get('Body')),
-                'thumbnail': content_data.get('VideoPlayerPreviewImage'),
+                'thumbnail': content_data.get('Thumbnail') or content_data.get('VideoPlayerPreviewImage'),
                'duration': int_or_none(content_data.get('MediaLengthInSeconds')),
                'timestamp': parse_iso8601(content_data.get('PublishedDate')),
                'avg_rating': int_or_none(content_data.get('Rating')),
--- a/youtube_dl/extractor/cloudflarestream.py
+++ b/youtube_dl/extractor/cloudflarestream.py
@ -1,24 +1,20 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import base64
 import re
 from .common import InfoExtractor
 class CloudflareStreamIE(InfoExtractor):
    _DOMAIN_RE = r'(?:cloudflarestream\.com|(?:videodelivery|bytehighway)\.net)'
    _EMBED_RE = r'embed\.%s/embed/[^/]+\.js\?.*?\bvideo=' % _DOMAIN_RE
    _ID_RE = r'[\da-f]{32}|[\w-]+\.[\w-]+\.[\w-]+'
    _VALID_URL = r'''(?x)
                    https?://
                        (?:
-                            (?:watch\.)?%s/|
+                            (?:watch\.)?(?:cloudflarestream\.com|videodelivery\.net)/|
-                            %s
+                            embed\.(?:cloudflarestream\.com|videodelivery\.net)/embed/[^/]+\.js\?.*?\bvideo=
                        )
-                        (?P<id>%s)
+                        (?P<id>[\da-f]+)
-                    ''' % (_DOMAIN_RE, _EMBED_RE, _ID_RE)
+                    '''
    _TESTS = [{
        'url': 'https://embed.cloudflarestream.com/embed/we4g.fla9.latest.js?video=31c9291ab41fac05471db4e73aa11717',
        'info_dict': {
@ -45,28 +41,23 @@ class CloudflareStreamIE(InfoExtractor):
        return [
            mobj.group('url')
            for mobj in re.finditer(
-                r'<script[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//%s(?:%s).*?)\1' % (CloudflareStreamIE._EMBED_RE, CloudflareStreamIE._ID_RE),
+                r'<script[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//embed\.(?:cloudflarestream\.com|videodelivery\.net)/embed/[^/]+\.js\?.*?\bvideo=[\da-f]+?.*?)\1',
                webpage)]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        domain = 'bytehighway.net' if 'bytehighway.net/' in url else 'videodelivery.net'
        base_url = 'https://%s/%s/' % (domain, video_id)
        if '.' in video_id:
            video_id = self._parse_json(base64.urlsafe_b64decode(
                video_id.split('.')[1]), video_id)['sub']
        manifest_base_url = base_url + 'manifest/video.'
        formats = self._extract_m3u8_formats(
-            manifest_base_url + 'm3u8', video_id, 'mp4',
+            'https://cloudflarestream.com/%s/manifest/video.m3u8' % video_id,
-            'm3u8_native', m3u8_id='hls', fatal=False)
+            video_id, 'mp4', entry_protocol='m3u8_native', m3u8_id='hls',
            fatal=False)
        formats.extend(self._extract_mpd_formats(
-            manifest_base_url + 'mpd', video_id, mpd_id='dash', fatal=False))
+            'https://cloudflarestream.com/%s/manifest/video.mpd' % video_id,
            video_id, mpd_id='dash', fatal=False))
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': video_id,
            'thumbnail': base_url + 'thumbnails/thumbnail.jpg',
            'formats': formats,
        }
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@ -10,13 +10,12 @@ import os
 import random
 import re
 import socket
 import ssl
 import sys
 import time
 import math
 from ..compat import (
-    compat_cookiejar_Cookie,
+    compat_cookiejar,
    compat_cookies,
    compat_etree_Element,
    compat_etree_fromstring,
@ -68,7 +67,6 @@ from ..utils import (
    sanitized_Request,
    sanitize_filename,
    str_or_none,
    str_to_int,
    strip_or_none,
    unescapeHTML,
    unified_strdate,
@ -625,12 +623,9 @@ class InfoExtractor(object):
                url_or_request = update_url_query(url_or_request, query)
            if data is not None or headers:
                url_or_request = sanitized_Request(url_or_request, data, headers)
        exceptions = [compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error]
        if hasattr(ssl, 'CertificateError'):
            exceptions.append(ssl.CertificateError)
        try:
            return self._downloader.urlopen(url_or_request)
-        except tuple(exceptions) as err:
+        except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
            if isinstance(err, compat_urllib_error.HTTPError):
                if self.__can_accept_status_code(err, expected_status):
                    # Retain reference to error to prevent file object from
@ -1187,33 +1182,16 @@ class InfoExtractor(object):
                                      'twitter card player')
    def _search_json_ld(self, html, video_id, expected_type=None, **kwargs):
-        json_ld_list = list(re.finditer(JSON_LD_RE, html))
+        json_ld = self._search_regex(
            JSON_LD_RE, html, 'JSON-LD', group='json_ld', **kwargs)
        default = kwargs.get('default', NO_DEFAULT)
        if not json_ld:
            return default if default is not NO_DEFAULT else {}
        # JSON-LD may be malformed and thus `fatal` should be respected.
        # At the same time `default` may be passed that assumes `fatal=False`
        # for _search_regex. Let's simulate the same behavior here as well.
        fatal = kwargs.get('fatal', True) if default == NO_DEFAULT else False
-        json_ld = []
+        return self._json_ld(json_ld, video_id, fatal=fatal, expected_type=expected_type)
        for mobj in json_ld_list:
            json_ld_item = self._parse_json(
                mobj.group('json_ld'), video_id, fatal=fatal)
            if not json_ld_item:
                continue
            if isinstance(json_ld_item, dict):
                json_ld.append(json_ld_item)
            elif isinstance(json_ld_item, (list, tuple)):
                json_ld.extend(json_ld_item)
        if json_ld:
            json_ld = self._json_ld(json_ld, video_id, fatal=fatal, expected_type=expected_type)
        if json_ld:
            return json_ld
        if default is not NO_DEFAULT:
            return default
        elif fatal:
            raise RegexNotFoundError('Unable to extract JSON-LD')
        else:
            self._downloader.report_warning('unable to extract JSON-LD %s' % bug_reports_message())
            return {}
    def _json_ld(self, json_ld, video_id, fatal=True, expected_type=None):
        if isinstance(json_ld, compat_str):
@ -1249,10 +1227,7 @@ class InfoExtractor(object):
                interaction_type = is_e.get('interactionType')
                if not isinstance(interaction_type, compat_str):
                    continue
-                # For interaction count some sites provide string instead of
+                interaction_count = int_or_none(is_e.get('userInteractionCount'))
                # an integer (as per spec) with non digit characters (e.g. ",")
                # so extracting count with more relaxed str_to_int
                interaction_count = str_to_int(is_e.get('userInteractionCount'))
                if interaction_count is None:
                    continue
                count_kind = INTERACTION_TYPE_MAP.get(interaction_type.split('/')[-1])
@ -1272,7 +1247,6 @@ class InfoExtractor(object):
                'thumbnail': url_or_none(e.get('thumbnailUrl') or e.get('thumbnailURL')),
                'duration': parse_duration(e.get('duration')),
                'timestamp': unified_timestamp(e.get('uploadDate')),
                'uploader': str_or_none(e.get('author')),
                'filesize': float_or_none(e.get('contentSize')),
                'tbr': int_or_none(e.get('bitrate')),
                'width': int_or_none(e.get('width')),
@ -1282,10 +1256,10 @@ class InfoExtractor(object):
            extract_interaction_statistic(e)
        for e in json_ld:
-            if '@context' in e:
+            if isinstance(e.get('@context'), compat_str) and re.match(r'^https?://schema.org/?$', e.get('@context')):
                item_type = e.get('@type')
                if expected_type is not None and expected_type != item_type:
-                    continue
+                    return info
                if item_type in ('TVEpisode', 'Episode'):
                    episode_name = unescapeHTML(e.get('name'))
                    info.update({
@ -1319,17 +1293,11 @@ class InfoExtractor(object):
                    })
                elif item_type == 'VideoObject':
                    extract_video_object(e)
-                    if expected_type is None:
+                    continue
                        continue
                    else:
                        break
                video = e.get('video')
                if isinstance(video, dict) and video.get('@type') == 'VideoObject':
                    extract_video_object(video)
-                if expected_type is None:
+                break
                    continue
                else:
                    break
        return dict((k, v) for k, v in info.items() if v is not None)
    @staticmethod
@ -2372,8 +2340,6 @@ class InfoExtractor(object):
        if res is False:
            return []
        ism_doc, urlh = res
        if ism_doc is None:
            return []
        return self._parse_ism_formats(ism_doc, urlh.geturl(), ism_id)
@ -2852,7 +2818,7 @@ class InfoExtractor(object):
    def _set_cookie(self, domain, name, value, expire_time=None, port=None,
                    path='/', secure=False, discard=False, rest={}, **kwargs):
-        cookie = compat_cookiejar_Cookie(
+        cookie = compat_cookiejar.Cookie(
            0, name, value, port, port is not None, domain, True,
            domain.startswith('.'), path, True, secure, expire_time,
            discard, None, None, rest)
--- a/youtube_dl/extractor/crunchyroll.py
+++ b/youtube_dl/extractor/crunchyroll.py
@ -13,7 +13,6 @@ from ..compat import (
    compat_b64decode,
    compat_etree_Element,
    compat_etree_fromstring,
    compat_str,
    compat_urllib_parse_urlencode,
    compat_urllib_request,
    compat_urlparse,
@ -26,9 +25,9 @@ from ..utils import (
    intlist_to_bytes,
    int_or_none,
    lowercase_escape,
    merge_dicts,
    remove_end,
    sanitized_Request,
    unified_strdate,
    urlencode_postdata,
    xpath_text,
 )
@ -137,7 +136,6 @@ class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
            # rtmp
            'skip_download': True,
        },
        'skip': 'Video gone',
    }, {
        'url': 'http://www.crunchyroll.com/media-589804/culture-japan-1',
        'info_dict': {
@ -159,12 +157,11 @@ class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
        'info_dict': {
            'id': '702409',
            'ext': 'mp4',
-            'title': compat_str,
+            'title': 'Re:ZERO -Starting Life in Another World- Episode 5 – The Morning of Our Promise Is Still Distant',
-            'description': compat_str,
+            'description': 'md5:97664de1ab24bbf77a9c01918cb7dca9',
            'thumbnail': r're:^https?://.*\.jpg$',
-            'uploader': 'Re:Zero Partners',
+            'uploader': 'TV TOKYO',
-            'timestamp': 1462098900,
+            'upload_date': '20160508',
            'upload_date': '20160501',
        },
        'params': {
            # m3u8 download
@ -175,13 +172,12 @@ class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
        'info_dict': {
            'id': '727589',
            'ext': 'mp4',
-            'title': compat_str,
+            'title': "KONOSUBA -God's blessing on this wonderful world! 2 Episode 1 – Give Me Deliverance From This Judicial Injustice!",
-            'description': compat_str,
+            'description': 'md5:cbcf05e528124b0f3a0a419fc805ea7d',
            'thumbnail': r're:^https?://.*\.jpg$',
            'uploader': 'Kadokawa Pictures Inc.',
-            'timestamp': 1484130900,
+            'upload_date': '20170118',
-            'upload_date': '20170111',
+            'series': "KONOSUBA -God's blessing on this wonderful world!",
            'series': compat_str,
            'season': "KONOSUBA -God's blessing on this wonderful world! 2",
            'season_number': 2,
            'episode': 'Give Me Deliverance From This Judicial Injustice!',
@ -204,11 +200,10 @@ class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
        'info_dict': {
            'id': '535080',
            'ext': 'mp4',
-            'title': compat_str,
+            'title': '11eyes Episode 1 – Red Night ~ Piros éjszaka',
-            'description': compat_str,
+            'description': 'Kakeru and Yuka are thrown into an alternate nightmarish world they call "Red Night".',
            'uploader': 'Marvelous AQL Inc.',
-            'timestamp': 1255512600,
+            'upload_date': '20091021',
            'upload_date': '20091014',
        },
        'params': {
            # Just test metadata extraction
@ -229,17 +224,15 @@ class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
            # just test metadata extraction
            'skip_download': True,
        },
        'skip': 'Video gone',
    }, {
        # A video with a vastly different season name compared to the series name
        'url': 'http://www.crunchyroll.com/nyarko-san-another-crawling-chaos/episode-1-test-590532',
        'info_dict': {
            'id': '590532',
            'ext': 'mp4',
-            'title': compat_str,
+            'title': 'Haiyoru! Nyaruani (ONA) Episode 1 – Test',
-            'description': compat_str,
+            'description': 'Mahiro and Nyaruko talk about official certification.',
            'uploader': 'TV TOKYO',
            'timestamp': 1330956000,
            'upload_date': '20120305',
            'series': 'Nyarko-san: Another Crawling Chaos',
            'season': 'Haiyoru! Nyaruani (ONA)',
@ -449,21 +442,23 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            webpage, 'language', default=None, group='lang')
        video_title = self._html_search_regex(
-            (r'(?s)<h1[^>]*>((?:(?!<h1).)*?<(?:span[^>]+itemprop=["\']title["\']|meta[^>]+itemprop=["\']position["\'])[^>]*>(?:(?!<h1).)+?)</h1>',
+            r'(?s)<h1[^>]*>((?:(?!<h1).)*?<span[^>]+itemprop=["\']title["\'][^>]*>(?:(?!<h1).)+?)</h1>',
-             r'<title>(.+?),\s+-\s+.+? Crunchyroll'),
+            webpage, 'video_title')
            webpage, 'video_title', default=None)
        if not video_title:
            video_title = re.sub(r'^Watch\s+', '', self._og_search_description(webpage))
        video_title = re.sub(r' {2,}', ' ', video_title)
        video_description = (self._parse_json(self._html_search_regex(
            r'<script[^>]*>\s*.+?\[media_id=%s\].+?({.+?"description"\s*:.+?})\);' % video_id,
            webpage, 'description', default='{}'), video_id) or media_metadata).get('description')
        if video_description:
            video_description = lowercase_escape(video_description.replace(r'\r\n', '\n'))
        video_upload_date = self._html_search_regex(
            [r'<div>Availability for free users:(.+?)</div>', r'<div>[^<>]+<span>\s*(.+?\d{4})\s*</span></div>'],
            webpage, 'video_upload_date', fatal=False, flags=re.DOTALL)
        if video_upload_date:
            video_upload_date = unified_strdate(video_upload_date)
        video_uploader = self._html_search_regex(
            # try looking for both an uploader that's a link and one that's not
            [r'<a[^>]+href="/publisher/[^"]+"[^>]*>([^<]+)</a>', r'<div>\s*Publisher:\s*<span>\s*(.+?)\s*</span>\s*</div>'],
-            webpage, 'video_uploader', default=False)
+            webpage, 'video_uploader', fatal=False)
        formats = []
        for stream in media.get('streams', []):
@ -616,15 +611,14 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            r'(?s)<h\d[^>]+id=["\']showmedia_about_episode_num[^>]+>.+?</h\d>\s*<h4>\s*Season (\d+)',
            webpage, 'season number', default=None))
-        info = self._search_json_ld(webpage, video_id, default={})
+        return {
        return merge_dicts({
            'id': video_id,
            'title': video_title,
            'description': video_description,
            'duration': duration,
            'thumbnail': thumbnail,
            'uploader': video_uploader,
            'upload_date': video_upload_date,
            'series': series,
            'season': season,
            'season_number': season_number,
@ -632,7 +626,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            'episode_number': episode_number,
            'subtitles': subtitles,
            'formats': formats,
-        }, info)
+        }
 class CrunchyrollShowPlaylistIE(CrunchyrollBaseIE):
--- a/youtube_dl/extractor/dailymotion.py
+++ b/youtube_dl/extractor/dailymotion.py
@ -32,7 +32,7 @@ class DailymotionBaseInfoExtractor(InfoExtractor):
    @staticmethod
    def _get_cookie_value(cookies, name):
-        cookie = cookies.get(name)
+        cookie = cookies.get('name')
        if cookie:
            return cookie.value
--- a/youtube_dl/extractor/dctp.py
+++ b/youtube_dl/extractor/dctp.py
@ -16,11 +16,10 @@ class DctpTvIE(InfoExtractor):
    _TESTS = [{
        # 4x3
        'url': 'http://www.dctp.tv/filme/videoinstallation-fuer-eine-kaufhausfassade/',
        'md5': '3ffbd1556c3fe210724d7088fad723e3',
        'info_dict': {
            'id': '95eaa4f33dad413aa17b4ee613cccc6c',
            'display_id': 'videoinstallation-fuer-eine-kaufhausfassade',
-            'ext': 'm4v',
+            'ext': 'flv',
            'title': 'Videoinstallation für eine Kaufhausfassade',
            'description': 'Kurzfilm',
            'thumbnail': r're:^https?://.*\.jpg$',
@ -28,6 +27,10 @@ class DctpTvIE(InfoExtractor):
            'timestamp': 1302172322,
            'upload_date': '20110407',
        },
        'params': {
            # rtmp download
            'skip_download': True,
        },
    }, {
        # 16x9
        'url': 'http://www.dctp.tv/filme/sind-youtuber-die-besseren-lehrer/',
@ -56,26 +59,33 @@ class DctpTvIE(InfoExtractor):
        uuid = media['uuid']
        title = media['title']
-        is_wide = media.get('is_wide')
+        ratio = '16x9' if media.get('is_wide') else '4x3'
-        formats = []
+        play_path = 'mp4:%s_dctp_0500_%s.m4v' % (uuid, ratio)
-        def add_formats(suffix):
+        servers = self._download_json(
-            templ = 'https://%%s/%s_dctp_%s.m4v' % (uuid, suffix)
+            'http://www.dctp.tv/streaming_servers/', display_id,
-            formats.extend([{
+            note='Downloading server list JSON', fatal=False)
                'format_id': 'hls-' + suffix,
                'url': templ % 'cdn-segments.dctp.tv' + '/playlist.m3u8',
                'protocol': 'm3u8_native',
            }, {
                'format_id': 's3-' + suffix,
                'url': templ % 'completed-media.s3.amazonaws.com',
            }, {
                'format_id': 'http-' + suffix,
                'url': templ % 'cdn-media.dctp.tv',
            }])
-        add_formats('0500_' + ('16x9' if is_wide else '4x3'))
+        if servers:
-        if is_wide:
+            endpoint = next(
-            add_formats('720p')
+                server['endpoint']
                for server in servers
                if url_or_none(server.get('endpoint'))
                and 'cloudfront' in server['endpoint'])
        else:
            endpoint = 'rtmpe://s2pqqn4u96e4j8.cloudfront.net/cfx/st/'
        app = self._search_regex(
            r'^rtmpe?://[^/]+/(?P<app>.*)$', endpoint, 'app')
        formats = [{
            'url': endpoint,
            'app': app,
            'play_path': play_path,
            'page_url': url,
            'player_url': 'http://svm-prod-dctptv-static.s3.amazonaws.com/dctptv-relaunch2012-110.swf',
            'ext': 'flv',
        }]
        thumbnails = []
        images = media.get('images')
--- a/youtube_dl/extractor/discovery.py
+++ b/youtube_dl/extractor/discovery.py
@ -13,8 +13,8 @@ from ..compat import compat_HTTPError
 class DiscoveryIE(DiscoveryGoBaseIE):
    _VALID_URL = r'''(?x)https?://
        (?P<site>
-            go\.discovery|
+            (?:(?:www|go)\.)?discovery|
-            www\.
+            (?:www\.)?
                (?:
                    investigationdiscovery|
                    discoverylife|
@ -22,7 +22,8 @@ class DiscoveryIE(DiscoveryGoBaseIE):
                    ahctv|
                    destinationamerica|
                    sciencechannel|
-                    tlc
+                    tlc|
                    velocity
                )|
            watch\.
                (?:
@ -82,7 +83,7 @@ class DiscoveryIE(DiscoveryGoBaseIE):
                    'authRel': 'authorization',
                    'client_id': '3020a40c2356a645b4b4',
                    'nonce': ''.join([random.choice(string.ascii_letters) for _ in range(32)]),
-                    'redirectUri': 'https://www.discovery.com/',
+                    'redirectUri': 'https://fusion.ddmcdn.com/app/mercury-sdk/180/redirectHandler.html?https://www.%s.com' % site,
                })['access_token']
        headers = self.geo_verification_headers()
--- a/youtube_dl/extractor/eporner.py
+++ b/youtube_dl/extractor/eporner.py
@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    encode_base_n,
    ExtractorError,
@ -54,7 +55,7 @@ class EpornerIE(InfoExtractor):
        webpage, urlh = self._download_webpage_handle(url, display_id)
-        video_id = self._match_id(urlh.geturl())
+        video_id = self._match_id(compat_str(urlh.geturl()))
        hash = self._search_regex(
            r'hash\s*:\s*["\']([\da-f]{32})', webpage, 'hash')
--- a/youtube_dl/extractor/expressen.py
+++ b/youtube_dl/extractor/expressen.py
@ -15,7 +15,7 @@ from ..utils import (
 class ExpressenIE(InfoExtractor):
    _VALID_URL = r'''(?x)
                    https?://
-                        (?:www\.)?(?:expressen|di)\.se/
+                        (?:www\.)?expressen\.se/
                        (?:(?:tvspelare/video|videoplayer/embed)/)?
                        tv/(?:[^/]+/)*
                        (?P<id>[^/?#&]+)
@ -42,16 +42,13 @@ class ExpressenIE(InfoExtractor):
    }, {
        'url': 'https://www.expressen.se/videoplayer/embed/tv/ditv/ekonomistudion/experterna-har-ar-fragorna-som-avgor-valet/?embed=true&external=true&autoplay=true&startVolume=0&partnerId=di',
        'only_matching': True,
    }, {
        'url': 'https://www.di.se/videoplayer/embed/tv/ditv/borsmorgon/implantica-rusar-70--under-borspremiaren-hor-styrelsemedlemmen/?embed=true&external=true&autoplay=true&startVolume=0&partnerId=di',
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return [
            mobj.group('url') for mobj in re.finditer(
-                r'<iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//(?:www\.)?(?:expressen|di)\.se/(?:tvspelare/video|videoplayer/embed)/tv/.+?)\1',
+                r'<iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//(?:www\.)?expressen\.se/(?:tvspelare/video|videoplayer/embed)/tv/.+?)\1',
                webpage)]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@ -21,7 +21,6 @@ from .acast import (
 from .adn import ADNIE
 from .adobeconnect import AdobeConnectIE
 from .adobetv import (
    AdobeTVEmbedIE,
    AdobeTVIE,
    AdobeTVShowIE,
    AdobeTVChannelIE,
@ -105,7 +104,6 @@ from .bilibili import (
    BiliBiliBangumiIE,
    BilibiliAudioIE,
    BilibiliAudioAlbumIE,
    BiliBiliPlayerIE,
 )
 from .biobiochiletv import BioBioChileTVIE
 from .bitchute import (
@ -498,6 +496,7 @@ from .jeuxvideo import JeuxVideoIE
 from .jove import JoveIE
 from .joj import JojIE
 from .jwplatform import JWPlatformIE
 from .jpopsukitv import JpopsukiIE
 from .kakao import KakaoIE
 from .kaltura import KalturaIE
 from .kanalplay import KanalPlayIE
@ -511,6 +510,7 @@ from .kickstarter import KickStarterIE
 from .kinja import KinjaEmbedIE
 from .kinopoisk import KinoPoiskIE
 from .konserthusetplay import KonserthusetPlayIE
 from .kontrtube import KontrTubeIE
 from .krasview import KrasViewIE
 from .ku6 import Ku6IE
 from .kusi import KUSIIE
@ -636,10 +636,7 @@ from .mixcloud import (
 from .mlb import MLBIE
 from .mnet import MnetIE
 from .moevideo import MoeVideoIE
-from .mofosex import (
+from .mofosex import MofosexIE
    MofosexIE,
    MofosexEmbedIE,
 )
 from .mojvideo import MojvideoIE
 from .morningstar import MorningstarIE
 from .motherless import (
@ -659,6 +656,7 @@ from .mtv import (
    MTVJapanIE,
 )
 from .muenchentv import MuenchenTVIE
 from .musicplayon import MusicPlayOnIE
 from .mwave import MwaveIE, MwaveMeetGreetIE
 from .mychannels import MyChannelsIE
 from .myspace import MySpaceIE, MySpaceAlbumIE
@ -804,16 +802,6 @@ from .orf import (
    ORFFM4IE,
    ORFFM4StoryIE,
    ORFOE1IE,
    ORFOE3IE,
    ORFNOEIE,
    ORFWIEIE,
    ORFBGLIE,
    ORFOOEIE,
    ORFSTMIE,
    ORFKTNIE,
    ORFSBGIE,
    ORFTIRIE,
    ORFVBGIE,
    ORFIPTVIE,
 )
 from .outsidetv import OutsideTVIE
@ -821,6 +809,7 @@ from .packtpub import (
    PacktPubIE,
    PacktPubCourseIE,
 )
 from .pandatv import PandaTVIE
 from .pandoratv import PandoraTVIE
 from .parliamentliveuk import ParliamentLiveUKIE
 from .patreon import PatreonIE
@ -863,7 +852,6 @@ from .polskieradio import (
    PolskieRadioIE,
    PolskieRadioCategoryIE,
 )
 from .popcorntimes import PopcorntimesIE
 from .popcorntv import PopcornTVIE
 from .porn91 import Porn91IE
 from .porncom import PornComIE
@ -918,9 +906,7 @@ from .rbmaradio import RBMARadioIE
 from .rds import RDSIE
 from .redbulltv import (
    RedBullTVIE,
    RedBullEmbedIE,
    RedBullTVRrnContentIE,
    RedBullIE,
 )
 from .reddit import (
    RedditIE,
@ -978,10 +964,7 @@ from .savefrom import SaveFromIE
 from .sbs import SBSIE
 from .screencast import ScreencastIE
 from .screencastomatic import ScreencastOMaticIE
-from .scrippsnetworks import (
+from .scrippsnetworks import ScrippsNetworksWatchIE
    ScrippsNetworksWatchIE,
    ScrippsNetworksIE,
 )
 from .scte import (
    SCTEIE,
    SCTECourseIE,
@ -1184,12 +1167,8 @@ from .turbo import TurboIE
 from .tv2 import (
    TV2IE,
    TV2ArticleIE,
    KatsomoIE,
 )
 from .tv2dk import (
    TV2DKIE,
    TV2DKBornholmPlayIE,
 )
 from .tv2dk import TV2DKIE
 from .tv2hu import TV2HuIE
 from .tv4 import TV4IE
 from .tv5mondeplus import TV5MondePlusIE
@ -1231,11 +1210,14 @@ from .twentymin import TwentyMinutenIE
 from .twentythreevideo import TwentyThreeVideoIE
 from .twitcasting import TwitCastingIE
 from .twitch import (
    TwitchVideoIE,
    TwitchChapterIE,
    TwitchVodIE,
-    TwitchCollectionIE,
+    TwitchProfileIE,
-    TwitchVideosIE,
+    TwitchAllVideosIE,
-    TwitchVideosClipsIE,
+    TwitchUploadsIE,
-    TwitchVideosCollectionsIE,
+    TwitchPastBroadcastsIE,
    TwitchHighlightsIE,
    TwitchStreamIE,
    TwitchClipsIE,
 )
@ -1250,10 +1232,7 @@ from .udemy import (
    UdemyCourseIE
 )
 from .udn import UDNEmbedIE
-from .ufctv import (
+from .ufctv import UFCTVIE
    UFCTVIE,
    UFCArabiaIE,
 )
 from .uktvplay import UKTVPlayIE
 from .digiteka import DigitekaIE
 from .dlive import (
@ -1307,6 +1286,7 @@ from .videomore import (
    VideomoreVideoIE,
    VideomoreSeasonIE,
 )
 from .videopremium import VideoPremiumIE
 from .videopress import VideoPressIE
 from .vidio import VidioIE
 from .vidlii import VidLiiIE
--- a/youtube_dl/extractor/facebook.py
+++ b/youtube_dl/extractor/facebook.py
@ -466,18 +466,15 @@ class FacebookIE(InfoExtractor):
            return info_dict
        if '/posts/' in url:
-            video_id_json = self._search_regex(
+            entries = [
-                r'(["\'])video_ids\1\s*:\s*(?P<ids>\[.+?\])', webpage, 'video ids', group='ids',
+                self.url_result('facebook:%s' % vid, FacebookIE.ie_key())
-                default='')
+                for vid in self._parse_json(
-            if video_id_json:
+                    self._search_regex(
-                entries = [
+                        r'(["\'])video_ids\1\s*:\s*(?P<ids>\[.+?\])',
-                    self.url_result('facebook:%s' % vid, FacebookIE.ie_key())
+                        webpage, 'video ids', group='ids'),
-                    for vid in self._parse_json(video_id_json, video_id)]
+                    video_id)]
                return self.playlist_result(entries, video_id)
-            # Single Video?
+            return self.playlist_result(entries, video_id)
            video_id = self._search_regex(r'video_id:\s*"([0-9]+)"', webpage, 'single video id')
            return self.url_result('facebook:%s' % video_id, FacebookIE.ie_key())
        else:
            _, info_dict = self._extract_from_url(
                self._VIDEO_PAGE_TEMPLATE % video_id,
--- a/youtube_dl/extractor/franceculture.py
+++ b/youtube_dl/extractor/franceculture.py
@ -31,13 +31,7 @@ class FranceCultureIE(InfoExtractor):
        webpage = self._download_webpage(url, display_id)
        video_data = extract_attributes(self._search_regex(
-            r'''(?sx)
+            r'(?s)<div[^>]+class="[^"]*?(?:title-zone-diffusion|heading-zone-(?:wrapper|player-button))[^"]*?"[^>]*>.*?(<button[^>]+data-asset-source="[^"]+"[^>]+>)',
                (?:
                    </h1>|
                    <div[^>]+class="[^"]*?(?:title-zone-diffusion|heading-zone-(?:wrapper|player-button))[^"]*?"[^>]*>
                ).*?
                (<button[^>]+data-asset-source="[^"]+"[^>]+>)
            ''',
            webpage, 'video data'))
        video_url = video_data['data-asset-source']
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@ -60,9 +60,6 @@ from .tnaflix import TNAFlixNetworkEmbedIE
 from .drtuber import DrTuberIE
 from .redtube import RedTubeIE
 from .tube8 import Tube8IE
 from .mofosex import MofosexEmbedIE
 from .spankwire import SpankwireIE
 from .youporn import YouPornIE
 from .vimeo import VimeoIE
 from .dailymotion import DailymotionIE
 from .dailymail import DailyMailIE
@ -1708,15 +1705,6 @@ class GenericIE(InfoExtractor):
            },
            'add_ie': ['Kaltura'],
        },
        {
            # multiple kaltura embeds, nsfw
            'url': 'https://www.quartier-rouge.be/prive/femmes/kamila-avec-video-jaime-sadomie.html',
            'info_dict': {
                'id': 'kamila-avec-video-jaime-sadomie',
                'title': "Kamila avec vídeo “J'aime sadomie”",
            },
            'playlist_count': 8,
        },
        {
            # Non-standard Vimeo embed
            'url': 'https://openclassrooms.com/courses/understanding-the-web',
@ -2110,9 +2098,6 @@ class GenericIE(InfoExtractor):
                'ext': 'mp4',
                'title': 'Smoky Barbecue Favorites',
                'thumbnail': r're:^https?://.*\.jpe?g',
                'description': 'md5:5ff01e76316bd8d46508af26dc86023b',
                'upload_date': '20170909',
                'timestamp': 1504915200,
            },
            'add_ie': [ZypeIE.ie_key()],
            'params': {
@ -2299,7 +2284,7 @@ class GenericIE(InfoExtractor):
        if head_response is not False:
            # Check for redirect
-            new_url = head_response.geturl()
+            new_url = compat_str(head_response.geturl())
            if url != new_url:
                self.report_following_redirect(new_url)
                if force_videoid:
@ -2399,12 +2384,12 @@ class GenericIE(InfoExtractor):
                return self.playlist_result(
                    self._parse_xspf(
                        doc, video_id, xspf_url=url,
-                        xspf_base_url=full_response.geturl()),
+                        xspf_base_url=compat_str(full_response.geturl())),
                    video_id)
            elif re.match(r'(?i)^(?:{[^}]+})?MPD$', doc.tag):
                info_dict['formats'] = self._parse_mpd_formats(
                    doc,
-                    mpd_base_url=full_response.geturl().rpartition('/')[0],
+                    mpd_base_url=compat_str(full_response.geturl()).rpartition('/')[0],
                    mpd_url=url)
                self._sort_formats(info_dict['formats'])
                return info_dict
@ -2548,21 +2533,15 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                dailymail_urls, video_id, video_title, ie=DailyMailIE.ie_key())
        # Look for Teachable embeds, must be before Wistia
        teachable_url = TeachableIE._extract_url(webpage, url)
        if teachable_url:
            return self.url_result(teachable_url)
        # Look for embedded Wistia player
-        wistia_urls = WistiaIE._extract_urls(webpage)
+        wistia_url = WistiaIE._extract_url(webpage)
-        if wistia_urls:
+        if wistia_url:
-            playlist = self.playlist_from_matches(wistia_urls, video_id, video_title, ie=WistiaIE.ie_key())
+            return {
-            for entry in playlist['entries']:
+                '_type': 'url_transparent',
-                entry.update({
+                'url': self._proto_relative_url(wistia_url),
-                    '_type': 'url_transparent',
+                'ie_key': WistiaIE.ie_key(),
-                    'uploader': video_uploader,
+                'uploader': video_uploader,
-                })
+            }
            return playlist
        # Look for SVT player
        svt_url = SVTIE._extract_url(webpage)
@ -2727,21 +2706,6 @@ class GenericIE(InfoExtractor):
        if tube8_urls:
            return self.playlist_from_matches(tube8_urls, video_id, video_title, ie=Tube8IE.ie_key())
        # Look for embedded Mofosex player
        mofosex_urls = MofosexEmbedIE._extract_urls(webpage)
        if mofosex_urls:
            return self.playlist_from_matches(mofosex_urls, video_id, video_title, ie=MofosexEmbedIE.ie_key())
        # Look for embedded Spankwire player
        spankwire_urls = SpankwireIE._extract_urls(webpage)
        if spankwire_urls:
            return self.playlist_from_matches(spankwire_urls, video_id, video_title, ie=SpankwireIE.ie_key())
        # Look for embedded YouPorn player
        youporn_urls = YouPornIE._extract_urls(webpage)
        if youporn_urls:
            return self.playlist_from_matches(youporn_urls, video_id, video_title, ie=YouPornIE.ie_key())
        # Look for embedded Tvigle player
        mobj = re.search(
            r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//cloud\.tvigle\.ru/video/.+?)\1', webpage)
@ -2853,12 +2817,9 @@ class GenericIE(InfoExtractor):
            return self.url_result(mobj.group('url'), 'Zapiks')
        # Look for Kaltura embeds
-        kaltura_urls = KalturaIE._extract_urls(webpage)
+        kaltura_url = KalturaIE._extract_url(webpage)
-        if kaltura_urls:
+        if kaltura_url:
-            return self.playlist_from_matches(
+            return self.url_result(smuggle_url(kaltura_url, {'source_url': url}), KalturaIE.ie_key())
                kaltura_urls, video_id, video_title,
                getter=lambda x: smuggle_url(x, {'source_url': url}),
                ie=KalturaIE.ie_key())
        # Look for EaglePlatform embeds
        eagleplatform_url = EaglePlatformIE._extract_url(webpage)
@ -2999,7 +2960,7 @@ class GenericIE(InfoExtractor):
        # Look for VODPlatform embeds
        mobj = re.search(
-            r'<iframe[^>]+src=(["\'])(?P<url>(?:https?:)?//(?:(?:www\.)?vod-platform\.net|embed\.kwikmotion\.com)/[eE]mbed/.+?)\1',
+            r'<iframe[^>]+src=(["\'])(?P<url>(?:https?:)?//(?:www\.)?vod-platform\.net/[eE]mbed/.+?)\1',
            webpage)
        if mobj is not None:
            return self.url_result(
@ -3176,6 +3137,10 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                peertube_urls, video_id, video_title, ie=PeerTubeIE.ie_key())
        teachable_url = TeachableIE._extract_url(webpage, url)
        if teachable_url:
            return self.url_result(teachable_url)
        indavideo_urls = IndavideoEmbedIE._extract_urls(webpage)
        if indavideo_urls:
            return self.playlist_from_matches(
--- a/youtube_dl/extractor/giantbomb.py
+++ b/youtube_dl/extractor/giantbomb.py
@ -13,10 +13,10 @@ from ..utils import (
 class GiantBombIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?giantbomb\.com/(?:videos|shows)/(?P<display_id>[^/]+)/(?P<id>\d+-\d+)'
+    _VALID_URL = r'https?://(?:www\.)?giantbomb\.com/videos/(?P<display_id>[^/]+)/(?P<id>\d+-\d+)'
-    _TESTS = [{
+    _TEST = {
        'url': 'http://www.giantbomb.com/videos/quick-look-destiny-the-dark-below/2300-9782/',
-        'md5': '132f5a803e7e0ab0e274d84bda1e77ae',
+        'md5': 'c8ea694254a59246a42831155dec57ac',
        'info_dict': {
            'id': '2300-9782',
            'display_id': 'quick-look-destiny-the-dark-below',
@ -26,10 +26,7 @@ class GiantBombIE(InfoExtractor):
            'duration': 2399,
            'thumbnail': r're:^https?://.*\.jpg$',
        }
-    }, {
+    }
        'url': 'https://www.giantbomb.com/shows/ben-stranding/2970-20212',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
--- a/youtube_dl/extractor/googledrive.py
+++ b/youtube_dl/extractor/googledrive.py
@ -220,27 +220,19 @@ class GoogleDriveIE(InfoExtractor):
                'id': video_id,
                'export': 'download',
            })
-
+        urlh = self._request_webpage(
-        def request_source_file(source_url, kind):
+            source_url, video_id, note='Requesting source file',
-            return self._request_webpage(
+            errnote='Unable to request source file', fatal=False)
                source_url, video_id, note='Requesting %s file' % kind,
                errnote='Unable to request %s file' % kind, fatal=False)
        urlh = request_source_file(source_url, 'source')
        if urlh:
-            def add_source_format(urlh):
+            def add_source_format(src_url):
                formats.append({
-                    # Use redirect URLs as download URLs in order to calculate
+                    'url': src_url,
                    # correct cookies in _calc_cookies.
                    # Using original URLs may result in redirect loop due to
                    # google.com's cookies mistakenly used for googleusercontent.com
                    # redirect URLs (see #23919).
                    'url': urlh.geturl(),
                    'ext': determine_ext(title, 'mp4').lower(),
                    'format_id': 'source',
                    'quality': 1,
                })
            if urlh.headers.get('Content-Disposition'):
-                add_source_format(urlh)
+                add_source_format(source_url)
            else:
                confirmation_webpage = self._webpage_read_content(
                    urlh, url, video_id, note='Downloading confirmation page',
@ -250,12 +242,9 @@ class GoogleDriveIE(InfoExtractor):
                        r'confirm=([^&"\']+)', confirmation_webpage,
                        'confirmation code', fatal=False)
                    if confirm:
-                        confirmed_source_url = update_url_query(source_url, {
+                        add_source_format(update_url_query(source_url, {
                            'confirm': confirm,
-                        })
+                        }))
                        urlh = request_source_file(confirmed_source_url, 'confirmed source')
                        if urlh and urlh.headers.get('Content-Disposition'):
                            add_source_format(urlh)
        if not formats:
            reason = self._search_regex(
--- a/youtube_dl/extractor/hellporno.py
+++ b/youtube_dl/extractor/hellporno.py
@ -1,11 +1,12 @@
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
-    int_or_none,
+    js_to_json,
    merge_dicts,
    remove_end,
-    unified_timestamp,
+    determine_ext,
 )
@ -13,21 +14,15 @@ class HellPornoIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?hellporno\.(?:com/videos|net/v)/(?P<id>[^/]+)'
    _TESTS = [{
        'url': 'http://hellporno.com/videos/dixie-is-posing-with-naked-ass-very-erotic/',
-        'md5': 'f0a46ebc0bed0c72ae8fe4629f7de5f3',
+        'md5': '1fee339c610d2049699ef2aa699439f1',
        'info_dict': {
            'id': '149116',
            'display_id': 'dixie-is-posing-with-naked-ass-very-erotic',
            'ext': 'mp4',
            'title': 'Dixie is posing with naked ass very erotic',
            'description': 'md5:9a72922749354edb1c4b6e540ad3d215',
            'categories': list,
            'thumbnail': r're:https?://.*\.jpg$',
            'duration': 240,
            'timestamp': 1398762720,
            'upload_date': '20140429',
            'view_count': int,
            'age_limit': 18,
-        },
+        }
    }, {
        'url': 'http://hellporno.net/v/186271/',
        'only_matching': True,
@ -41,36 +36,40 @@ class HellPornoIE(InfoExtractor):
        title = remove_end(self._html_search_regex(
            r'<title>([^<]+)</title>', webpage, 'title'), ' - Hell Porno')
-        info = self._parse_html5_media_entries(url, webpage, display_id)[0]
+        flashvars = self._parse_json(self._search_regex(
-        self._sort_formats(info['formats'])
+            r'var\s+flashvars\s*=\s*({.+?});', webpage, 'flashvars'),
            display_id, transform_source=js_to_json)
-        video_id = self._search_regex(
+        video_id = flashvars.get('video_id')
-            (r'chs_object\s*=\s*["\'](\d+)',
+        thumbnail = flashvars.get('preview_url')
-             r'params\[["\']video_id["\']\]\s*=\s*(\d+)'), webpage, 'video id',
+        ext = determine_ext(flashvars.get('postfix'), 'mp4')
            default=display_id)
        description = self._search_regex(
            r'class=["\']desc_video_view_v2[^>]+>([^<]+)', webpage,
            'description', fatal=False)
        categories = [
            c.strip()
            for c in self._html_search_meta(
                'keywords', webpage, 'categories', default='').split(',')
            if c.strip()]
        duration = int_or_none(self._og_search_property(
            'video:duration', webpage, fatal=False))
        timestamp = unified_timestamp(self._og_search_property(
            'video:release_date', webpage, fatal=False))
        view_count = int_or_none(self._search_regex(
            r'>Views\s+(\d+)', webpage, 'view count', fatal=False))
-        return merge_dicts(info, {
+        formats = []
        for video_url_key in ['video_url', 'video_alt_url']:
            video_url = flashvars.get(video_url_key)
            if not video_url:
                continue
            video_text = flashvars.get('%s_text' % video_url_key)
            fmt = {
                'url': video_url,
                'ext': ext,
                'format_id': video_text,
            }
            m = re.search(r'^(?P<height>\d+)[pP]', video_text)
            if m:
                fmt['height'] = int(m.group('height'))
            formats.append(fmt)
        self._sort_formats(formats)
        categories = self._html_search_meta(
            'keywords', webpage, 'categories', default='').split(',')
        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
-            'description': description,
+            'thumbnail': thumbnail,
            'categories': categories,
            'duration': duration,
            'timestamp': timestamp,
            'view_count': view_count,
            'age_limit': 18,
-        })
+            'formats': formats,
        }
--- a/youtube_dl/extractor/imdb.py
+++ b/youtube_dl/extractor/imdb.py
@ -1,7 +1,5 @@
 from __future__ import unicode_literals
 import base64
 import json
 import re
 from .common import InfoExtractor
@ -10,7 +8,6 @@ from ..utils import (
    mimetype2ext,
    parse_duration,
    qualities,
    try_get,
    url_or_none,
 )
@ -18,16 +15,15 @@ from ..utils import (
 class ImdbIE(InfoExtractor):
    IE_NAME = 'imdb'
    IE_DESC = 'Internet Movie Database trailers'
-    _VALID_URL = r'https?://(?:www|m)\.imdb\.com/(?:video|title|list).*?[/-]vi(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www|m)\.imdb\.com/(?:video|title|list).+?[/-]vi(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://www.imdb.com/video/imdb/vi2524815897',
        'info_dict': {
            'id': '2524815897',
            'ext': 'mp4',
-            'title': 'No. 2',
+            'title': 'No. 2 from Ice Age: Continental Drift (2012)',
            'description': 'md5:87bd0bdc61e351f21f20d2d7441cb4e7',
            'duration': 152,
        }
    }, {
        'url': 'http://www.imdb.com/video/_/vi2524815897',
@ -51,23 +47,21 @@ class ImdbIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-
+        webpage = self._download_webpage(
-        data = self._download_json(
+            'https://www.imdb.com/videoplayer/vi' + video_id, video_id)
-            'https://www.imdb.com/ve/data/VIDEO_PLAYBACK_DATA', video_id,
+        video_metadata = self._parse_json(self._search_regex(
-            query={
+            r'window\.IMDbReactInitialState\.push\(({.+?})\);', webpage,
-                'key': base64.b64encode(json.dumps({
+            'video metadata'), video_id)['videos']['videoMetadata']['vi' + video_id]
-                    'type': 'VIDEO_PLAYER',
+        title = self._html_search_meta(
-                    'subType': 'FORCE_LEGACY',
+            ['og:title', 'twitter:title'], webpage) or self._html_search_regex(
-                    'id': 'vi%s' % video_id,
+            r'<title>(.+?)</title>', webpage, 'title', fatal=False) or video_metadata['title']
                }).encode()).decode(),
            })[0]
        quality = qualities(('SD', '480p', '720p', '1080p'))
        formats = []
-        for encoding in data['videoLegacyEncodings']:
+        for encoding in video_metadata.get('encodings', []):
            if not encoding or not isinstance(encoding, dict):
                continue
-            video_url = url_or_none(encoding.get('url'))
+            video_url = url_or_none(encoding.get('videoUrl'))
            if not video_url:
                continue
            ext = mimetype2ext(encoding.get(
@ -75,7 +69,7 @@ class ImdbIE(InfoExtractor):
            if ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
                    video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                    preference=1, m3u8_id='hls', fatal=False))
+                    m3u8_id='hls', fatal=False))
                continue
            format_id = encoding.get('definition')
            formats.append({
@ -86,33 +80,13 @@ class ImdbIE(InfoExtractor):
            })
        self._sort_formats(formats)
        webpage = self._download_webpage(
            'https://www.imdb.com/video/vi' + video_id, video_id)
        video_metadata = self._parse_json(self._search_regex(
            r'args\.push\(\s*({.+?})\s*\)\s*;', webpage,
            'video metadata'), video_id)
        video_info = video_metadata.get('VIDEO_INFO')
        if video_info and isinstance(video_info, dict):
            info = try_get(
                video_info, lambda x: x[list(video_info.keys())[0]][0], dict)
        else:
            info = {}
        title = self._html_search_meta(
            ['og:title', 'twitter:title'], webpage) or self._html_search_regex(
            r'<title>(.+?)</title>', webpage, 'title',
            default=None) or info['videoTitle']
        return {
            'id': video_id,
            'title': title,
            'alt_title': info.get('videoSubTitle'),
            'formats': formats,
-            'description': info.get('videoDescription'),
+            'description': video_metadata.get('description'),
-            'thumbnail': url_or_none(try_get(
+            'thumbnail': video_metadata.get('slate', {}).get('url'),
-                video_metadata, lambda x: x['videoSlate']['source'])),
+            'duration': parse_duration(video_metadata.get('duration')),
            'duration': parse_duration(info.get('videoRuntime')),
        }
--- a/youtube_dl/extractor/imggaming.py
+++ b/youtube_dl/extractor/imggaming.py
@ -1,133 +0,0 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import json
 import re
 from .common import InfoExtractor
 from ..compat import compat_HTTPError
 from ..utils import (
    ExtractorError,
    int_or_none,
    str_or_none,
    try_get,
 )
 class ImgGamingBaseIE(InfoExtractor):
    _API_BASE = 'https://dce-frontoffice.imggaming.com/api/v2/'
    _API_KEY = '857a1e5d-e35e-4fdf-805b-a87b6f8364bf'
    _HEADERS = None
    _MANIFEST_HEADERS = {'Accept-Encoding': 'identity'}
    _REALM = None
    _VALID_URL_TEMPL = r'https?://(?P<domain>%s)/(?P<type>live|playlist|video)/(?P<id>\d+)(?:\?.*?\bplaylistId=(?P<playlist_id>\d+))?'
    def _real_initialize(self):
        self._HEADERS = {
            'Realm': 'dce.' + self._REALM,
            'x-api-key': self._API_KEY,
        }
        email, password = self._get_login_info()
        if email is None:
            self.raise_login_required()
        p_headers = self._HEADERS.copy()
        p_headers['Content-Type'] = 'application/json'
        self._HEADERS['Authorization'] = 'Bearer ' + self._download_json(
            self._API_BASE + 'login',
            None, 'Logging in', data=json.dumps({
                'id': email,
                'secret': password,
            }).encode(), headers=p_headers)['authorisationToken']
    def _call_api(self, path, media_id):
        return self._download_json(
            self._API_BASE + path + media_id, media_id, headers=self._HEADERS)
    def _extract_dve_api_url(self, media_id, media_type):
        stream_path = 'stream'
        if media_type == 'video':
            stream_path += '/vod/'
        else:
            stream_path += '?eventId='
        try:
            return self._call_api(
                stream_path, media_id)['playerUrlCallback']
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
                raise ExtractorError(
                    self._parse_json(e.cause.read().decode(), media_id)['messages'][0],
                    expected=True)
            raise
    def _real_extract(self, url):
        domain, media_type, media_id, playlist_id = re.match(self._VALID_URL, url).groups()
        if playlist_id:
            if self._downloader.params.get('noplaylist'):
                self.to_screen('Downloading just video %s because of --no-playlist' % media_id)
            else:
                self.to_screen('Downloading playlist %s - add --no-playlist to just download video' % playlist_id)
                media_type, media_id = 'playlist', playlist_id
        if media_type == 'playlist':
            playlist = self._call_api('vod/playlist/', media_id)
            entries = []
            for video in try_get(playlist, lambda x: x['videos']['vods']) or []:
                video_id = str_or_none(video.get('id'))
                if not video_id:
                    continue
                entries.append(self.url_result(
                    'https://%s/video/%s' % (domain, video_id),
                    self.ie_key(), video_id))
            return self.playlist_result(
                entries, media_id, playlist.get('title'),
                playlist.get('description'))
        dve_api_url = self._extract_dve_api_url(media_id, media_type)
        video_data = self._download_json(dve_api_url, media_id)
        is_live = media_type == 'live'
        if is_live:
            title = self._live_title(self._call_api('event/', media_id)['title'])
        else:
            title = video_data['name']
        formats = []
        for proto in ('hls', 'dash'):
            media_url = video_data.get(proto + 'Url') or try_get(video_data, lambda x: x[proto]['url'])
            if not media_url:
                continue
            if proto == 'hls':
                m3u8_formats = self._extract_m3u8_formats(
                    media_url, media_id, 'mp4', 'm3u8' if is_live else 'm3u8_native',
                    m3u8_id='hls', fatal=False, headers=self._MANIFEST_HEADERS)
                for f in m3u8_formats:
                    f.setdefault('http_headers', {}).update(self._MANIFEST_HEADERS)
                    formats.append(f)
            else:
                formats.extend(self._extract_mpd_formats(
                    media_url, media_id, mpd_id='dash', fatal=False,
                    headers=self._MANIFEST_HEADERS))
        self._sort_formats(formats)
        subtitles = {}
        for subtitle in video_data.get('subtitles', []):
            subtitle_url = subtitle.get('url')
            if not subtitle_url:
                continue
            subtitles.setdefault(subtitle.get('lang', 'en_US'), []).append({
                'url': subtitle_url,
            })
        return {
            'id': media_id,
            'title': title,
            'formats': formats,
            'thumbnail': video_data.get('thumbnailUrl'),
            'description': video_data.get('description'),
            'duration': int_or_none(video_data.get('duration')),
            'tags': video_data.get('tags'),
            'is_live': is_live,
            'subtitles': subtitles,
        }
--- a/youtube_dl/extractor/indavideo.py
+++ b/youtube_dl/extractor/indavideo.py
@ -58,7 +58,7 @@ class IndavideoEmbedIE(InfoExtractor):
        video_id = self._match_id(url)
        video = self._download_json(
-            'https://amfphp.indavideo.hu/SYm0json.php/player.playerHandler.getVideoData/%s' % video_id,
+            'http://amfphp.indavideo.hu/SYm0json.php/player.playerHandler.getVideoData/%s' % video_id,
            video_id)['data']
        title = video['title']
--- a/youtube_dl/extractor/iprima.py
+++ b/youtube_dl/extractor/iprima.py
@ -16,22 +16,12 @@ class IPrimaIE(InfoExtractor):
    _GEO_BYPASS = False
    _TESTS = [{
-        'url': 'https://prima.iprima.cz/particka/92-epizoda',
+        'url': 'http://play.iprima.cz/gondici-s-r-o-33',
        'info_dict': {
-            'id': 'p51388',
+            'id': 'p136534',
            'ext': 'mp4',
-            'title': 'Partička (92)',
+            'title': 'Gondíci s. r. o. (34)',
-            'description': 'md5:859d53beae4609e6dd7796413f1b6cac',
+            'description': 'md5:16577c629d006aa91f59ca8d8e7f99bd',
        },
        'params': {
            'skip_download': True,  # m3u8 download
        },
    }, {
        'url': 'https://cnn.iprima.cz/videa/70-epizoda',
        'info_dict': {
            'id': 'p681554',
            'ext': 'mp4',
            'title': 'HLAVNÍ ZPRÁVY 3.5.2020',
        },
        'params': {
            'skip_download': True,  # m3u8 download
@ -78,16 +68,9 @@ class IPrimaIE(InfoExtractor):
        webpage = self._download_webpage(url, video_id)
        title = self._og_search_title(
            webpage, default=None) or self._search_regex(
            r'<h1>([^<]+)', webpage, 'title')
        video_id = self._search_regex(
            (r'<iframe[^>]+\bsrc=["\'](?:https?:)?//(?:api\.play-backend\.iprima\.cz/prehravac/embedded|prima\.iprima\.cz/[^/]+/[^/]+)\?.*?\bid=(p\d+)',
-             r'data-product="([^"]+)">',
+             r'data-product="([^"]+)">'),
             r'id=["\']player-(p\d+)"',
             r'playerId\s*:\s*["\']player-(p\d+)',
             r'\bvideos\s*=\s*["\'](p\d+)'),
            webpage, 'real id')
        playerpage = self._download_webpage(
@ -142,8 +125,8 @@ class IPrimaIE(InfoExtractor):
        return {
            'id': video_id,
-            'title': title,
+            'title': self._og_search_title(webpage),
-            'thumbnail': self._og_search_thumbnail(webpage, default=None),
+            'thumbnail': self._og_search_thumbnail(webpage),
            'formats': formats,
-            'description': self._og_search_description(webpage, default=None),
+            'description': self._og_search_description(webpage),
        }
--- a/youtube_dl/extractor/iqiyi.py
+++ b/youtube_dl/extractor/iqiyi.py
@ -150,7 +150,7 @@ class IqiyiSDKInterpreter(object):
            elif function in other_functions:
                other_functions[function]()
            else:
-                raise ExtractorError('Unknown function %s' % function)
+                raise ExtractorError('Unknown funcion %s' % function)
        return sdk.target
--- a/youtube_dl/extractor/ivi.py
+++ b/youtube_dl/extractor/ivi.py
@ -239,7 +239,7 @@ class IviCompilationIE(InfoExtractor):
            self.url_result(
                'http://www.ivi.ru/watch/%s/%s' % (compilation_id, serie), IviIE.ie_key())
            for serie in re.findall(
-                r'<a\b[^>]+\bhref=["\']/watch/%s/(\d+)["\']' % compilation_id, html)]
+                r'<a href="/watch/%s/(\d+)"[^>]+data-id="\1"' % compilation_id, html)]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
--- a/youtube_dl/extractor/jpopsukitv.py
+++ b/youtube_dl/extractor/jpopsukitv.py
@ -0,0 +1,68 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    unified_strdate,
 )
 class JpopsukiIE(InfoExtractor):
    IE_NAME = 'jpopsuki.tv'
    _VALID_URL = r'https?://(?:www\.)?jpopsuki\.tv/(?:category/)?video/[^/]+/(?P<id>\S+)'
    _TEST = {
        'url': 'http://www.jpopsuki.tv/video/ayumi-hamasaki---evolution/00be659d23b0b40508169cdee4545771',
        'md5': '88018c0c1a9b1387940e90ec9e7e198e',
        'info_dict': {
            'id': '00be659d23b0b40508169cdee4545771',
            'ext': 'mp4',
            'title': 'ayumi hamasaki - evolution',
            'description': 'Release date: 2001.01.31\r\n浜崎あゆみ - evolution',
            'thumbnail': 'http://www.jpopsuki.tv/cache/89722c74d2a2ebe58bcac65321c115b2.jpg',
            'uploader': 'plama_chan',
            'uploader_id': '404',
            'upload_date': '20121101'
        }
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        video_url = 'http://www.jpopsuki.tv' + self._html_search_regex(
            r'<source src="(.*?)" type', webpage, 'video url')
        video_title = self._og_search_title(webpage)
        description = self._og_search_description(webpage)
        thumbnail = self._og_search_thumbnail(webpage)
        uploader = self._html_search_regex(
            r'<li>from: <a href="/user/view/user/(.*?)/uid/',
            webpage, 'video uploader', fatal=False)
        uploader_id = self._html_search_regex(
            r'<li>from: <a href="/user/view/user/\S*?/uid/(\d*)',
            webpage, 'video uploader_id', fatal=False)
        upload_date = unified_strdate(self._html_search_regex(
            r'<li>uploaded: (.*?)</li>', webpage, 'video upload_date',
            fatal=False))
        view_count_str = self._html_search_regex(
            r'<li>Hits: ([0-9]+?)</li>', webpage, 'video view_count',
            fatal=False)
        comment_count_str = self._html_search_regex(
            r'<h2>([0-9]+?) comments</h2>', webpage, 'video comment_count',
            fatal=False)
        return {
            'id': video_id,
            'url': video_url,
            'title': video_title,
            'description': description,
            'thumbnail': thumbnail,
            'uploader': uploader,
            'uploader_id': uploader_id,
            'upload_date': upload_date,
            'view_count': int_or_none(view_count_str),
            'comment_count': int_or_none(comment_count_str),
        }
--- a/youtube_dl/extractor/jwplatform.py
+++ b/youtube_dl/extractor/jwplatform.py
@ -4,7 +4,6 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import unsmuggle_url
 class JWPlatformIE(InfoExtractor):
@ -33,14 +32,10 @@ class JWPlatformIE(InfoExtractor):
    @staticmethod
    def _extract_urls(webpage):
        return re.findall(
-            r'<(?:script|iframe)[^>]+?src=["\']((?:https?:)?//(?:content\.jwplatform|cdn\.jwplayer)\.com/players/[a-zA-Z0-9]{8})',
+            r'<(?:script|iframe)[^>]+?src=["\']((?:https?:)?//content\.jwplatform\.com/players/[a-zA-Z0-9]{8})',
            webpage)
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
        self._initialize_geo_bypass({
            'countries': smuggled_data.get('geo_countries'),
        })
        video_id = self._match_id(url)
        json_data = self._download_json('https://cdn.jwplayer.com/v2/media/' + video_id, video_id)
        return self._parse_jwplayer_data(json_data, video_id)
--- a/youtube_dl/extractor/kaltura.py
+++ b/youtube_dl/extractor/kaltura.py
@ -113,14 +113,9 @@ class KalturaIE(InfoExtractor):
    @staticmethod
    def _extract_url(webpage):
        urls = KalturaIE._extract_urls(webpage)
        return urls[0] if urls else None
    @staticmethod
    def _extract_urls(webpage):
        # Embed codes: https://knowledge.kaltura.com/embedding-kaltura-media-players-your-site
-        finditer = (
+        mobj = (
-            re.finditer(
+            re.search(
                r"""(?xs)
                    kWidget\.(?:thumb)?[Ee]mbed\(
                    \{.*?
@ -129,7 +124,7 @@ class KalturaIE(InfoExtractor):
                        (?P<q3>['"])entry_?[Ii]d(?P=q3)\s*:\s*
                        (?P<q4>['"])(?P<id>(?:(?!(?P=q4)).)+)(?P=q4)(?:,|\s*\})
                """, webpage)
-            or re.finditer(
+            or re.search(
                r'''(?xs)
                    (?P<q1>["'])
                        (?:https?:)?//cdnapi(?:sec)?\.kaltura\.com(?::\d+)?/(?:(?!(?P=q1)).)*\b(?:p|partner_id)/(?P<partner_id>\d+)(?:(?!(?P=q1)).)*
@ -143,7 +138,7 @@ class KalturaIE(InfoExtractor):
                    )
                    (?P<q3>["'])(?P<id>(?:(?!(?P=q3)).)+)(?P=q3)
                ''', webpage)
-            or re.finditer(
+            or re.search(
                r'''(?xs)
                    <(?:iframe[^>]+src|meta[^>]+\bcontent)=(?P<q1>["'])
                      (?:https?:)?//(?:(?:www|cdnapi(?:sec)?)\.)?kaltura\.com/(?:(?!(?P=q1)).)*\b(?:p|partner_id)/(?P<partner_id>\d+)
@ -153,8 +148,7 @@ class KalturaIE(InfoExtractor):
                    (?P=q1)
                ''', webpage)
        )
-        urls = []
+        if mobj:
        for mobj in finditer:
            embed_info = mobj.groupdict()
            for k, v in embed_info.items():
                if v:
@ -166,8 +160,7 @@ class KalturaIE(InfoExtractor):
                webpage)
            if service_mobj:
                url = smuggle_url(url, {'service_url': service_mobj.group('id')})
-            urls.append(url)
+            return url
        return urls
    def _kaltura_api_call(self, video_id, actions, service_url=None, *args, **kwargs):
        params = actions[0]
--- a/youtube_dl/extractor/kontrtube.py
+++ b/youtube_dl/extractor/kontrtube.py
@ -0,0 +1,73 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    parse_duration,
 )
 class KontrTubeIE(InfoExtractor):
    IE_NAME = 'kontrtube'
    IE_DESC = 'KontrTube.ru - Труба зовёт'
    _VALID_URL = r'https?://(?:www\.)?kontrtube\.ru/videos/(?P<id>\d+)/(?P<display_id>[^/]+)/'
    _TEST = {
        'url': 'http://www.kontrtube.ru/videos/2678/nad-olimpiyskoy-derevney-v-sochi-podnyat-rossiyskiy-flag/',
        'md5': '975a991a4926c9a85f383a736a2e6b80',
        'info_dict': {
            'id': '2678',
            'display_id': 'nad-olimpiyskoy-derevney-v-sochi-podnyat-rossiyskiy-flag',
            'ext': 'mp4',
            'title': 'Над олимпийской деревней в Сочи поднят российский флаг',
            'description': 'md5:80edc4c613d5887ae8ccf1d59432be41',
            'thumbnail': 'http://www.kontrtube.ru/contents/videos_screenshots/2000/2678/preview.mp4.jpg',
            'duration': 270,
        }
    }
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('id')
        display_id = mobj.group('display_id')
        webpage = self._download_webpage(
            url, display_id, 'Downloading page')
        video_url = self._search_regex(
            r"video_url\s*:\s*'(.+?)/?',", webpage, 'video URL')
        thumbnail = self._search_regex(
            r"preview_url\s*:\s*'(.+?)/?',", webpage, 'thumbnail', fatal=False)
        title = self._html_search_regex(
            r'(?s)<h2>(.+?)</h2>', webpage, 'title')
        description = self._html_search_meta(
            'description', webpage, 'description')
        duration = self._search_regex(
            r'Длительность: <em>([^<]+)</em>', webpage, 'duration', fatal=False)
        if duration:
            duration = parse_duration(duration.replace('мин', 'min').replace('сек', 'sec'))
        view_count = self._search_regex(
            r'Просмотров: <em>([^<]+)</em>',
            webpage, 'view count', fatal=False)
        if view_count:
            view_count = int_or_none(view_count.replace(' ', ''))
        comment_count = int_or_none(self._search_regex(
            r'Комментарии \((\d+)\)<', webpage, ' comment count', fatal=False))
        return {
            'id': video_id,
            'display_id': display_id,
            'url': video_url,
            'thumbnail': thumbnail,
            'title': title,
            'description': description,
            'duration': duration,
            'view_count': int_or_none(view_count),
            'comment_count': int_or_none(comment_count),
        }
--- a/youtube_dl/extractor/lecturio.py
+++ b/youtube_dl/extractor/lecturio.py
@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    clean_html,
    determine_ext,
@ -35,7 +36,7 @@ class LecturioBaseIE(InfoExtractor):
            self._LOGIN_URL, None, 'Downloading login popup')
        def is_logged(url_handle):
-            return self._LOGIN_URL not in url_handle.geturl()
+            return self._LOGIN_URL not in compat_str(url_handle.geturl())
        # Already logged in
        if is_logged(urlh):
--- a/youtube_dl/extractor/lego.py
+++ b/youtube_dl/extractor/lego.py
@ -2,24 +2,23 @@
 from __future__ import unicode_literals
 import re
 import uuid
 from .common import InfoExtractor
-from ..compat import compat_HTTPError
+from ..compat import compat_str
 from ..utils import (
-    ExtractorError,
+    unescapeHTML,
-    int_or_none,
+    parse_duration,
-    qualities,
+    get_element_by_class,
 )
 class LEGOIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?lego\.com/(?P<locale>[a-z]{2}-[a-z]{2})/(?:[^/]+/)*videos/(?:[^/]+/)*[^/?#]+-(?P<id>[0-9a-f]{32})'
+    _VALID_URL = r'https?://(?:www\.)?lego\.com/(?P<locale>[^/]+)/(?:[^/]+/)*videos/(?:[^/]+/)*[^/?#]+-(?P<id>[0-9a-f]+)'
    _TESTS = [{
        'url': 'http://www.lego.com/en-us/videos/themes/club/blocumentary-kawaguchi-55492d823b1b4d5e985787fa8c2973b1',
        'md5': 'f34468f176cfd76488767fc162c405fa',
        'info_dict': {
-            'id': '55492d82-3b1b-4d5e-9857-87fa8c2973b1_en-US',
+            'id': '55492d823b1b4d5e985787fa8c2973b1',
            'ext': 'mp4',
            'title': 'Blocumentary Great Creations: Akiyuki Kawaguchi',
            'description': 'Blocumentary Great Creations: Akiyuki Kawaguchi',
@ -27,123 +26,103 @@ class LEGOIE(InfoExtractor):
    }, {
        # geo-restricted but the contentUrl contain a valid url
        'url': 'http://www.lego.com/nl-nl/videos/themes/nexoknights/episode-20-kingdom-of-heroes-13bdc2299ab24d9685701a915b3d71e7##sp=399',
-        'md5': 'c7420221f7ffd03ff056f9db7f8d807c',
+        'md5': '4c3fec48a12e40c6e5995abc3d36cc2e',
        'info_dict': {
-            'id': '13bdc229-9ab2-4d96-8570-1a915b3d71e7_nl-NL',
+            'id': '13bdc2299ab24d9685701a915b3d71e7',
            'ext': 'mp4',
-            'title': 'Aflevering 20:  Helden van het koninkrijk',
+            'title': 'Aflevering 20 - Helden van het koninkrijk',
            'description': 'md5:8ee499aac26d7fa8bcb0cedb7f9c3941',
            'age_limit': 5,
        },
    }, {
-        # with subtitle
+        # special characters in title
-        'url': 'https://www.lego.com/nl-nl/kids/videos/classic/creative-storytelling-the-little-puppy-aa24f27c7d5242bc86102ebdc0f24cba',
+        'url': 'http://www.lego.com/en-us/starwars/videos/lego-star-wars-force-surprise-9685ee9d12e84ff38e84b4e3d0db533d',
        'info_dict': {
-            'id': 'aa24f27c-7d52-42bc-8610-2ebdc0f24cba_nl-NL',
+            'id': '9685ee9d12e84ff38e84b4e3d0db533d',
            'ext': 'mp4',
-            'title': 'De kleine puppy',
+            'title': 'Force Surprise – LEGO® Star Wars™ Microfighters',
-            'description': 'md5:5b725471f849348ac73f2e12cfb4be06',
+            'description': 'md5:9c673c96ce6f6271b88563fe9dc56de3',
            'age_limit': 1,
            'subtitles': {
                'nl': [{
                    'ext': 'srt',
                    'url': r're:^https://.+\.srt$',
                }],
            },
        },
        'params': {
            'skip_download': True,
        },
    }]
-    _QUALITIES = {
+    _BITRATES = [256, 512, 1024, 1536, 2560]
        'Lowest': (64, 180, 320),
        'Low': (64, 270, 480),
        'Medium': (96, 360, 640),
        'High': (128, 540, 960),
        'Highest': (128, 720, 1280),
    }
    def _real_extract(self, url):
        locale, video_id = re.match(self._VALID_URL, url).groups()
-        countries = [locale.split('-')[1].upper()]
+        webpage = self._download_webpage(url, video_id)
-        self._initialize_geo_bypass({
+        title = get_element_by_class('video-header', webpage).strip()
-            'countries': countries,
+        progressive_base = 'https://lc-mediaplayerns-live-s.legocdn.com/'
-        })
+        streaming_base = 'http://legoprod-f.akamaihd.net/'
        content_url = self._html_search_meta('contentUrl', webpage)
        path = self._search_regex(
            r'(?:https?:)?//[^/]+/(?:[iz]/s/)?public/(.+)_[0-9,]+\.(?:mp4|webm)',
            content_url, 'video path', default=None)
        if not path:
            player_url = self._proto_relative_url(self._search_regex(
                r'<iframe[^>]+src="((?:https?)?//(?:www\.)?lego\.com/[^/]+/mediaplayer/video/[^"]+)',
                webpage, 'player url', default=None))
            if not player_url:
                base_url = self._proto_relative_url(self._search_regex(
                    r'data-baseurl="([^"]+)"', webpage, 'base url',
                    default='http://www.lego.com/%s/mediaplayer/video/' % locale))
                player_url = base_url + video_id
            player_webpage = self._download_webpage(player_url, video_id)
            video_data = self._parse_json(unescapeHTML(self._search_regex(
                r"video='([^']+)'", player_webpage, 'video data')), video_id)
            progressive_base = self._search_regex(
                r'data-video-progressive-url="([^"]+)"',
                player_webpage, 'progressive base', default='https://lc-mediaplayerns-live-s.legocdn.com/')
            streaming_base = self._search_regex(
                r'data-video-streaming-url="([^"]+)"',
                player_webpage, 'streaming base', default='http://legoprod-f.akamaihd.net/')
            item_id = video_data['ItemId']
-        try:
+            net_storage_path = video_data.get('NetStoragePath') or '/'.join([item_id[:2], item_id[2:4]])
-            item = self._download_json(
+            base_path = '_'.join([item_id, video_data['VideoId'], video_data['Locale'], compat_str(video_data['VideoVersion'])])
-                # https://contentfeed.services.lego.com/api/v2/item/[VIDEO_ID]?culture=[LOCALE]&contentType=Video
+            path = '/'.join([net_storage_path, base_path])
-                'https://services.slingshot.lego.com/mediaplayer/v2',
+        streaming_path = ','.join(map(lambda bitrate: compat_str(bitrate), self._BITRATES))
                video_id, query={
                    'videoId': '%s_%s' % (uuid.UUID(video_id), locale),
                }, headers=self.geo_verification_headers())
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 451:
                self.raise_geo_restricted(countries=countries)
            raise
-        video = item['Video']
+        formats = self._extract_akamai_formats(
-        video_id = video['Id']
+            '%si/s/public/%s_,%s,.mp4.csmil/master.m3u8' % (streaming_base, path, streaming_path), video_id)
-        title = video['Title']
+        m3u8_formats = list(filter(
-
+            lambda f: f.get('protocol') == 'm3u8_native' and f.get('vcodec') != 'none',
-        q = qualities(['Lowest', 'Low', 'Medium', 'High', 'Highest'])
+            formats))
-        formats = []
+        if len(m3u8_formats) == len(self._BITRATES):
-        for video_source in item.get('VideoFormats', []):
+            self._sort_formats(m3u8_formats)
-            video_source_url = video_source.get('Url')
+            for bitrate, m3u8_format in zip(self._BITRATES, m3u8_formats):
-            if not video_source_url:
+                progressive_base_url = '%spublic/%s_%d.' % (progressive_base, path, bitrate)
-                continue
+                mp4_f = m3u8_format.copy()
-            video_source_format = video_source.get('Format')
+                mp4_f.update({
-            if video_source_format == 'F4M':
+                    'url': progressive_base_url + 'mp4',
-                formats.extend(self._extract_f4m_formats(
+                    'format_id': m3u8_format['format_id'].replace('hls', 'mp4'),
-                    video_source_url, video_id,
+                    'protocol': 'http',
                    f4m_id=video_source_format, fatal=False))
            elif video_source_format == 'M3U8':
                formats.extend(self._extract_m3u8_formats(
                    video_source_url, video_id, 'mp4', 'm3u8_native',
                    m3u8_id=video_source_format, fatal=False))
            else:
                video_source_quality = video_source.get('Quality')
                format_id = []
                for v in (video_source_format, video_source_quality):
                    if v:
                        format_id.append(v)
                f = {
                    'format_id': '-'.join(format_id),
                    'quality': q(video_source_quality),
                    'url': video_source_url,
                }
                quality = self._QUALITIES.get(video_source_quality)
                if quality:
                    f.update({
                        'abr': quality[0],
                        'height': quality[1],
                        'width': quality[2],
                    }),
                formats.append(f)
        self._sort_formats(formats)
        subtitles = {}
        sub_file_id = video.get('SubFileId')
        if sub_file_id and sub_file_id != '00000000-0000-0000-0000-000000000000':
            net_storage_path = video.get('NetstoragePath')
            invariant_id = video.get('InvariantId')
            video_file_id = video.get('VideoFileId')
            video_version = video.get('VideoVersion')
            if net_storage_path and invariant_id and video_file_id and video_version:
                subtitles.setdefault(locale[:2], []).append({
                    'url': 'https://lc-mediaplayerns-live-s.legocdn.com/public/%s/%s_%s_%s_%s_sub.srt' % (net_storage_path, invariant_id, video_file_id, locale, video_version),
                })
                web_f = {
                    'url': progressive_base_url + 'webm',
                    'format_id': m3u8_format['format_id'].replace('hls', 'webm'),
                    'width': m3u8_format['width'],
                    'height': m3u8_format['height'],
                    'tbr': m3u8_format.get('tbr'),
                    'ext': 'webm',
                }
                formats.extend([web_f, mp4_f])
        else:
            for bitrate in self._BITRATES:
                for ext in ('web', 'mp4'):
                    formats.append({
                        'format_id': '%s-%s' % (ext, bitrate),
                        'url': '%spublic/%s_%d.%s' % (progressive_base, path, bitrate, ext),
                        'tbr': bitrate,
                        'ext': ext,
                    })
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': title,
-            'description': video.get('Description'),
+            'description': self._html_search_meta('description', webpage),
-            'thumbnail': video.get('GeneratedCoverImage') or video.get('GeneratedThumbnail'),
+            'thumbnail': self._html_search_meta('thumbnail', webpage),
-            'duration': int_or_none(video.get('Length')),
+            'duration': parse_duration(self._html_search_meta('duration', webpage)),
            'formats': formats,
            'subtitles': subtitles,
            'age_limit': int_or_none(video.get('AgeFrom')),
            'season': video.get('SeasonTitle'),
            'season_number': int_or_none(video.get('Season')) or None,
            'episode_number': int_or_none(video.get('Episode')) or None,
        }
--- a/youtube_dl/extractor/limelight.py
+++ b/youtube_dl/extractor/limelight.py
@ -18,6 +18,7 @@ from ..utils import (
 class LimelightBaseIE(InfoExtractor):
    _PLAYLIST_SERVICE_URL = 'http://production-ps.lvp.llnw.net/r/PlaylistService/%s/%s/%s'
    _API_URL = 'http://api.video.limelight.com/rest/organizations/%s/%s/%s/%s.json'
    @classmethod
    def _extract_urls(cls, webpage, source_url):
@ -69,8 +70,7 @@ class LimelightBaseIE(InfoExtractor):
        try:
            return self._download_json(
                self._PLAYLIST_SERVICE_URL % (self._PLAYLIST_SERVICE_PATH, item_id, method),
-                item_id, 'Downloading PlaylistService %s JSON' % method,
+                item_id, 'Downloading PlaylistService %s JSON' % method, fatal=fatal, headers=headers)
                fatal=fatal, headers=headers)
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
                error = self._parse_json(e.cause.read().decode(), item_id)['detail']['contentAccessPermission']
@ -79,22 +79,22 @@ class LimelightBaseIE(InfoExtractor):
                raise ExtractorError(error, expected=True)
            raise
-    def _extract(self, item_id, pc_method, mobile_method, referer=None):
+    def _call_api(self, organization_id, item_id, method):
        return self._download_json(
            self._API_URL % (organization_id, self._API_PATH, item_id, method),
            item_id, 'Downloading API %s JSON' % method)
    def _extract(self, item_id, pc_method, mobile_method, meta_method, referer=None):
        pc = self._call_playlist_service(item_id, pc_method, referer=referer)
-        mobile = self._call_playlist_service(
+        metadata = self._call_api(pc['orgId'], item_id, meta_method)
-            item_id, mobile_method, fatal=False, referer=referer)
+        mobile = self._call_playlist_service(item_id, mobile_method, fatal=False, referer=referer)
-        return pc, mobile
+        return pc, mobile, metadata
    def _extract_info(self, pc, mobile, i, referer):
        get_item = lambda x, y: try_get(x, lambda x: x[y][i], dict) or {}
        pc_item = get_item(pc, 'playlistItems')
        mobile_item = get_item(mobile, 'mediaList')
        video_id = pc_item.get('mediaId') or mobile_item['mediaId']
        title = pc_item.get('title') or mobile_item['title']
    def _extract_info(self, streams, mobile_urls, properties):
        video_id = properties['media_id']
        formats = []
        urls = []
-        for stream in pc_item.get('streams', []):
+        for stream in streams:
            stream_url = stream.get('url')
            if not stream_url or stream.get('drmProtected') or stream_url in urls:
                continue
@ -155,7 +155,7 @@ class LimelightBaseIE(InfoExtractor):
                    })
                formats.append(fmt)
-        for mobile_url in mobile_item.get('mobileUrls', []):
+        for mobile_url in mobile_urls:
            media_url = mobile_url.get('mobileUrl')
            format_id = mobile_url.get('targetMediaPlatform')
            if not media_url or format_id in ('Widevine', 'SmoothStreaming') or media_url in urls:
@ -179,34 +179,54 @@ class LimelightBaseIE(InfoExtractor):
        self._sort_formats(formats)
-        subtitles = {}
+        title = properties['title']
-        for flag in mobile_item.get('flags'):
+        description = properties.get('description')
-            if flag == 'ClosedCaptions':
+        timestamp = int_or_none(properties.get('publish_date') or properties.get('create_date'))
-                closed_captions = self._call_playlist_service(
+        duration = float_or_none(properties.get('duration_in_milliseconds'), 1000)
-                    video_id, 'getClosedCaptionsDetailsByMediaId',
+        filesize = int_or_none(properties.get('total_storage_in_bytes'))
-                    False, referer) or []
+        categories = [properties.get('category')]
-                for cc in closed_captions:
+        tags = properties.get('tags', [])
-                    cc_url = cc.get('webvttFileUrl')
+        thumbnails = [{
-                    if not cc_url:
+            'url': thumbnail['url'],
-                        continue
+            'width': int_or_none(thumbnail.get('width')),
-                    lang = cc.get('languageCode') or self._search_regex(r'/[a-z]{2}\.vtt', cc_url, 'lang', default='en')
+            'height': int_or_none(thumbnail.get('height')),
-                    subtitles.setdefault(lang, []).append({
+        } for thumbnail in properties.get('thumbnails', []) if thumbnail.get('url')]
                        'url': cc_url,
                    })
                break
-        get_meta = lambda x: pc_item.get(x) or mobile_item.get(x)
+        subtitles = {}
        for caption in properties.get('captions', []):
            lang = caption.get('language_code')
            subtitles_url = caption.get('url')
            if lang and subtitles_url:
                subtitles.setdefault(lang, []).append({
                    'url': subtitles_url,
                })
        closed_captions_url = properties.get('closed_captions_url')
        if closed_captions_url:
            subtitles.setdefault('en', []).append({
                'url': closed_captions_url,
                'ext': 'ttml',
            })
        return {
            'id': video_id,
            'title': title,
-            'description': get_meta('description'),
+            'description': description,
            'formats': formats,
-            'duration': float_or_none(get_meta('durationInMilliseconds'), 1000),
+            'timestamp': timestamp,
-            'thumbnail': get_meta('previewImageUrl') or get_meta('thumbnailImageUrl'),
+            'duration': duration,
            'filesize': filesize,
            'categories': categories,
            'tags': tags,
            'thumbnails': thumbnails,
            'subtitles': subtitles,
        }
    def _extract_info_helper(self, pc, mobile, i, metadata):
        return self._extract_info(
            try_get(pc, lambda x: x['playlistItems'][i]['streams'], list) or [],
            try_get(mobile, lambda x: x['mediaList'][i]['mobileUrls'], list) or [],
            metadata)
 class LimelightMediaIE(LimelightBaseIE):
    IE_NAME = 'limelight'
@ -231,6 +251,8 @@ class LimelightMediaIE(LimelightBaseIE):
            'description': 'md5:8005b944181778e313d95c1237ddb640',
            'thumbnail': r're:^https?://.*\.jpeg$',
            'duration': 144.23,
            'timestamp': 1244136834,
            'upload_date': '20090604',
        },
        'params': {
            # m3u8 download
@ -246,29 +268,30 @@ class LimelightMediaIE(LimelightBaseIE):
            'title': '3Play Media Overview Video',
            'thumbnail': r're:^https?://.*\.jpeg$',
            'duration': 78.101,
-            # TODO: extract all languages that were accessible via API
+            'timestamp': 1338929955,
-            # 'subtitles': 'mincount:9',
+            'upload_date': '20120605',
-            'subtitles': 'mincount:1',
+            'subtitles': 'mincount:9',
        },
    }, {
        'url': 'https://assets.delvenetworks.com/player/loader.swf?mediaId=8018a574f08d416e95ceaccae4ba0452',
        'only_matching': True,
    }]
    _PLAYLIST_SERVICE_PATH = 'media'
    _API_PATH = 'media'
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
        video_id = self._match_id(url)
        source_url = smuggled_data.get('source_url')
        self._initialize_geo_bypass({
            'countries': smuggled_data.get('geo_countries'),
        })
-        pc, mobile = self._extract(
+        pc, mobile, metadata = self._extract(
            video_id, 'getPlaylistByMediaId',
-            'getMobilePlaylistByMediaId', source_url)
+            'getMobilePlaylistByMediaId', 'properties',
            smuggled_data.get('source_url'))
-        return self._extract_info(pc, mobile, 0, source_url)
+        return self._extract_info_helper(pc, mobile, 0, metadata)
 class LimelightChannelIE(LimelightBaseIE):
@ -290,7 +313,6 @@ class LimelightChannelIE(LimelightBaseIE):
        'info_dict': {
            'id': 'ab6a524c379342f9b23642917020c082',
            'title': 'Javascript Sample Code',
            'description': 'Javascript Sample Code - http://www.delvenetworks.com/sample-code/playerCode-demo.html',
        },
        'playlist_mincount': 3,
    }, {
@ -298,23 +320,22 @@ class LimelightChannelIE(LimelightBaseIE):
        'only_matching': True,
    }]
    _PLAYLIST_SERVICE_PATH = 'channel'
    _API_PATH = 'channels'
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
        channel_id = self._match_id(url)
        source_url = smuggled_data.get('source_url')
-        pc, mobile = self._extract(
+        pc, mobile, medias = self._extract(
            channel_id, 'getPlaylistByChannelId',
            'getMobilePlaylistWithNItemsByChannelId?begin=0&count=-1',
-            source_url)
+            'media', smuggled_data.get('source_url'))
        entries = [
-            self._extract_info(pc, mobile, i, source_url)
+            self._extract_info_helper(pc, mobile, i, medias['media_list'][i])
-            for i in range(len(pc['playlistItems']))]
+            for i in range(len(medias['media_list']))]
-        return self.playlist_result(
+        return self.playlist_result(entries, channel_id, pc['title'])
            entries, channel_id, pc.get('title'), mobile.get('description'))
 class LimelightChannelListIE(LimelightBaseIE):
@ -347,12 +368,10 @@ class LimelightChannelListIE(LimelightBaseIE):
    def _real_extract(self, url):
        channel_list_id = self._match_id(url)
-        channel_list = self._call_playlist_service(
+        channel_list = self._call_playlist_service(channel_list_id, 'getMobileChannelListById')
            channel_list_id, 'getMobileChannelListById')
        entries = [
            self.url_result('limelight:channel:%s' % channel['id'], 'LimelightChannel')
            for channel in channel_list['channelList']]
-        return self.playlist_result(
+        return self.playlist_result(entries, channel_list_id, channel_list['title'])
            entries, channel_list_id, channel_list['title'])
--- a/youtube_dl/extractor/linuxacademy.py
+++ b/youtube_dl/extractor/linuxacademy.py
@ -8,6 +8,7 @@ from .common import InfoExtractor
 from ..compat import (
    compat_b64decode,
    compat_HTTPError,
    compat_str,
 )
 from ..utils import (
    ExtractorError,
@ -98,7 +99,7 @@ class LinuxAcademyIE(InfoExtractor):
            'sso': 'true',
        })
-        login_state_url = urlh.geturl()
+        login_state_url = compat_str(urlh.geturl())
        try:
            login_page = self._download_webpage(
@ -128,7 +129,7 @@ class LinuxAcademyIE(InfoExtractor):
            })
        access_token = self._search_regex(
-            r'access_token=([^=&]+)', urlh.geturl(),
+            r'access_token=([^=&]+)', compat_str(urlh.geturl()),
            'access token')
        self._download_webpage(
--- a/youtube_dl/extractor/mailru.py
+++ b/youtube_dl/extractor/mailru.py
@ -20,10 +20,10 @@ class MailRuIE(InfoExtractor):
    IE_DESC = 'Видео@Mail.Ru'
    _VALID_URL = r'''(?x)
                    https?://
-                        (?:(?:www|m)\.)?my\.mail\.ru/+
+                        (?:(?:www|m)\.)?my\.mail\.ru/
                        (?:
                            video/.*\#video=/?(?P<idv1>(?:[^/]+/){3}\d+)|
-                            (?:(?P<idv2prefix>(?:[^/]+/+){2})video/(?P<idv2suffix>[^/]+/\d+))\.html|
+                            (?:(?P<idv2prefix>(?:[^/]+/){2})video/(?P<idv2suffix>[^/]+/\d+))\.html|
                            (?:video/embed|\+/video/meta)/(?P<metaid>\d+)
                        )
                    '''
@ -85,14 +85,6 @@ class MailRuIE(InfoExtractor):
        {
            'url': 'http://my.mail.ru/+/video/meta/7949340477499637815',
            'only_matching': True,
        },
        {
            'url': 'https://my.mail.ru//list/sinyutin10/video/_myvideo/4.html',
            'only_matching': True,
        },
        {
            'url': 'https://my.mail.ru//list//sinyutin10/video/_myvideo/4.html',
            'only_matching': True,
        }
    ]
@ -128,12 +120,6 @@ class MailRuIE(InfoExtractor):
                'http://api.video.mail.ru/videos/%s.json?new=1' % video_id,
                video_id, 'Downloading video JSON')
        headers = {}
        video_key = self._get_cookies('https://my.mail.ru').get('video_key')
        if video_key:
            headers['Cookie'] = 'video_key=%s' % video_key.value
        formats = []
        for f in video_data['videos']:
            video_url = f.get('url')
@ -146,7 +132,6 @@ class MailRuIE(InfoExtractor):
                'url': video_url,
                'format_id': format_id,
                'height': height,
                'http_headers': headers,
            })
        self._sort_formats(formats)
@ -252,7 +237,7 @@ class MailRuMusicSearchBaseIE(InfoExtractor):
 class MailRuMusicIE(MailRuMusicSearchBaseIE):
    IE_NAME = 'mailru:music'
    IE_DESC = 'Музыка@Mail.Ru'
-    _VALID_URL = r'https?://my\.mail\.ru/+music/+songs/+[^/?#&]+-(?P<id>[\da-f]+)'
+    _VALID_URL = r'https?://my\.mail\.ru/music/songs/[^/?#&]+-(?P<id>[\da-f]+)'
    _TESTS = [{
        'url': 'https://my.mail.ru/music/songs/%D0%BC8%D0%BB8%D1%82%D1%85-l-a-h-luciferian-aesthetics-of-herrschaft-single-2017-4e31f7125d0dfaef505d947642366893',
        'md5': '0f8c22ef8c5d665b13ac709e63025610',
@ -288,7 +273,7 @@ class MailRuMusicIE(MailRuMusicSearchBaseIE):
 class MailRuMusicSearchIE(MailRuMusicSearchBaseIE):
    IE_NAME = 'mailru:music:search'
    IE_DESC = 'Музыка@Mail.Ru'
-    _VALID_URL = r'https?://my\.mail\.ru/+music/+search/+(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://my\.mail\.ru/music/search/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://my.mail.ru/music/search/black%20shadow',
        'info_dict': {
--- a/youtube_dl/extractor/malltv.py
+++ b/youtube_dl/extractor/malltv.py
@ -8,7 +8,7 @@ from ..utils import merge_dicts
 class MallTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:www|sk)\.)?mall\.tv/(?:[^/]+/)*(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?mall\.tv/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://www.mall.tv/18-miliard-pro-neziskovky-opravdu-jsou-sportovci-nebo-clovek-v-tisni-pijavice',
        'md5': '1c4a37f080e1f3023103a7b43458e518',
@ -26,9 +26,6 @@ class MallTVIE(InfoExtractor):
    }, {
        'url': 'https://www.mall.tv/kdo-to-plati/18-miliard-pro-neziskovky-opravdu-jsou-sportovci-nebo-clovek-v-tisni-pijavice',
        'only_matching': True,
    }, {
        'url': 'https://sk.mall.tv/gejmhaus/reklamacia-nehreje-vyrobnik-tepla-alebo-spekacka',
        'only_matching': True,
    }]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/mediaset.py
+++ b/youtube_dl/extractor/mediaset.py
@ -6,6 +6,7 @@ import re
 from .theplatform import ThePlatformBaseIE
 from ..compat import (
    compat_parse_qs,
    compat_str,
    compat_urllib_parse_urlparse,
 )
 from ..utils import (
@ -113,7 +114,7 @@ class MediasetIE(ThePlatformBaseIE):
                continue
            urlh = ie._request_webpage(
                embed_url, video_id, note='Following embed URL redirect')
-            embed_url = urlh.geturl()
+            embed_url = compat_str(urlh.geturl())
            program_guid = _program_guid(_qs(embed_url))
            if program_guid:
                entries.append(embed_url)
@ -122,7 +123,7 @@ class MediasetIE(ThePlatformBaseIE):
    def _parse_smil_formats(self, smil, smil_url, video_id, namespace=None, f4m_params=None, transform_rtmp_url=None):
        for video in smil.findall(self._xpath_ns('.//video', namespace)):
            video.attrib['src'] = re.sub(r'(https?://vod05)t(-mediaset-it\.akamaized\.net/.+?.mpd)\?.+', r'\1\2', video.attrib['src'])
-        return super(MediasetIE, self)._parse_smil_formats(smil, smil_url, video_id, namespace, f4m_params, transform_rtmp_url)
+        return super()._parse_smil_formats(smil, smil_url, video_id, namespace, f4m_params, transform_rtmp_url)
    def _real_extract(self, url):
        guid = self._match_id(url)
--- a/youtube_dl/extractor/mediasite.py
+++ b/youtube_dl/extractor/mediasite.py
@ -129,7 +129,7 @@ class MediasiteIE(InfoExtractor):
        query = mobj.group('query')
        webpage, urlh = self._download_webpage_handle(url, resource_id)  # XXX: add UrlReferrer?
-        redirect_url = urlh.geturl()
+        redirect_url = compat_str(urlh.geturl())
        # XXX: might have also extracted UrlReferrer and QueryString from the html
        service_path = compat_urlparse.urljoin(redirect_url, self._html_search_regex(
--- a/youtube_dl/extractor/mitele.py
+++ b/youtube_dl/extractor/mitele.py
@ -4,8 +4,8 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    parse_iso8601,
    smuggle_url,
    parse_duration,
 )
@ -18,18 +18,16 @@ class MiTeleIE(InfoExtractor):
        'info_dict': {
            'id': 'FhYW1iNTE6J6H7NkQRIEzfne6t2quqPg',
            'ext': 'mp4',
-            'title': 'Diario de La redacción Programa 144',
+            'title': 'Tor, la web invisible',
-            'description': 'md5:07c35a7b11abb05876a6a79185b58d27',
+            'description': 'md5:3b6fce7eaa41b2d97358726378d9369f',
            'series': 'Diario de',
-            'season': 'Season 14',
+            'season': 'La redacción',
            'season_number': 14,
-            'episode': 'Tor, la web invisible',
+            'season_id': 'diario_de_t14_11981',
            'episode': 'Programa 144',
            'episode_number': 3,
            'thumbnail': r're:(?i)^https?://.*\.jpg$',
            'duration': 2913,
            'age_limit': 16,
            'timestamp': 1471209401,
            'upload_date': '20160814',
        },
        'add_ie': ['Ooyala'],
    }, {
@ -41,15 +39,13 @@ class MiTeleIE(InfoExtractor):
            'title': 'Cuarto Milenio Temporada 6 Programa 226',
            'description': 'md5:5ff132013f0cd968ffbf1f5f3538a65f',
            'series': 'Cuarto Milenio',
-            'season': 'Season 6',
+            'season': 'Temporada 6',
            'season_number': 6,
-            'episode': 'Episode 24',
+            'season_id': 'cuarto_milenio_t06_12715',
            'episode': 'Programa 226',
            'episode_number': 24,
            'thumbnail': r're:(?i)^https?://.*\.jpg$',
            'duration': 7313,
            'age_limit': 12,
            'timestamp': 1471209021,
            'upload_date': '20160814',
        },
        'params': {
            'skip_download': True,
@ -58,36 +54,67 @@ class MiTeleIE(InfoExtractor):
    }, {
        'url': 'http://www.mitele.es/series-online/la-que-se-avecina/57aac5c1c915da951a8b45ed/player',
        'only_matching': True,
    }, {
        'url': 'https://www.mitele.es/programas-tv/diario-de/la-redaccion/programa-144-40_1006364575251/player/',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        display_id = self._match_id(url)
+        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
+
-        pre_player = self._parse_json(self._search_regex(
+        paths = self._download_json(
-            r'window\.\$REACTBASE_STATE\.prePlayer_mtweb\s*=\s*({.+})',
+            'https://www.mitele.es/amd/agp/web/metadata/general_configuration',
-            webpage, 'Pre Player'), display_id)['prePlayer']
+            video_id, 'Downloading paths JSON')
-        title = pre_player['title']
+
-        video = pre_player['video']
+        ooyala_s = paths['general_configuration']['api_configuration']['ooyala_search']
-        video_id = video['dataMediaId']
+        base_url = ooyala_s.get('base_url', 'cdn-search-mediaset.carbyne.ps.ooyala.com')
-        content = pre_player.get('content') or {}
+        full_path = ooyala_s.get('full_path', '/search/v1/full/providers/')
-        info = content.get('info') or {}
+        source = self._download_json(
            '%s://%s%s%s/docs/%s' % (
                ooyala_s.get('protocol', 'https'), base_url, full_path,
                ooyala_s.get('provider_id', '104951'), video_id),
            video_id, 'Downloading data JSON', query={
                'include_titles': 'Series,Season',
                'product_name': ooyala_s.get('product_name', 'test'),
                'format': 'full',
            })['hits']['hits'][0]['_source']
        embedCode = source['offers'][0]['embed_codes'][0]
        titles = source['localizable_titles'][0]
        title = titles.get('title_medium') or titles['title_long']
        description = titles.get('summary_long') or titles.get('summary_medium')
        def get(key1, key2):
            value1 = source.get(key1)
            if not value1 or not isinstance(value1, list):
                return
            if not isinstance(value1[0], dict):
                return
            return value1[0].get(key2)
        series = get('localizable_titles_series', 'title_medium')
        season = get('localizable_titles_season', 'title_medium')
        season_number = int_or_none(source.get('season_number'))
        season_id = source.get('season_id')
        episode = titles.get('title_sort_name')
        episode_number = int_or_none(source.get('episode_number'))
        duration = parse_duration(get('videos', 'duration'))
        return {
            '_type': 'url_transparent',
            # for some reason only HLS is supported
-            'url': smuggle_url('ooyala:' + video_id, {'supportedformats': 'm3u8,dash'}),
+            'url': smuggle_url('ooyala:' + embedCode, {'supportedformats': 'm3u8,dash'}),
            'id': video_id,
            'title': title,
-            'description': info.get('synopsis'),
+            'description': description,
-            'series': content.get('title'),
+            'series': series,
-            'season_number': int_or_none(info.get('season_number')),
+            'season': season,
-            'episode': content.get('subtitle'),
+            'season_number': season_number,
-            'episode_number': int_or_none(info.get('episode_number')),
+            'season_id': season_id,
-            'duration': int_or_none(info.get('duration')),
+            'episode': episode,
-            'thumbnail': video.get('dataPoster'),
+            'episode_number': episode_number,
-            'age_limit': int_or_none(info.get('rating')),
+            'duration': duration,
-            'timestamp': parse_iso8601(pre_player.get('publishedTime')),
+            'thumbnail': get('images', 'url'),
        }
--- a/youtube_dl/extractor/mofosex.py
+++ b/youtube_dl/extractor/mofosex.py
@ -1,8 +1,5 @@
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    str_to_int,
@ -57,23 +54,3 @@ class MofosexIE(KeezMoviesIE):
        })
        return info
 class MofosexEmbedIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?mofosex\.com/embed/?\?.*?\bvideoid=(?P<id>\d+)'
    _TESTS = [{
        'url': 'https://www.mofosex.com/embed/?videoid=318131&referrer=KM',
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return re.findall(
            r'<iframe[^>]+\bsrc=["\']((?:https?:)?//(?:www\.)?mofosex\.com/embed/?\?.*?\bvideoid=\d+)',
            webpage)
    def _real_extract(self, url):
        video_id = self._match_id(url)
        return self.url_result(
            'http://www.mofosex.com/videos/{0}/{0}.html'.format(video_id),
            ie=MofosexIE.ie_key(), video_id=video_id)
--- a/youtube_dl/extractor/motherless.py
+++ b/youtube_dl/extractor/motherless.py
@ -26,7 +26,7 @@ class MotherlessIE(InfoExtractor):
            'categories': ['Gaming', 'anal', 'reluctant', 'rough', 'Wife'],
            'upload_date': '20100913',
            'uploader_id': 'famouslyfuckedup',
-            'thumbnail': r're:https?://.*\.jpg',
+            'thumbnail': r're:http://.*\.jpg',
            'age_limit': 18,
        }
    }, {
@ -40,7 +40,7 @@ class MotherlessIE(InfoExtractor):
                           'game', 'hairy'],
            'upload_date': '20140622',
            'uploader_id': 'Sulivana7x',
-            'thumbnail': r're:https?://.*\.jpg',
+            'thumbnail': r're:http://.*\.jpg',
            'age_limit': 18,
        },
        'skip': '404',
@ -54,7 +54,7 @@ class MotherlessIE(InfoExtractor):
            'categories': ['superheroine heroine  superher'],
            'upload_date': '20140827',
            'uploader_id': 'shade0230',
-            'thumbnail': r're:https?://.*\.jpg',
+            'thumbnail': r're:http://.*\.jpg',
            'age_limit': 18,
        }
    }, {
@ -76,8 +76,7 @@ class MotherlessIE(InfoExtractor):
            raise ExtractorError('Video %s is for friends only' % video_id, expected=True)
        title = self._html_search_regex(
-            (r'(?s)<div[^>]+\bclass=["\']media-meta-title[^>]+>(.+?)</div>',
+            r'id="view-upload-title">\s+([^<]+)<', webpage, 'title')
             r'id="view-upload-title">\s+([^<]+)<'), webpage, 'title')
        video_url = (self._html_search_regex(
            (r'setup\(\{\s*["\']file["\']\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1',
             r'fileurl\s*=\s*(["\'])(?P<url>(?:(?!\1).)+)\1'),
@ -85,15 +84,14 @@ class MotherlessIE(InfoExtractor):
            or 'http://cdn4.videos.motherlessmedia.com/videos/%s.mp4?fs=opencloud' % video_id)
        age_limit = self._rta_search(webpage)
        view_count = str_to_int(self._html_search_regex(
-            (r'>(\d+)\s+Views<', r'<strong>Views</strong>\s+([^<]+)<'),
+            r'<strong>Views</strong>\s+([^<]+)<',
            webpage, 'view count', fatal=False))
        like_count = str_to_int(self._html_search_regex(
-            (r'>(\d+)\s+Favorites<', r'<strong>Favorited</strong>\s+([^<]+)<'),
+            r'<strong>Favorited</strong>\s+([^<]+)<',
            webpage, 'like count', fatal=False))
        upload_date = self._html_search_regex(
-            (r'class=["\']count[^>]+>(\d+\s+[a-zA-Z]{3}\s+\d{4})<',
+            r'<strong>Uploaded</strong>\s+([^<]+)<', webpage, 'upload date')
             r'<strong>Uploaded</strong>\s+([^<]+)<'), webpage, 'upload date')
        if 'Ago' in upload_date:
            days = int(re.search(r'([0-9]+)', upload_date).group(1))
            upload_date = (datetime.datetime.now() - datetime.timedelta(days=days)).strftime('%Y%m%d')
--- a/youtube_dl/extractor/msn.py
+++ b/youtube_dl/extractor/msn.py
@ -14,27 +14,20 @@ from ..utils import (
 class MSNIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:www|preview)\.)?msn\.com/(?:[^/]+/)+(?P<display_id>[^/]+)/[a-z]{2}-(?P<id>[\da-zA-Z]+)'
+    _VALID_URL = r'https?://(?:www\.)?msn\.com/(?:[^/]+/)+(?P<display_id>[^/]+)/[a-z]{2}-(?P<id>[\da-zA-Z]+)'
    _TESTS = [{
-        'url': 'https://www.msn.com/en-in/money/video/7-ways-to-get-rid-of-chest-congestion/vi-BBPxU6d',
+        'url': 'http://www.msn.com/en-ae/foodanddrink/joinourtable/criminal-minds-shemar-moore-shares-a-touching-goodbye-message/vp-BBqQYNE',
-        'md5': '087548191d273c5c55d05028f8d2cbcd',
+        'md5': '8442f66c116cbab1ff7098f986983458',
        'info_dict': {
-            'id': 'BBPxU6d',
+            'id': 'BBqQYNE',
-            'display_id': '7-ways-to-get-rid-of-chest-congestion',
+            'display_id': 'criminal-minds-shemar-moore-shares-a-touching-goodbye-message',
            'ext': 'mp4',
-            'title': 'Seven ways to get rid of chest congestion',
+            'title': 'Criminal Minds - Shemar Moore Shares A Touching Goodbye Message',
-            'description': '7 Ways to Get Rid of Chest Congestion',
+            'description': 'md5:e8e89b897b222eb33a6b5067a8f1bc25',
-            'duration': 88,
+            'duration': 104,
-            'uploader': 'Health',
+            'uploader': 'CBS Entertainment',
-            'uploader_id': 'BBPrMqa',
+            'uploader_id': 'IT0X5aoJ6bJgYerJXSDCgFmYPB1__54v',
        },
    }, {
        # Article, multiple Dailymotion Embeds
        'url': 'https://www.msn.com/en-in/money/sports/hottest-football-wags-greatest-footballers-turned-managers-and-more/ar-BBpc7Nl',
        'info_dict': {
            'id': 'BBpc7Nl',
        },
        'playlist_mincount': 4,
    }, {
        'url': 'http://www.msn.com/en-ae/news/offbeat/meet-the-nine-year-old-self-made-millionaire/ar-BBt6ZKf',
        'only_matching': True,
@ -50,122 +43,93 @@ class MSNIE(InfoExtractor):
        'only_matching': True,
    }, {
        # Vidible(AOL) Embed
-        'url': 'https://www.msn.com/en-us/money/other/jupiter-is-about-to-come-so-close-you-can-see-its-moons-with-binoculars/vi-AACqsHR',
+        'url': 'https://www.msn.com/en-us/video/animals/yellowstone-park-staffers-catch-deer-engaged-in-behavior-they-cant-explain/vi-AAGfdg1',
        'only_matching': True,
    }, {
        # Dailymotion Embed
        'url': 'https://www.msn.com/es-ve/entretenimiento/watch/winston-salem-paire-refait-des-siennes-en-perdant-sa-raquette-au-service/vp-AAG704L',
        'only_matching': True,
    }, {
        # YouTube Embed
        'url': 'https://www.msn.com/en-in/money/news/meet-vikram-%E2%80%94-chandrayaan-2s-lander/vi-AAGUr0v',
        'only_matching': True,
    }, {
        # NBCSports Embed
        'url': 'https://www.msn.com/en-us/money/football_nfl/week-13-preview-redskins-vs-panthers/vi-BBXsCDb',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        display_id, page_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
        video_id, display_id = mobj.group('id', 'display_id')
        webpage = self._download_webpage(url, display_id)
-        entries = []
+        video = self._parse_json(
-        for _, metadata in re.findall(r'data-metadata\s*=\s*(["\'])(?P<data>.+?)\1', webpage):
+            self._search_regex(
-            video = self._parse_json(unescapeHTML(metadata), display_id)
+                r'data-metadata\s*=\s*(["\'])(?P<data>.+?)\1',
                webpage, 'video data', default='{}', group='data'),
            display_id, transform_source=unescapeHTML)
-            provider_id = video.get('providerId')
+        if not video:
            player_name = video.get('playerName')
            if player_name and provider_id:
                entry = None
                if player_name == 'AOL':
                    if provider_id.startswith('http'):
                        provider_id = self._search_regex(
                            r'https?://delivery\.vidible\.tv/video/redirect/([0-9a-f]{24})',
                            provider_id, 'vidible id')
                    entry = self.url_result(
                        'aol-video:' + provider_id, 'Aol', provider_id)
                elif player_name == 'Dailymotion':
                    entry = self.url_result(
                        'https://www.dailymotion.com/video/' + provider_id,
                        'Dailymotion', provider_id)
                elif player_name == 'YouTube':
                    entry = self.url_result(
                        provider_id, 'Youtube', provider_id)
                elif player_name == 'NBCSports':
                    entry = self.url_result(
                        'http://vplayer.nbcsports.com/p/BxmELC/nbcsports_embed/select/media/' + provider_id,
                        'NBCSportsVPlayer', provider_id)
                if entry:
                    entries.append(entry)
                    continue
            video_id = video['uuid']
            title = video['title']
            formats = []
            for file_ in video.get('videoFiles', []):
                format_url = file_.get('url')
                if not format_url:
                    continue
                if 'format=m3u8-aapl' in format_url:
                    # m3u8_native should not be used here until
                    # https://github.com/ytdl-org/youtube-dl/issues/9913 is fixed
                    formats.extend(self._extract_m3u8_formats(
                        format_url, display_id, 'mp4',
                        m3u8_id='hls', fatal=False))
                elif 'format=mpd-time-csf' in format_url:
                    formats.extend(self._extract_mpd_formats(
                        format_url, display_id, 'dash', fatal=False))
                elif '.ism' in format_url:
                    if format_url.endswith('.ism'):
                        format_url += '/manifest'
                    formats.extend(self._extract_ism_formats(
                        format_url, display_id, 'mss', fatal=False))
                else:
                    format_id = file_.get('formatCode')
                    formats.append({
                        'url': format_url,
                        'ext': 'mp4',
                        'format_id': format_id,
                        'width': int_or_none(file_.get('width')),
                        'height': int_or_none(file_.get('height')),
                        'vbr': int_or_none(self._search_regex(r'_(\d+)\.mp4', format_url, 'vbr', default=None)),
                        'preference': 1 if format_id == '1001' else None,
                    })
            self._sort_formats(formats)
            subtitles = {}
            for file_ in video.get('files', []):
                format_url = file_.get('url')
                format_code = file_.get('formatCode')
                if not format_url or not format_code:
                    continue
                if compat_str(format_code) == '3100':
                    subtitles.setdefault(file_.get('culture', 'en'), []).append({
                        'ext': determine_ext(format_url, 'ttml'),
                        'url': format_url,
                    })
            entries.append({
                'id': video_id,
                'display_id': display_id,
                'title': title,
                'description': video.get('description'),
                'thumbnail': video.get('headlineImage', {}).get('url'),
                'duration': int_or_none(video.get('durationSecs')),
                'uploader': video.get('sourceFriendly'),
                'uploader_id': video.get('providerId'),
                'creator': video.get('creator'),
                'subtitles': subtitles,
                'formats': formats,
            })
        if not entries:
            error = unescapeHTML(self._search_regex(
                r'data-error=(["\'])(?P<error>.+?)\1',
                webpage, 'error', group='error'))
            raise ExtractorError('%s said: %s' % (self.IE_NAME, error), expected=True)
-        return self.playlist_result(entries, page_id)
+        player_name = video.get('playerName')
        if player_name:
            provider_id = video.get('providerId')
            if provider_id:
                if player_name == 'AOL':
                    return self.url_result(
                        'aol-video:' + provider_id, 'Aol', provider_id)
                elif player_name == 'Dailymotion':
                    return self.url_result(
                        'https://www.dailymotion.com/video/' + provider_id,
                        'Dailymotion', provider_id)
        title = video['title']
        formats = []
        for file_ in video.get('videoFiles', []):
            format_url = file_.get('url')
            if not format_url:
                continue
            if 'm3u8' in format_url:
                # m3u8_native should not be used here until
                # https://github.com/ytdl-org/youtube-dl/issues/9913 is fixed
                m3u8_formats = self._extract_m3u8_formats(
                    format_url, display_id, 'mp4',
                    m3u8_id='hls', fatal=False)
                formats.extend(m3u8_formats)
            elif determine_ext(format_url) == 'ism':
                formats.extend(self._extract_ism_formats(
                    format_url + '/Manifest', display_id, 'mss', fatal=False))
            else:
                formats.append({
                    'url': format_url,
                    'ext': 'mp4',
                    'format_id': 'http',
                    'width': int_or_none(file_.get('width')),
                    'height': int_or_none(file_.get('height')),
                })
        self._sort_formats(formats)
        subtitles = {}
        for file_ in video.get('files', []):
            format_url = file_.get('url')
            format_code = file_.get('formatCode')
            if not format_url or not format_code:
                continue
            if compat_str(format_code) == '3100':
                subtitles.setdefault(file_.get('culture', 'en'), []).append({
                    'ext': determine_ext(format_url, 'ttml'),
                    'url': format_url,
                })
        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
            'description': video.get('description'),
            'thumbnail': video.get('headlineImage', {}).get('url'),
            'duration': int_or_none(video.get('durationSecs')),
            'uploader': video.get('sourceFriendly'),
            'uploader_id': video.get('providerId'),
            'creator': video.get('creator'),
            'subtitles': subtitles,
            'formats': formats,
        }
--- a/youtube_dl/extractor/musicplayon.py
+++ b/youtube_dl/extractor/musicplayon.py
@ -0,0 +1,66 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..compat import compat_urlparse
 from ..utils import (
    int_or_none,
    js_to_json,
    mimetype2ext,
 )
 class MusicPlayOnIE(InfoExtractor):
    _VALID_URL = r'https?://(?:.+?\.)?musicplayon\.com/play(?:-touch)?\?(?:v|pl=\d+&play)=(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://en.musicplayon.com/play?v=433377',
        'md5': '00cdcdea1726abdf500d1e7fd6dd59bb',
        'info_dict': {
            'id': '433377',
            'ext': 'mp4',
            'title': 'Rick Ross - Interview On Chelsea Lately (2014)',
            'description': 'Rick Ross Interview On Chelsea Lately',
            'duration': 342,
            'uploader': 'ultrafish',
        },
    }, {
        'url': 'http://en.musicplayon.com/play?pl=102&play=442629',
        'only_matching': True,
    }]
    _URL_TEMPLATE = 'http://en.musicplayon.com/play?v=%s'
    def _real_extract(self, url):
        video_id = self._match_id(url)
        url = self._URL_TEMPLATE % video_id
        page = self._download_webpage(url, video_id)
        title = self._og_search_title(page)
        description = self._og_search_description(page)
        thumbnail = self._og_search_thumbnail(page)
        duration = self._html_search_meta('video:duration', page, 'duration', fatal=False)
        view_count = self._og_search_property('count', page, fatal=False)
        uploader = self._html_search_regex(
            r'<div>by&nbsp;<a href="[^"]+" class="purple">([^<]+)</a></div>', page, 'uploader', fatal=False)
        sources = self._parse_json(
            self._search_regex(r'setup\[\'_sources\'\]\s*=\s*([^;]+);', page, 'video sources'),
            video_id, transform_source=js_to_json)
        formats = [{
            'url': compat_urlparse.urljoin(url, source['src']),
            'ext': mimetype2ext(source.get('type')),
            'format_note': source.get('data-res'),
        } for source in sources]
        return {
            'id': video_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'uploader': uploader,
            'duration': int_or_none(duration),
            'view_count': int_or_none(view_count),
            'formats': formats,
        }
--- a/youtube_dl/extractor/naver.py
+++ b/youtube_dl/extractor/naver.py
@ -1,33 +1,68 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    clean_html,
    dict_get,
    ExtractorError,
    int_or_none,
    parse_duration,
    try_get,
    update_url_query,
 )
-class NaverBaseIE(InfoExtractor):
+class NaverIE(InfoExtractor):
-    _CAPTION_EXT_RE = r'\.(?:ttml|vtt)'
+    _VALID_URL = r'https?://(?:m\.)?tv(?:cast)?\.naver\.com/v/(?P<id>\d+)'
-    def _extract_video_info(self, video_id, vid, key):
+    _TESTS = [{
        'url': 'http://tv.naver.com/v/81652',
        'info_dict': {
            'id': '81652',
            'ext': 'mp4',
            'title': '[9월 모의고사 해설강의][수학_김상희] 수학 A형 16~20번',
            'description': '합격불변의 법칙 메가스터디 | 메가스터디 수학 김상희 선생님이 9월 모의고사 수학A형 16번에서 20번까지 해설강의를 공개합니다.',
            'upload_date': '20130903',
        },
    }, {
        'url': 'http://tv.naver.com/v/395837',
        'md5': '638ed4c12012c458fefcddfd01f173cd',
        'info_dict': {
            'id': '395837',
            'ext': 'mp4',
            'title': '9년이 지나도 아픈 기억, 전효성의 아버지',
            'description': 'md5:5bf200dcbf4b66eb1b350d1eb9c753f7',
            'upload_date': '20150519',
        },
        'skip': 'Georestricted',
    }, {
        'url': 'http://tvcast.naver.com/v/81652',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        vid = self._search_regex(
            r'videoId["\']\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1', webpage,
            'video id', fatal=None, group='value')
        in_key = self._search_regex(
            r'inKey["\']\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1', webpage,
            'key', default=None, group='value')
        if not vid or not in_key:
            error = self._html_search_regex(
                r'(?s)<div class="(?:nation_error|nation_box|error_box)">\s*(?:<!--.*?-->)?\s*<p class="[^"]+">(?P<msg>.+?)</p>\s*</div>',
                webpage, 'error', default=None)
            if error:
                raise ExtractorError(error, expected=True)
            raise ExtractorError('couldn\'t extract vid and key')
        video_data = self._download_json(
            'http://play.rmcnmv.naver.com/vod/play/v2.0/' + vid,
            video_id, query={
-                'key': key,
+                'key': in_key,
            })
        meta = video_data['meta']
        title = meta['subject']
        formats = []
        get_list = lambda x: try_get(video_data, lambda y: y[x + 's']['list'], list) or []
        def extract_formats(streams, stream_type, query={}):
            for stream in streams:
@ -38,7 +73,7 @@ class NaverBaseIE(InfoExtractor):
                encoding_option = stream.get('encodingOption', {})
                bitrate = stream.get('bitrate', {})
                formats.append({
-                    'format_id': '%s_%s' % (stream.get('type') or stream_type, dict_get(encoding_option, ('name', 'id'))),
+                    'format_id': '%s_%s' % (stream.get('type') or stream_type, encoding_option.get('id') or encoding_option.get('name')),
                    'url': stream_url,
                    'width': int_or_none(encoding_option.get('width')),
                    'height': int_or_none(encoding_option.get('height')),
@ -48,7 +83,7 @@ class NaverBaseIE(InfoExtractor):
                    'protocol': 'm3u8_native' if stream_type == 'HLS' else None,
                })
-        extract_formats(get_list('video'), 'H264')
+        extract_formats(video_data.get('videos', {}).get('list', []), 'H264')
        for stream_set in video_data.get('streams', []):
            query = {}
            for param in stream_set.get('keys', []):
@ -66,101 +101,28 @@ class NaverBaseIE(InfoExtractor):
                    'mp4', 'm3u8_native', m3u8_id=stream_type, fatal=False))
        self._sort_formats(formats)
        replace_ext = lambda x, y: re.sub(self._CAPTION_EXT_RE, '.' + y, x)
        def get_subs(caption_url):
            if re.search(self._CAPTION_EXT_RE, caption_url):
                return [{
                    'url': replace_ext(caption_url, 'ttml'),
                }, {
                    'url': replace_ext(caption_url, 'vtt'),
                }]
            else:
                return [{'url': caption_url}]
        automatic_captions = {}
        subtitles = {}
-        for caption in get_list('caption'):
+        for caption in video_data.get('captions', {}).get('list', []):
            caption_url = caption.get('source')
            if not caption_url:
                continue
-            sub_dict = automatic_captions if caption.get('type') == 'auto' else subtitles
+            subtitles.setdefault(caption.get('language') or caption.get('locale'), []).append({
-            sub_dict.setdefault(dict_get(caption, ('locale', 'language')), []).extend(get_subs(caption_url))
+                'url': caption_url,
            })
-        user = meta.get('user', {})
+        upload_date = self._search_regex(
            r'<span[^>]+class="date".*?(\d{4}\.\d{2}\.\d{2})',
            webpage, 'upload date', fatal=False)
        if upload_date:
            upload_date = upload_date.replace('.', '')
        return {
            'id': video_id,
            'title': title,
            'formats': formats,
            'subtitles': subtitles,
-            'automatic_captions': automatic_captions,
+            'description': self._og_search_description(webpage),
-            'thumbnail': try_get(meta, lambda x: x['cover']['source']),
+            'thumbnail': meta.get('cover', {}).get('source') or self._og_search_thumbnail(webpage),
            'view_count': int_or_none(meta.get('count')),
-            'uploader_id': user.get('id'),
+            'upload_date': upload_date,
            'uploader': user.get('name'),
            'uploader_url': user.get('url'),
        }
 class NaverIE(NaverBaseIE):
    _VALID_URL = r'https?://(?:m\.)?tv(?:cast)?\.naver\.com/(?:v|embed)/(?P<id>\d+)'
    _GEO_BYPASS = False
    _TESTS = [{
        'url': 'http://tv.naver.com/v/81652',
        'info_dict': {
            'id': '81652',
            'ext': 'mp4',
            'title': '[9월 모의고사 해설강의][수학_김상희] 수학 A형 16~20번',
            'description': '메가스터디 수학 김상희 선생님이 9월 모의고사 수학A형 16번에서 20번까지 해설강의를 공개합니다.',
            'timestamp': 1378200754,
            'upload_date': '20130903',
            'uploader': '메가스터디, 합격불변의 법칙',
            'uploader_id': 'megastudy',
        },
    }, {
        'url': 'http://tv.naver.com/v/395837',
        'md5': '8a38e35354d26a17f73f4e90094febd3',
        'info_dict': {
            'id': '395837',
            'ext': 'mp4',
            'title': '9년이 지나도 아픈 기억, 전효성의 아버지',
            'description': 'md5:eb6aca9d457b922e43860a2a2b1984d3',
            'timestamp': 1432030253,
            'upload_date': '20150519',
            'uploader': '4가지쇼 시즌2',
            'uploader_id': 'wrappinguser29',
        },
        'skip': 'Georestricted',
    }, {
        'url': 'http://tvcast.naver.com/v/81652',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        content = self._download_json(
            'https://tv.naver.com/api/json/v/' + video_id,
            video_id, headers=self.geo_verification_headers())
        player_info_json = content.get('playerInfoJson') or {}
        current_clip = player_info_json.get('currentClip') or {}
        vid = current_clip.get('videoId')
        in_key = current_clip.get('inKey')
        if not vid or not in_key:
            player_auth = try_get(player_info_json, lambda x: x['playerOption']['auth'])
            if player_auth == 'notCountry':
                self.raise_geo_restricted(countries=['KR'])
            elif player_auth == 'notLogin':
                self.raise_login_required()
            raise ExtractorError('couldn\'t extract vid and key')
        info = self._extract_video_info(video_id, vid, in_key)
        info.update({
            'description': clean_html(current_clip.get('description')),
            'timestamp': int_or_none(current_clip.get('firstExposureTime'), 1000),
            'duration': parse_duration(current_clip.get('displayPlayTime')),
            'like_count': int_or_none(current_clip.get('recommendPoint')),
            'age_limit': 19 if current_clip.get('adult') else None,
        })
        return info
--- a/youtube_dl/extractor/nbc.py
+++ b/youtube_dl/extractor/nbc.py
@ -87,25 +87,11 @@ class NBCIE(AdobePassIE):
    def _real_extract(self, url):
        permalink, video_id = re.match(self._VALID_URL, url).groups()
        permalink = 'http' + compat_urllib_parse_unquote(permalink)
-        video_data = self._download_json(
+        response = self._download_json(
            'https://friendship.nbc.co/v2/graphql', video_id, query={
-                'query': '''query bonanzaPage(
+                'query': '''{
-  $app: NBCUBrands! = nbc
+  page(name: "%s", platform: web, type: VIDEO, userId: "0") {
-  $name: String!
+    data {
  $oneApp: Boolean
  $platform: SupportedPlatforms! = web
  $type: EntityPageType! = VIDEO
  $userId: String!
 ) {
  bonanzaPage(
    app: $app
    name: $name
    oneApp: $oneApp
    platform: $platform
    type: $type
    userId: $userId
  ) {
    metadata {
      ... on VideoPageData {
        description
        episodeNumber
@ -114,20 +100,15 @@ class NBCIE(AdobePassIE):
        mpxAccountId
        mpxGuid
        rating
        resourceId
        seasonNumber
        secondaryTitle
        seriesShortTitle
      }
    }
  }
-}''',
+}''' % permalink,
-                'variables': json.dumps({
+            })
-                    'name': permalink,
+        video_data = response['data']['page']['data']
                    'oneApp': True,
                    'userId': '0',
                }),
            })['data']['bonanzaPage']['metadata']
        query = {
            'mbr': 'true',
            'manifest': 'm3u',
@ -136,8 +117,8 @@ class NBCIE(AdobePassIE):
        title = video_data['secondaryTitle']
        if video_data.get('locked'):
            resource = self._get_mvpd_resource(
-                video_data.get('resourceId') or 'nbcentertainment',
+                'nbcentertainment', title, video_id,
-                title, video_id, video_data.get('rating'))
+                video_data.get('rating'))
            query['auth'] = self._extract_mvpd_auth(
                url, video_id, 'nbcentertainment', resource)
        theplatform_url = smuggle_url(update_url_query(
--- a/youtube_dl/extractor/ndr.py
+++ b/youtube_dl/extractor/ndr.py
@ -7,11 +7,8 @@ from .common import InfoExtractor
 from ..utils import (
    determine_ext,
    int_or_none,
    merge_dicts,
    parse_iso8601,
    qualities,
    try_get,
    urljoin,
 )
@ -88,25 +85,21 @@ class NDRIE(NDRBaseIE):
    def _extract_embed(self, webpage, display_id):
        embed_url = self._html_search_meta(
-            'embedURL', webpage, 'embed URL',
+            'embedURL', webpage, 'embed URL', fatal=True)
            default=None) or self._search_regex(
            r'\bembedUrl["\']\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage,
            'embed URL', group='url')
        description = self._search_regex(
            r'<p[^>]+itemprop="description">([^<]+)</p>',
            webpage, 'description', default=None) or self._og_search_description(webpage)
        timestamp = parse_iso8601(
            self._search_regex(
                r'<span[^>]+itemprop="(?:datePublished|uploadDate)"[^>]+content="([^"]+)"',
-                webpage, 'upload date', default=None))
+                webpage, 'upload date', fatal=False))
-        info = self._search_json_ld(webpage, display_id, default={})
+        return {
        return merge_dicts({
            '_type': 'url_transparent',
            'url': embed_url,
            'display_id': display_id,
            'description': description,
            'timestamp': timestamp,
-        }, info)
+        }
 class NJoyIE(NDRBaseIE):
@ -227,17 +220,11 @@ class NDREmbedBaseIE(InfoExtractor):
        upload_date = ppjson.get('config', {}).get('publicationDate')
        duration = int_or_none(config.get('duration'))
-        thumbnails = []
+        thumbnails = [{
-        poster = try_get(config, lambda x: x['poster'], dict) or {}
+            'id': thumbnail.get('quality') or thumbnail_id,
-        for thumbnail_id, thumbnail in poster.items():
+            'url': thumbnail['src'],
-            thumbnail_url = urljoin(url, thumbnail.get('src'))
+            'preference': quality_key(thumbnail.get('quality')),
-            if not thumbnail_url:
+        } for thumbnail_id, thumbnail in config.get('poster', {}).items() if thumbnail.get('src')]
                continue
            thumbnails.append({
                'id': thumbnail.get('quality') or thumbnail_id,
                'url': thumbnail_url,
                'preference': quality_key(thumbnail.get('quality')),
            })
        return {
            'id': video_id,
--- a/youtube_dl/extractor/nhk.py
+++ b/youtube_dl/extractor/nhk.py
@ -6,7 +6,7 @@ from .common import InfoExtractor
 class NhkVodIE(InfoExtractor):
-    _VALID_URL = r'https?://www3\.nhk\.or\.jp/nhkworld/(?P<lang>[a-z]{2})/ondemand/(?P<type>video|audio)/(?P<id>\d{7}|[^/]+?-\d{8}-\d+)'
+    _VALID_URL = r'https?://www3\.nhk\.or\.jp/nhkworld/(?P<lang>[a-z]{2})/ondemand/(?P<type>video|audio)/(?P<id>\d{7}|[a-z]+-\d{8}-\d+)'
    # Content available only for a limited period of time. Visit
    # https://www3.nhk.or.jp/nhkworld/en/ondemand/ for working samples.
    _TESTS = [{
@ -30,11 +30,8 @@ class NhkVodIE(InfoExtractor):
    }, {
        'url': 'https://www3.nhk.or.jp/nhkworld/fr/ondemand/audio/plugin-20190404-1/',
        'only_matching': True,
    }, {
        'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/audio/j_art-20150903-1/',
        'only_matching': True,
    }]
-    _API_URL_TEMPLATE = 'https://api.nhk.or.jp/nhkworld/%sod%slist/v7a/episode/%s/%s/all%s.json'
+    _API_URL_TEMPLATE = 'https://api.nhk.or.jp/nhkworld/%sod%slist/v7/episode/%s/%s/all%s.json'
    def _real_extract(self, url):
        lang, m_type, episode_id = re.match(self._VALID_URL, url).groups()
@ -85,9 +82,15 @@ class NhkVodIE(InfoExtractor):
            audio = episode['audio']
            audio_path = audio['audio']
            info['formats'] = self._extract_m3u8_formats(
-                'https://nhkworld-vh.akamaihd.net/i%s/master.m3u8' % audio_path,
+                'https://nhks-vh.akamaihd.net/i%s/master.m3u8' % audio_path,
-                episode_id, 'm4a', entry_protocol='m3u8_native',
+                episode_id, 'm4a', m3u8_id='hls', fatal=False)
-                m3u8_id='hls', fatal=False)
+            for proto in ('rtmpt', 'rtmp'):
                info['formats'].append({
                    'ext': 'flv',
                    'format_id': proto,
                    'url': '%s://flv.nhk.or.jp/ondemand/mp4:flv%s' % (proto, audio_path),
                    'vcodec': 'none',
                })
            for f in info['formats']:
                f['language'] = lang
        return info
--- a/youtube_dl/extractor/nintendo.py
+++ b/youtube_dl/extractor/nintendo.py
@ -5,12 +5,13 @@ import re
 from .common import InfoExtractor
 from .ooyala import OoyalaIE
 from ..utils import unescapeHTML
 class NintendoIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?nintendo\.com/(?:games/detail|nintendo-direct)/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?nintendo\.com/games/detail/(?P<id>[^/?#&]+)'
    _TESTS = [{
-        'url': 'https://www.nintendo.com/games/detail/duck-hunt-wii-u/',
+        'url': 'http://www.nintendo.com/games/detail/yEiAzhU2eQI1KZ7wOHhngFoAHc1FpHwj',
        'info_dict': {
            'id': 'MzMmticjp0VPzO3CCj4rmFOuohEuEWoW',
            'ext': 'flv',
@ -27,19 +28,7 @@ class NintendoIE(InfoExtractor):
            'id': 'tokyo-mirage-sessions-fe-wii-u',
            'title': 'Tokyo Mirage Sessions ♯FE',
        },
-        'playlist_count': 4,
+        'playlist_count': 3,
    }, {
        'url': 'https://www.nintendo.com/nintendo-direct/09-04-2019/',
        'info_dict': {
            'id': 'J2bXdmaTE6fe3dWJTPcc7m23FNbc_A1V',
            'ext': 'mp4',
            'title': 'Switch_ROS_ND0904-H264.mov',
            'duration': 2324.758,
        },
        'params': {
            'skip_download': True,
        },
        'add_ie': ['Ooyala'],
    }]
    def _real_extract(self, url):
@ -50,11 +39,8 @@ class NintendoIE(InfoExtractor):
        entries = [
            OoyalaIE._build_url_result(m.group('code'))
            for m in re.finditer(
-                r'data-(?:video-id|directVideoId)=(["\'])(?P<code>(?:(?!\1).)+)\1', webpage)]
+                r'class=(["\'])embed-video\1[^>]+data-video-code=(["\'])(?P<code>(?:(?!\2).)+)\2',
-
+                webpage)]
        title = self._html_search_regex(
            r'(?s)<(?:span|div)[^>]+class="(?:title|wrapper)"[^>]*>.*?<h1>(.+?)</h1>',
            webpage, 'title', fatal=False)
        return self.playlist_result(
-            entries, page_id, title)
+            entries, page_id, unescapeHTML(self._og_search_title(webpage, fatal=False)))
--- a/youtube_dl/extractor/nova.py
+++ b/youtube_dl/extractor/nova.py
@ -6,7 +6,6 @@ import re
 from .common import InfoExtractor
 from ..utils import (
    clean_html,
    determine_ext,
    int_or_none,
    js_to_json,
    qualities,
@ -19,7 +18,7 @@ class NovaEmbedIE(InfoExtractor):
    _VALID_URL = r'https?://media\.cms\.nova\.cz/embed/(?P<id>[^/?#&]+)'
    _TEST = {
        'url': 'https://media.cms.nova.cz/embed/8o0n0r?autoplay=1',
-        'md5': 'ee009bafcc794541570edd44b71cbea3',
+        'md5': 'b3834f6de5401baabf31ed57456463f7',
        'info_dict': {
            'id': '8o0n0r',
            'ext': 'mp4',
@ -34,76 +33,36 @@ class NovaEmbedIE(InfoExtractor):
        webpage = self._download_webpage(url, video_id)
-        duration = None
+        bitrates = self._parse_json(
        formats = []
        player = self._parse_json(
            self._search_regex(
-                r'Player\.init\s*\([^,]+,\s*({.+?})\s*,\s*{.+?}\s*\)\s*;',
+                r'(?s)(?:src|bitrates)\s*=\s*({.+?})\s*;', webpage, 'formats'),
-                webpage, 'player', default='{}'), video_id, fatal=False)
+            video_id, transform_source=js_to_json)
-        if player:
+
-            for format_id, format_list in player['tracks'].items():
+        QUALITIES = ('lq', 'mq', 'hq', 'hd')
-                if not isinstance(format_list, list):
+        quality_key = qualities(QUALITIES)
-                    format_list = [format_list]
+
-                for format_dict in format_list:
+        formats = []
-                    if not isinstance(format_dict, dict):
+        for format_id, format_list in bitrates.items():
-                        continue
+            if not isinstance(format_list, list):
-                    format_url = url_or_none(format_dict.get('src'))
+                continue
-                    format_type = format_dict.get('type')
+            for format_url in format_list:
-                    ext = determine_ext(format_url)
+                format_url = url_or_none(format_url)
-                    if (format_type == 'application/x-mpegURL'
+                if not format_url:
-                            or format_id == 'HLS' or ext == 'm3u8'):
+                    continue
-                        formats.extend(self._extract_m3u8_formats(
+                f = {
-                            format_url, video_id, 'mp4',
+                    'url': format_url,
-                            entry_protocol='m3u8_native', m3u8_id='hls',
+                }
-                            fatal=False))
+                f_id = format_id
-                    elif (format_type == 'application/dash+xml'
+                for quality in QUALITIES:
-                          or format_id == 'DASH' or ext == 'mpd'):
+                    if '%s.mp4' % quality in format_url:
-                        formats.extend(self._extract_mpd_formats(
+                        f_id += '-%s' % quality
-                            format_url, video_id, mpd_id='dash', fatal=False))
+                        f.update({
-                    else:
+                            'quality': quality_key(quality),
-                        formats.append({
+                            'format_note': quality.upper(),
                            'url': format_url,
                        })
-            duration = int_or_none(player.get('duration'))
+                        break
-        else:
+                f['format_id'] = f_id
-            # Old path, not actual as of 08.04.2020
+                formats.append(f)
            bitrates = self._parse_json(
                self._search_regex(
                    r'(?s)(?:src|bitrates)\s*=\s*({.+?})\s*;', webpage, 'formats'),
                video_id, transform_source=js_to_json)
            QUALITIES = ('lq', 'mq', 'hq', 'hd')
            quality_key = qualities(QUALITIES)
            for format_id, format_list in bitrates.items():
                if not isinstance(format_list, list):
                    format_list = [format_list]
                for format_url in format_list:
                    format_url = url_or_none(format_url)
                    if not format_url:
                        continue
                    if format_id == 'hls':
                        formats.extend(self._extract_m3u8_formats(
                            format_url, video_id, ext='mp4',
                            entry_protocol='m3u8_native', m3u8_id='hls',
                            fatal=False))
                        continue
                    f = {
                        'url': format_url,
                    }
                    f_id = format_id
                    for quality in QUALITIES:
                        if '%s.mp4' % quality in format_url:
                            f_id += '-%s' % quality
                            f.update({
                                'quality': quality_key(quality),
                                'format_note': quality.upper(),
                            })
                            break
                    f['format_id'] = f_id
                    formats.append(f)
        self._sort_formats(formats)
        title = self._og_search_title(
@ -116,8 +75,7 @@ class NovaEmbedIE(InfoExtractor):
            r'poster\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1', webpage,
            'thumbnail', fatal=False, group='value')
        duration = int_or_none(self._search_regex(
-            r'videoDuration\s*:\s*(\d+)', webpage, 'duration',
+            r'videoDuration\s*:\s*(\d+)', webpage, 'duration', fatal=False))
            default=duration))
        return {
            'id': video_id,
@ -133,7 +91,7 @@ class NovaIE(InfoExtractor):
    _VALID_URL = r'https?://(?:[^.]+\.)?(?P<site>tv(?:noviny)?|tn|novaplus|vymena|fanda|krasna|doma|prask)\.nova\.cz/(?:[^/]+/)+(?P<id>[^/]+?)(?:\.html|/|$)'
    _TESTS = [{
        'url': 'http://tn.nova.cz/clanek/tajemstvi-ukryte-v-podzemi-specialni-nemocnice-v-prazske-krci.html#player_13260',
-        'md5': '249baab7d0104e186e78b0899c7d5f28',
+        'md5': '1dd7b9d5ea27bc361f110cd855a19bd3',
        'info_dict': {
            'id': '1757139',
            'display_id': 'tajemstvi-ukryte-v-podzemi-specialni-nemocnice-v-prazske-krci',
@ -155,8 +113,7 @@ class NovaIE(InfoExtractor):
        'params': {
            # rtmp download
            'skip_download': True,
-        },
+        }
        'skip': 'gone',
    }, {
        # media.cms.nova.cz embed
        'url': 'https://novaplus.nova.cz/porad/ulice/epizoda/18760-2180-dil',
@ -171,7 +128,6 @@ class NovaIE(InfoExtractor):
            'skip_download': True,
        },
        'add_ie': [NovaEmbedIE.ie_key()],
        'skip': 'CHYBA 404: STRÁNKA NENALEZENA',
    }, {
        'url': 'http://sport.tn.nova.cz/clanek/sport/hokej/nhl/zivot-jde-dal-hodnotil-po-vyrazeni-z-playoff-jiri-sekac.html',
        'only_matching': True,
@ -196,29 +152,14 @@ class NovaIE(InfoExtractor):
        webpage = self._download_webpage(url, display_id)
        description = clean_html(self._og_search_description(webpage, default=None))
        if site == 'novaplus':
            upload_date = unified_strdate(self._search_regex(
                r'(\d{1,2}-\d{1,2}-\d{4})$', display_id, 'upload date', default=None))
        elif site == 'fanda':
            upload_date = unified_strdate(self._search_regex(
                r'<span class="date_time">(\d{1,2}\.\d{1,2}\.\d{4})', webpage, 'upload date', default=None))
        else:
            upload_date = None
        # novaplus
        embed_id = self._search_regex(
            r'<iframe[^>]+\bsrc=["\'](?:https?:)?//media\.cms\.nova\.cz/embed/([^/?#&]+)',
            webpage, 'embed url', default=None)
        if embed_id:
-            return {
+            return self.url_result(
-                '_type': 'url_transparent',
+                'https://media.cms.nova.cz/embed/%s' % embed_id,
-                'url': 'https://media.cms.nova.cz/embed/%s' % embed_id,
+                ie=NovaEmbedIE.ie_key(), video_id=embed_id)
                'ie_key': NovaEmbedIE.ie_key(),
                'id': embed_id,
                'description': description,
                'upload_date': upload_date
            }
        video_id = self._search_regex(
            [r"(?:media|video_id)\s*:\s*'(\d+)'",
@ -292,8 +233,18 @@ class NovaIE(InfoExtractor):
        self._sort_formats(formats)
        title = mediafile.get('meta', {}).get('title') or self._og_search_title(webpage)
        description = clean_html(self._og_search_description(webpage, default=None))
        thumbnail = config.get('poster')
        if site == 'novaplus':
            upload_date = unified_strdate(self._search_regex(
                r'(\d{1,2}-\d{1,2}-\d{4})$', display_id, 'upload date', default=None))
        elif site == 'fanda':
            upload_date = unified_strdate(self._search_regex(
                r'<span class="date_time">(\d{1,2}\.\d{1,2}\.\d{4})', webpage, 'upload date', default=None))
        else:
            upload_date = None
        return {
            'id': video_id,
            'display_id': display_id,
--- a/youtube_dl/extractor/npr.py
+++ b/youtube_dl/extractor/npr.py
@ -4,7 +4,6 @@ from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    qualities,
    url_or_none,
 )
@ -49,10 +48,6 @@ class NprIE(InfoExtractor):
            },
        }],
        'expected_warnings': ['Failed to download m3u8 information'],
    }, {
        # multimedia, no formats, stream
        'url': 'https://www.npr.org/2020/02/14/805476846/laura-stevenson-tiny-desk-concert',
        'only_matching': True,
    }]
    def _real_extract(self, url):
@ -100,17 +95,6 @@ class NprIE(InfoExtractor):
                            'format_id': format_id,
                            'quality': quality(format_id),
                        })
            for stream_id, stream_entry in media.get('stream', {}).items():
                if not isinstance(stream_entry, dict):
                    continue
                if stream_id != 'hlsUrl':
                    continue
                stream_url = url_or_none(stream_entry.get('$text'))
                if not stream_url:
                    continue
                formats.extend(self._extract_m3u8_formats(
                    stream_url, stream_id, 'mp4', 'm3u8_native',
                    m3u8_id='hls', fatal=False))
            self._sort_formats(formats)
            entries.append({
--- a/youtube_dl/extractor/nrk.py
+++ b/youtube_dl/extractor/nrk.py
@ -11,7 +11,7 @@ from ..compat import (
 from ..utils import (
    ExtractorError,
    int_or_none,
-    js_to_json,
+    JSON_LD_RE,
    NO_DEFAULT,
    parse_age_limit,
    parse_duration,
@ -105,7 +105,6 @@ class NRKBaseIE(InfoExtractor):
            MESSAGES = {
                'ProgramRightsAreNotReady': 'Du kan dessverre ikke se eller høre programmet',
                'ProgramRightsHasExpired': 'Programmet har gått ut',
                'NoProgramRights': 'Ikke tilgjengelig',
                'ProgramIsGeoBlocked': 'NRK har ikke rettigheter til å vise dette programmet utenfor Norge',
            }
            message_type = data.get('messageType', '')
@ -256,17 +255,6 @@ class NRKTVIE(NRKBaseIE):
                    ''' % _EPISODE_RE
    _API_HOSTS = ('psapi-ne.nrk.no', 'psapi-we.nrk.no')
    _TESTS = [{
        'url': 'https://tv.nrk.no/program/MDDP12000117',
        'md5': '8270824df46ec629b66aeaa5796b36fb',
        'info_dict': {
            'id': 'MDDP12000117AA',
            'ext': 'mp4',
            'title': 'Alarm Trolltunga',
            'description': 'md5:46923a6e6510eefcce23d5ef2a58f2ce',
            'duration': 2223,
            'age_limit': 6,
        },
    }, {
        'url': 'https://tv.nrk.no/serie/20-spoersmaal-tv/MUHH48000314/23-05-2014',
        'md5': '9a167e54d04671eb6317a37b7bc8a280',
        'info_dict': {
@ -278,7 +266,6 @@ class NRKTVIE(NRKBaseIE):
            'series': '20 spørsmål',
            'episode': '23.05.2014',
        },
        'skip': 'NoProgramRights',
    }, {
        'url': 'https://tv.nrk.no/program/mdfp15000514',
        'info_dict': {
@ -383,24 +370,7 @@ class NRKTVIE(NRKBaseIE):
 class NRKTVEpisodeIE(InfoExtractor):
    _VALID_URL = r'https?://tv\.nrk\.no/serie/(?P<id>[^/]+/sesong/\d+/episode/\d+)'
-    _TESTS = [{
+    _TEST = {
        'url': 'https://tv.nrk.no/serie/hellums-kro/sesong/1/episode/2',
        'info_dict': {
            'id': 'MUHH36005220BA',
            'ext': 'mp4',
            'title': 'Kro, krig og kjærlighet 2:6',
            'description': 'md5:b32a7dc0b1ed27c8064f58b97bda4350',
            'duration': 1563,
            'series': 'Hellums kro',
            'season_number': 1,
            'episode_number': 2,
            'episode': '2:6',
            'age_limit': 6,
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://tv.nrk.no/serie/backstage/sesong/1/episode/8',
        'info_dict': {
            'id': 'MSUI14000816AA',
@ -416,28 +386,20 @@ class NRKTVEpisodeIE(InfoExtractor):
        'params': {
            'skip_download': True,
        },
-        'skip': 'ProgramRightsHasExpired',
+    }
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
-        info = self._search_json_ld(webpage, display_id, default={})
+        nrk_id = self._parse_json(
-        nrk_id = info.get('@id') or self._html_search_meta(
+            self._search_regex(JSON_LD_RE, webpage, 'JSON-LD', group='json_ld'),
-            'nrk:program-id', webpage, default=None) or self._search_regex(
+            display_id)['@id']
            r'data-program-id=["\'](%s)' % NRKTVIE._EPISODE_RE, webpage,
            'nrk id')
        assert re.match(NRKTVIE._EPISODE_RE, nrk_id)
-        info.update({
+        assert re.match(NRKTVIE._EPISODE_RE, nrk_id)
-            '_type': 'url_transparent',
+        return self.url_result(
-            'id': nrk_id,
+            'nrk:%s' % nrk_id, ie=NRKIE.ie_key(), video_id=nrk_id)
            'url': 'nrk:%s' % nrk_id,
            'ie_key': NRKIE.ie_key(),
        })
        return info
 class NRKTVSerieBaseIE(InfoExtractor):
@ -447,7 +409,7 @@ class NRKTVSerieBaseIE(InfoExtractor):
                (r'INITIAL_DATA(?:_V\d)?_*\s*=\s*({.+?})\s*;',
                 r'({.+?})\s*,\s*"[^"]+"\s*\)\s*</script>'),
                webpage, 'config', default='{}' if not fatal else NO_DEFAULT),
-            display_id, fatal=False, transform_source=js_to_json)
+            display_id, fatal=False)
        if not config:
            return
        return try_get(
@ -517,14 +479,6 @@ class NRKTVSeriesIE(NRKTVSerieBaseIE):
    _VALID_URL = r'https?://(?:tv|radio)\.nrk(?:super)?\.no/serie/(?P<id>[^/]+)'
    _ITEM_RE = r'(?:data-season=["\']|id=["\']season-)(?P<id>\d+)'
    _TESTS = [{
        'url': 'https://tv.nrk.no/serie/blank',
        'info_dict': {
            'id': 'blank',
            'title': 'Blank',
            'description': 'md5:7664b4e7e77dc6810cd3bca367c25b6e',
        },
        'playlist_mincount': 30,
    }, {
        # new layout, seasons
        'url': 'https://tv.nrk.no/serie/backstage',
        'info_dict': {
@ -694,7 +648,7 @@ class NRKSkoleIE(InfoExtractor):
    _TESTS = [{
        'url': 'https://www.nrk.no/skole/?page=search&q=&mediaId=14099',
-        'md5': '18c12c3d071953c3bf8d54ef6b2587b7',
+        'md5': '6bc936b01f9dd8ed45bc58b252b2d9b6',
        'info_dict': {
            'id': '6021',
            'ext': 'mp4',
--- a/youtube_dl/extractor/nrl.py
+++ b/youtube_dl/extractor/nrl.py
@ -23,8 +23,8 @@ class NRLTVIE(InfoExtractor):
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
-        q_data = self._parse_json(self._html_search_regex(
+        q_data = self._parse_json(self._search_regex(
-            r'(?s)q-data="({.+?})"', webpage, 'player data'), display_id)
+            r"(?s)q-data='({.+?})'", webpage, 'player data'), display_id)
        ooyala_id = q_data['videoId']
        return self.url_result(
            'ooyala:' + ooyala_id, 'Ooyala', ooyala_id, q_data.get('title'))
--- a/youtube_dl/extractor/nytimes.py
+++ b/youtube_dl/extractor/nytimes.py
@ -69,10 +69,10 @@ class NYTimesBaseIE(InfoExtractor):
                    'width': int_or_none(video.get('width')),
                    'height': int_or_none(video.get('height')),
                    'filesize': get_file_size(video.get('file_size') or video.get('fileSize')),
-                    'tbr': int_or_none(video.get('bitrate'), 1000) or None,
+                    'tbr': int_or_none(video.get('bitrate'), 1000),
                    'ext': ext,
                })
-        self._sort_formats(formats, ('height', 'width', 'filesize', 'tbr', 'fps', 'format_id'))
+        self._sort_formats(formats)
        thumbnails = []
        for image in video_data.get('images', []):
--- a/youtube_dl/extractor/ooyala.py
+++ b/youtube_dl/extractor/ooyala.py
@ -1,12 +1,12 @@
 from __future__ import unicode_literals
 import base64
 import re
 from .common import InfoExtractor
 from ..compat import (
    compat_b64decode,
    compat_str,
    compat_urllib_parse_urlencode,
 )
 from ..utils import (
    determine_ext,
@ -21,9 +21,9 @@ from ..utils import (
 class OoyalaBaseIE(InfoExtractor):
    _PLAYER_BASE = 'http://player.ooyala.com/'
    _CONTENT_TREE_BASE = _PLAYER_BASE + 'player_api/v1/content_tree/'
-    _AUTHORIZATION_URL_TEMPLATE = _PLAYER_BASE + 'sas/player_api/v2/authorization/embed_code/%s/%s'
+    _AUTHORIZATION_URL_TEMPLATE = _PLAYER_BASE + 'sas/player_api/v2/authorization/embed_code/%s/%s?'
-    def _extract(self, content_tree_url, video_id, domain=None, supportedformats=None, embed_token=None):
+    def _extract(self, content_tree_url, video_id, domain='example.org', supportedformats=None, embed_token=None):
        content_tree = self._download_json(content_tree_url, video_id)['content_tree']
        metadata = content_tree[list(content_tree)[0]]
        embed_code = metadata['embed_code']
@ -31,62 +31,59 @@ class OoyalaBaseIE(InfoExtractor):
        title = metadata['title']
        auth_data = self._download_json(
-            self._AUTHORIZATION_URL_TEMPLATE % (pcode, embed_code),
+            self._AUTHORIZATION_URL_TEMPLATE % (pcode, embed_code)
-            video_id, headers=self.geo_verification_headers(), query={
+            + compat_urllib_parse_urlencode({
-                'domain': domain or 'player.ooyala.com',
+                'domain': domain,
                'supportedFormats': supportedformats or 'mp4,rtmp,m3u8,hds,dash,smooth',
                'embedToken': embed_token,
-            })['authorization_data'][embed_code]
+            }), video_id, headers=self.geo_verification_headers())
        cur_auth_data = auth_data['authorization_data'][embed_code]
        urls = []
        formats = []
-        streams = auth_data.get('streams') or [{
+        if cur_auth_data['authorized']:
-            'delivery_type': 'hls',
+            for stream in cur_auth_data['streams']:
-            'url': {
+                url_data = try_get(stream, lambda x: x['url']['data'], compat_str)
-                'data': base64.b64encode(('http://player.ooyala.com/hls/player/all/%s.m3u8' % embed_code).encode()).decode(),
+                if not url_data:
-            }
+                    continue
-        }]
+                s_url = compat_b64decode(url_data).decode('utf-8')
-        for stream in streams:
+                if not s_url or s_url in urls:
-            url_data = try_get(stream, lambda x: x['url']['data'], compat_str)
+                    continue
-            if not url_data:
+                urls.append(s_url)
-                continue
+                ext = determine_ext(s_url, None)
-            s_url = compat_b64decode(url_data).decode('utf-8')
+                delivery_type = stream.get('delivery_type')
-            if not s_url or s_url in urls:
+                if delivery_type == 'hls' or ext == 'm3u8':
-                continue
+                    formats.extend(self._extract_m3u8_formats(
-            urls.append(s_url)
+                        re.sub(r'/ip(?:ad|hone)/', '/all/', s_url), embed_code, 'mp4', 'm3u8_native',
-            ext = determine_ext(s_url, None)
+                        m3u8_id='hls', fatal=False))
-            delivery_type = stream.get('delivery_type')
+                elif delivery_type == 'hds' or ext == 'f4m':
-            if delivery_type == 'hls' or ext == 'm3u8':
+                    formats.extend(self._extract_f4m_formats(
-                formats.extend(self._extract_m3u8_formats(
+                        s_url + '?hdcore=3.7.0', embed_code, f4m_id='hds', fatal=False))
-                    re.sub(r'/ip(?:ad|hone)/', '/all/', s_url), embed_code, 'mp4', 'm3u8_native',
+                elif delivery_type == 'dash' or ext == 'mpd':
-                    m3u8_id='hls', fatal=False))
+                    formats.extend(self._extract_mpd_formats(
-            elif delivery_type == 'hds' or ext == 'f4m':
+                        s_url, embed_code, mpd_id='dash', fatal=False))
-                formats.extend(self._extract_f4m_formats(
+                elif delivery_type == 'smooth':
-                    s_url + '?hdcore=3.7.0', embed_code, f4m_id='hds', fatal=False))
+                    self._extract_ism_formats(
-            elif delivery_type == 'dash' or ext == 'mpd':
+                        s_url, embed_code, ism_id='mss', fatal=False)
-                formats.extend(self._extract_mpd_formats(
+                elif ext == 'smil':
-                    s_url, embed_code, mpd_id='dash', fatal=False))
+                    formats.extend(self._extract_smil_formats(
-            elif delivery_type == 'smooth':
+                        s_url, embed_code, fatal=False))
-                self._extract_ism_formats(
+                else:
-                    s_url, embed_code, ism_id='mss', fatal=False)
+                    formats.append({
-            elif ext == 'smil':
+                        'url': s_url,
-                formats.extend(self._extract_smil_formats(
+                        'ext': ext or delivery_type,
-                    s_url, embed_code, fatal=False))
+                        'vcodec': stream.get('video_codec'),
-            else:
+                        'format_id': delivery_type,
-                formats.append({
+                        'width': int_or_none(stream.get('width')),
-                    'url': s_url,
+                        'height': int_or_none(stream.get('height')),
-                    'ext': ext or delivery_type,
+                        'abr': int_or_none(stream.get('audio_bitrate')),
-                    'vcodec': stream.get('video_codec'),
+                        'vbr': int_or_none(stream.get('video_bitrate')),
-                    'format_id': delivery_type,
+                        'fps': float_or_none(stream.get('framerate')),
-                    'width': int_or_none(stream.get('width')),
+                    })
-                    'height': int_or_none(stream.get('height')),
+        else:
                    'abr': int_or_none(stream.get('audio_bitrate')),
                    'vbr': int_or_none(stream.get('video_bitrate')),
                    'fps': float_or_none(stream.get('framerate')),
                })
        if not formats and not auth_data.get('authorized'):
            raise ExtractorError('%s said: %s' % (
-                self.IE_NAME, auth_data['message']), expected=True)
+                self.IE_NAME, cur_auth_data['message']), expected=True)
        self._sort_formats(formats)
        subtitles = {}
--- a/youtube_dl/extractor/orf.py
+++ b/youtube_dl/extractor/orf.py
@ -6,14 +6,12 @@ import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    clean_html,
    determine_ext,
    float_or_none,
    HEADRequest,
    int_or_none,
    orderedSet,
    remove_end,
    str_or_none,
    strip_jsonp,
    unescapeHTML,
    unified_strdate,
@ -90,11 +88,8 @@ class ORFTVthekIE(InfoExtractor):
                format_id = '-'.join(format_id_list)
                ext = determine_ext(src)
                if ext == 'm3u8':
-                    m3u8_formats = self._extract_m3u8_formats(
+                    formats.extend(self._extract_m3u8_formats(
-                        src, video_id, 'mp4', m3u8_id=format_id, fatal=False)
+                        src, video_id, 'mp4', m3u8_id=format_id, fatal=False))
                    if any('/geoprotection' in f['url'] for f in m3u8_formats):
                        self.raise_geo_restricted()
                    formats.extend(m3u8_formats)
                elif ext == 'f4m':
                    formats.extend(self._extract_f4m_formats(
                        src, video_id, f4m_id=format_id, fatal=False))
@ -162,53 +157,48 @@ class ORFTVthekIE(InfoExtractor):
 class ORFRadioIE(InfoExtractor):
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        station = mobj.group('station')
        show_date = mobj.group('date')
        show_id = mobj.group('show')
-        data = self._download_json(
+        if station == 'fm4':
-            'http://audioapi.orf.at/%s/api/json/current/broadcast/%s/%s'
+            show_id = '4%s' % show_id
            % (self._API_STATION, show_id, show_date), show_id)
-        entries = []
+        data = self._download_json(
-        for info in data['streams']:
+            'http://audioapi.orf.at/%s/api/json/current/broadcast/%s/%s' % (station, show_id, show_date),
-            loop_stream_id = str_or_none(info.get('loopStreamId'))
+            show_id
-            if not loop_stream_id:
+        )
-                continue
+
-            title = str_or_none(data.get('title'))
+        def extract_entry_dict(info, title, subtitle):
-            if not title:
+            return {
-                continue
+                'id': info['loopStreamId'].replace('.mp3', ''),
-            start = int_or_none(info.get('start'), scale=1000)
+                'url': 'http://loopstream01.apa.at/?channel=%s&id=%s' % (station, info['loopStreamId']),
            end = int_or_none(info.get('end'), scale=1000)
            duration = end - start if end and start else None
            entries.append({
                'id': loop_stream_id.replace('.mp3', ''),
                'url': 'http://loopstream01.apa.at/?channel=%s&id=%s' % (self._LOOP_STATION, loop_stream_id),
                'title': title,
-                'description': clean_html(data.get('subtitle')),
+                'description': subtitle,
-                'duration': duration,
+                'duration': (info['end'] - info['start']) / 1000,
-                'timestamp': start,
+                'timestamp': info['start'] / 1000,
                'ext': 'mp3',
-                'series': data.get('programTitle'),
+                'series': data.get('programTitle')
-            })
+            }
        entries = [extract_entry_dict(t, data['title'], data['subtitle']) for t in data['streams']]
        return {
            '_type': 'playlist',
            'id': show_id,
-            'title': data.get('title'),
+            'title': data['title'],
-            'description': clean_html(data.get('subtitle')),
+            'description': data['subtitle'],
-            'entries': entries,
+            'entries': entries
        }
 class ORFFM4IE(ORFRadioIE):
    IE_NAME = 'orf:fm4'
    IE_DESC = 'radio FM4'
-    _VALID_URL = r'https?://(?P<station>fm4)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>4\w+)'
+    _VALID_URL = r'https?://(?P<station>fm4)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'fm4'
    _LOOP_STATION = 'fm4'
    _TEST = {
-        'url': 'http://fm4.orf.at/player/20170107/4CC',
+        'url': 'http://fm4.orf.at/player/20170107/CC',
        'md5': '2b0be47375432a7ef104453432a19212',
        'info_dict': {
            'id': '2017-01-07_2100_tl_54_7DaysSat18_31295',
@ -219,138 +209,7 @@ class ORFFM4IE(ORFRadioIE):
            'timestamp': 1483819257,
            'upload_date': '20170107',
        },
-        'skip': 'Shows from ORF radios are only available for 7 days.',
+        'skip': 'Shows from ORF radios are only available for 7 days.'
        'only_matching': True,
    }
 class ORFNOEIE(ORFRadioIE):
    IE_NAME = 'orf:noe'
    IE_DESC = 'Radio Niederösterreich'
    _VALID_URL = r'https?://(?P<station>noe)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'noe'
    _LOOP_STATION = 'oe2n'
    _TEST = {
        'url': 'https://noe.orf.at/player/20200423/NGM',
        'only_matching': True,
    }
 class ORFWIEIE(ORFRadioIE):
    IE_NAME = 'orf:wien'
    IE_DESC = 'Radio Wien'
    _VALID_URL = r'https?://(?P<station>wien)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'wie'
    _LOOP_STATION = 'oe2w'
    _TEST = {
        'url': 'https://wien.orf.at/player/20200423/WGUM',
        'only_matching': True,
    }
 class ORFBGLIE(ORFRadioIE):
    IE_NAME = 'orf:burgenland'
    IE_DESC = 'Radio Burgenland'
    _VALID_URL = r'https?://(?P<station>burgenland)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'bgl'
    _LOOP_STATION = 'oe2b'
    _TEST = {
        'url': 'https://burgenland.orf.at/player/20200423/BGM',
        'only_matching': True,
    }
 class ORFOOEIE(ORFRadioIE):
    IE_NAME = 'orf:oberoesterreich'
    IE_DESC = 'Radio Oberösterreich'
    _VALID_URL = r'https?://(?P<station>ooe)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'ooe'
    _LOOP_STATION = 'oe2o'
    _TEST = {
        'url': 'https://ooe.orf.at/player/20200423/OGMO',
        'only_matching': True,
    }
 class ORFSTMIE(ORFRadioIE):
    IE_NAME = 'orf:steiermark'
    IE_DESC = 'Radio Steiermark'
    _VALID_URL = r'https?://(?P<station>steiermark)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'stm'
    _LOOP_STATION = 'oe2st'
    _TEST = {
        'url': 'https://steiermark.orf.at/player/20200423/STGMS',
        'only_matching': True,
    }
 class ORFKTNIE(ORFRadioIE):
    IE_NAME = 'orf:kaernten'
    IE_DESC = 'Radio Kärnten'
    _VALID_URL = r'https?://(?P<station>kaernten)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'ktn'
    _LOOP_STATION = 'oe2k'
    _TEST = {
        'url': 'https://kaernten.orf.at/player/20200423/KGUMO',
        'only_matching': True,
    }
 class ORFSBGIE(ORFRadioIE):
    IE_NAME = 'orf:salzburg'
    IE_DESC = 'Radio Salzburg'
    _VALID_URL = r'https?://(?P<station>salzburg)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'sbg'
    _LOOP_STATION = 'oe2s'
    _TEST = {
        'url': 'https://salzburg.orf.at/player/20200423/SGUM',
        'only_matching': True,
    }
 class ORFTIRIE(ORFRadioIE):
    IE_NAME = 'orf:tirol'
    IE_DESC = 'Radio Tirol'
    _VALID_URL = r'https?://(?P<station>tirol)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'tir'
    _LOOP_STATION = 'oe2t'
    _TEST = {
        'url': 'https://tirol.orf.at/player/20200423/TGUMO',
        'only_matching': True,
    }
 class ORFVBGIE(ORFRadioIE):
    IE_NAME = 'orf:vorarlberg'
    IE_DESC = 'Radio Vorarlberg'
    _VALID_URL = r'https?://(?P<station>vorarlberg)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'vbg'
    _LOOP_STATION = 'oe2v'
    _TEST = {
        'url': 'https://vorarlberg.orf.at/player/20200423/VGUM',
        'only_matching': True,
    }
 class ORFOE3IE(ORFRadioIE):
    IE_NAME = 'orf:oe3'
    IE_DESC = 'Radio Österreich 3'
    _VALID_URL = r'https?://(?P<station>oe3)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'oe3'
    _LOOP_STATION = 'oe3'
    _TEST = {
        'url': 'https://oe3.orf.at/player/20200424/3WEK',
        'only_matching': True,
    }
@ -358,8 +217,6 @@ class ORFOE1IE(ORFRadioIE):
    IE_NAME = 'orf:oe1'
    IE_DESC = 'Radio Österreich 1'
    _VALID_URL = r'https?://(?P<station>oe1)\.orf\.at/player/(?P<date>[0-9]+)/(?P<show>\w+)'
    _API_STATION = 'oe1'
    _LOOP_STATION = 'oe1'
    _TEST = {
        'url': 'http://oe1.orf.at/player/20170108/456544',
--- a/youtube_dl/extractor/pandatv.py
+++ b/youtube_dl/extractor/pandatv.py
@ -0,0 +1,99 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    qualities,
 )
 class PandaTVIE(InfoExtractor):
    IE_DESC = '熊猫TV'
    _VALID_URL = r'https?://(?:www\.)?panda\.tv/(?P<id>[0-9]+)'
    _TESTS = [{
        'url': 'http://www.panda.tv/66666',
        'info_dict': {
            'id': '66666',
            'title': 're:.+',
            'uploader': '刘杀鸡',
            'ext': 'flv',
            'is_live': True,
        },
        'params': {
            'skip_download': True,
        },
        'skip': 'Live stream is offline',
    }, {
        'url': 'https://www.panda.tv/66666',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        config = self._download_json(
            'https://www.panda.tv/api_room_v2?roomid=%s' % video_id, video_id)
        error_code = config.get('errno', 0)
        if error_code != 0:
            raise ExtractorError(
                '%s returned error %s: %s'
                % (self.IE_NAME, error_code, config['errmsg']),
                expected=True)
        data = config['data']
        video_info = data['videoinfo']
        # 2 = live, 3 = offline
        if video_info.get('status') != '2':
            raise ExtractorError(
                'Live stream is offline', expected=True)
        title = data['roominfo']['name']
        uploader = data.get('hostinfo', {}).get('name')
        room_key = video_info['room_key']
        stream_addr = video_info.get(
            'stream_addr', {'OD': '1', 'HD': '1', 'SD': '1'})
        # Reverse engineered from web player swf
        # (http://s6.pdim.gs/static/07153e425f581151.swf at the moment of
        # writing).
        plflag0, plflag1 = video_info['plflag'].split('_')
        plflag0 = int(plflag0) - 1
        if plflag1 == '21':
            plflag0 = 10
            plflag1 = '4'
        live_panda = 'live_panda' if plflag0 < 1 else ''
        plflag_auth = self._parse_json(video_info['plflag_list'], video_id)
        sign = plflag_auth['auth']['sign']
        ts = plflag_auth['auth']['time']
        rid = plflag_auth['auth']['rid']
        quality_key = qualities(['OD', 'HD', 'SD'])
        suffix = ['_small', '_mid', '']
        formats = []
        for k, v in stream_addr.items():
            if v != '1':
                continue
            quality = quality_key(k)
            if quality <= 0:
                continue
            for pref, (ext, pl) in enumerate((('m3u8', '-hls'), ('flv', ''))):
                formats.append({
                    'url': 'https://pl%s%s.live.panda.tv/live_panda/%s%s%s.%s?sign=%s&ts=%s&rid=%s'
                    % (pl, plflag1, room_key, live_panda, suffix[quality], ext, sign, ts, rid),
                    'format_id': '%s-%s' % (k, ext),
                    'quality': quality,
                    'source_preference': pref,
                })
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': self._live_title(title),
            'uploader': uploader,
            'formats': formats,
            'is_live': True,
        }
--- a/youtube_dl/extractor/peertube.py
+++ b/youtube_dl/extractor/peertube.py
@ -8,7 +8,6 @@ from ..compat import compat_str
 from ..utils import (
    int_or_none,
    parse_resolution,
    str_or_none,
    try_get,
    unified_timestamp,
    url_or_none,
@ -416,7 +415,6 @@ class PeerTubeIE(InfoExtractor):
                            peertube\.cpy\.re
                        )'''
    _UUID_RE = r'[\da-fA-F]{8}-[\da-fA-F]{4}-[\da-fA-F]{4}-[\da-fA-F]{4}-[\da-fA-F]{12}'
    _API_BASE = 'https://%s/api/v1/videos/%s/%s'
    _VALID_URL = r'''(?x)
                    (?:
                        peertube:(?P<host>[^:]+):|
@ -425,30 +423,26 @@ class PeerTubeIE(InfoExtractor):
                    (?P<id>%s)
                    ''' % (_INSTANCES_RE, _UUID_RE)
    _TESTS = [{
-        'url': 'https://framatube.org/videos/watch/9c9de5e8-0a1e-484a-b099-e80766180a6d',
+        'url': 'https://peertube.cpy.re/videos/watch/2790feb0-8120-4e63-9af3-c943c69f5e6c',
-        'md5': '9bed8c0137913e17b86334e5885aacff',
+        'md5': '80f24ff364cc9d333529506a263e7feb',
        'info_dict': {
-            'id': '9c9de5e8-0a1e-484a-b099-e80766180a6d',
+            'id': '2790feb0-8120-4e63-9af3-c943c69f5e6c',
            'ext': 'mp4',
-            'title': 'What is PeerTube?',
+            'title': 'wow',
-            'description': 'md5:3fefb8dde2b189186ce0719fda6f7b10',
+            'description': 'wow such video, so gif',
            'thumbnail': r're:https?://.*\.(?:jpg|png)',
-            'timestamp': 1538391166,
+            'timestamp': 1519297480,
-            'upload_date': '20181001',
+            'upload_date': '20180222',
-            'uploader': 'Framasoft',
+            'uploader': 'Luclu7',
-            'uploader_id': '3',
+            'uploader_id': '7fc42640-efdb-4505-a45d-a15b1a5496f1',
-            'uploader_url': 'https://framatube.org/accounts/framasoft',
+            'uploder_url': 'https://peertube.nsa.ovh/accounts/luclu7',
-            'channel': 'Les vidéos de Framasoft',
+            'license': 'Unknown',
-            'channel_id': '2',
+            'duration': 3,
            'channel_url': 'https://framatube.org/video-channels/bf54d359-cfad-4935-9d45-9d6be93f63e8',
            'language': 'en',
            'license': 'Attribution - Share Alike',
            'duration': 113,
            'view_count': int,
            'like_count': int,
            'dislike_count': int,
-            'tags': ['framasoft', 'peertube'],
+            'tags': list,
-            'categories': ['Science & Technology'],
+            'categories': list,
        }
    }, {
        'url': 'https://peertube.tamanoir.foucry.net/videos/watch/0b04f13d-1e18-4f1d-814e-4979aa7c9c44',
@ -490,38 +484,13 @@ class PeerTubeIE(InfoExtractor):
                entries = [peertube_url]
        return entries
    def _call_api(self, host, video_id, path, note=None, errnote=None, fatal=True):
        return self._download_json(
            self._API_BASE % (host, video_id, path), video_id,
            note=note, errnote=errnote, fatal=fatal)
    def _get_subtitles(self, host, video_id):
        captions = self._call_api(
            host, video_id, 'captions', note='Downloading captions JSON',
            fatal=False)
        if not isinstance(captions, dict):
            return
        data = captions.get('data')
        if not isinstance(data, list):
            return
        subtitles = {}
        for e in data:
            language_id = try_get(e, lambda x: x['language']['id'], compat_str)
            caption_url = urljoin('https://%s' % host, e.get('captionPath'))
            if not caption_url:
                continue
            subtitles.setdefault(language_id or 'en', []).append({
                'url': caption_url,
            })
        return subtitles
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        host = mobj.group('host') or mobj.group('host_2')
        video_id = mobj.group('id')
-        video = self._call_api(
+        video = self._download_json(
-            host, video_id, '', note='Downloading video JSON')
+            'https://%s/api/v1/videos/%s' % (host, video_id), video_id)
        title = video['name']
@ -544,28 +513,10 @@ class PeerTubeIE(InfoExtractor):
            formats.append(f)
        self._sort_formats(formats)
-        full_description = self._call_api(
+        def account_data(field):
-            host, video_id, 'description', note='Downloading description JSON',
+            return try_get(video, lambda x: x['account'][field], compat_str)
            fatal=False)
-        description = None
+        category = try_get(video, lambda x: x['category']['label'], compat_str)
        if isinstance(full_description, dict):
            description = str_or_none(full_description.get('description'))
        if not description:
            description = video.get('description')
        subtitles = self.extract_subtitles(host, video_id)
        def data(section, field, type_):
            return try_get(video, lambda x: x[section][field], type_)
        def account_data(field, type_):
            return data('account', field, type_)
        def channel_data(field, type_):
            return data('channel', field, type_)
        category = data('category', 'label', compat_str)
        categories = [category] if category else None
        nsfw = video.get('nsfw')
@ -577,17 +528,14 @@ class PeerTubeIE(InfoExtractor):
        return {
            'id': video_id,
            'title': title,
-            'description': description,
+            'description': video.get('description'),
            'thumbnail': urljoin(url, video.get('thumbnailPath')),
            'timestamp': unified_timestamp(video.get('publishedAt')),
-            'uploader': account_data('displayName', compat_str),
+            'uploader': account_data('displayName'),
-            'uploader_id': str_or_none(account_data('id', int)),
+            'uploader_id': account_data('uuid'),
-            'uploader_url': url_or_none(account_data('url', compat_str)),
+            'uploder_url': account_data('url'),
-            'channel': channel_data('displayName', compat_str),
+            'license': try_get(
-            'channel_id': str_or_none(channel_data('id', int)),
+                video, lambda x: x['licence']['label'], compat_str),
            'channel_url': url_or_none(channel_data('url', compat_str)),
            'language': data('language', 'id', compat_str),
            'license': data('licence', 'label', compat_str),
            'duration': int_or_none(video.get('duration')),
            'view_count': int_or_none(video.get('views')),
            'like_count': int_or_none(video.get('likes')),
@ -596,5 +544,4 @@ class PeerTubeIE(InfoExtractor):
            'tags': try_get(video, lambda x: x['tags'], list),
            'categories': categories,
            'formats': formats,
            'subtitles': subtitles
        }
--- a/youtube_dl/extractor/periscope.py
+++ b/youtube_dl/extractor/periscope.py
@ -18,7 +18,7 @@ class PeriscopeBaseIE(InfoExtractor):
            item_id, query=query)
    def _parse_broadcast_data(self, broadcast, video_id):
-        title = broadcast.get('status') or 'Periscope Broadcast'
+        title = broadcast['status']
        uploader = broadcast.get('user_display_name') or broadcast.get('username')
        title = '%s - %s' % (uploader, title) if uploader else title
        is_live = broadcast.get('state').lower() == 'running'
--- a/youtube_dl/extractor/platzi.py
+++ b/youtube_dl/extractor/platzi.py
@ -46,7 +46,7 @@ class PlatziBaseIE(InfoExtractor):
            headers={'Referer': self._LOGIN_URL})
        # login succeeded
-        if 'platzi.com/login' not in urlh.geturl():
+        if 'platzi.com/login' not in compat_str(urlh.geturl()):
            return
        login_error = self._webpage_read_content(
--- a/youtube_dl/extractor/pokemon.py
+++ b/youtube_dl/extractor/pokemon.py
@ -20,16 +20,20 @@ class PokemonIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'The Ol’ Raise and Switch!',
            'description': 'md5:7db77f7107f98ba88401d3adc80ff7af',
            'timestamp': 1511824728,
            'upload_date': '20171127',
        },
        'add_id': ['LimelightMedia'],
    }, {
        # no data-video-title
-        'url': 'https://www.pokemon.com/fr/episodes-pokemon/films-pokemon/pokemon-lascension-de-darkrai-2008',
+        'url': 'https://www.pokemon.com/us/pokemon-episodes/pokemon-movies/pokemon-the-rise-of-darkrai-2008',
        'info_dict': {
-            'id': 'dfbaf830d7e54e179837c50c0c6cc0e1',
+            'id': '99f3bae270bf4e5097274817239ce9c8',
            'ext': 'mp4',
-            'title': "Pokémon : L'ascension de Darkrai",
+            'title': 'Pokémon: The Rise of Darkrai',
-            'description': 'md5:d1dbc9e206070c3e14a06ff557659fb5',
+            'description': 'md5:ea8fbbf942e1e497d54b19025dd57d9d',
            'timestamp': 1417778347,
            'upload_date': '20141205',
        },
        'add_id': ['LimelightMedia'],
        'params': {
--- a/youtube_dl/extractor/popcorntimes.py
+++ b/youtube_dl/extractor/popcorntimes.py
@ -1,99 +0,0 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import (
    compat_b64decode,
    compat_chr,
 )
 from ..utils import int_or_none
 class PopcorntimesIE(InfoExtractor):
    _VALID_URL = r'https?://popcorntimes\.tv/[^/]+/m/(?P<id>[^/]+)/(?P<display_id>[^/?#&]+)'
    _TEST = {
        'url': 'https://popcorntimes.tv/de/m/A1XCFvz/haensel-und-gretel-opera-fantasy',
        'md5': '93f210991ad94ba8c3485950a2453257',
        'info_dict': {
            'id': 'A1XCFvz',
            'display_id': 'haensel-und-gretel-opera-fantasy',
            'ext': 'mp4',
            'title': 'Hänsel und Gretel',
            'description': 'md5:1b8146791726342e7b22ce8125cf6945',
            'thumbnail': r're:^https?://.*\.jpg$',
            'creator': 'John Paul',
            'release_date': '19541009',
            'duration': 4260,
            'tbr': 5380,
            'width': 720,
            'height': 540,
        },
    }
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id, display_id = mobj.group('id', 'display_id')
        webpage = self._download_webpage(url, display_id)
        title = self._search_regex(
            r'<h1>([^<]+)', webpage, 'title',
            default=None) or self._html_search_meta(
            'ya:ovs:original_name', webpage, 'title', fatal=True)
        loc = self._search_regex(
            r'PCTMLOC\s*=\s*(["\'])(?P<value>(?:(?!\1).)+)\1', webpage, 'loc',
            group='value')
        loc_b64 = ''
        for c in loc:
            c_ord = ord(c)
            if ord('a') <= c_ord <= ord('z') or ord('A') <= c_ord <= ord('Z'):
                upper = ord('Z') if c_ord <= ord('Z') else ord('z')
                c_ord += 13
                if upper < c_ord:
                    c_ord -= 26
            loc_b64 += compat_chr(c_ord)
        video_url = compat_b64decode(loc_b64).decode('utf-8')
        description = self._html_search_regex(
            r'(?s)<div[^>]+class=["\']pt-movie-desc[^>]+>(.+?)</div>', webpage,
            'description', fatal=False)
        thumbnail = self._search_regex(
            r'<img[^>]+class=["\']video-preview[^>]+\bsrc=(["\'])(?P<value>(?:(?!\1).)+)\1',
            webpage, 'thumbnail', default=None,
            group='value') or self._og_search_thumbnail(webpage)
        creator = self._html_search_meta(
            'video:director', webpage, 'creator', default=None)
        release_date = self._html_search_meta(
            'video:release_date', webpage, default=None)
        if release_date:
            release_date = release_date.replace('-', '')
        def int_meta(name):
            return int_or_none(self._html_search_meta(
                name, webpage, default=None))
        return {
            'id': video_id,
            'display_id': display_id,
            'url': video_url,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'creator': creator,
            'release_date': release_date,
            'duration': int_meta('video:duration'),
            'tbr': int_meta('ya:ovs:bitrate'),
            'width': int_meta('og:video:width'),
            'height': int_meta('og:video:height'),
            'http_headers': {
                'Referer': url,
            },
        }
--- a/youtube_dl/extractor/pornhd.py
+++ b/youtube_dl/extractor/pornhd.py
@ -8,7 +8,6 @@ from ..utils import (
    ExtractorError,
    int_or_none,
    js_to_json,
    merge_dicts,
    urljoin,
 )
@ -28,22 +27,23 @@ class PornHdIE(InfoExtractor):
            'view_count': int,
            'like_count': int,
            'age_limit': 18,
-        },
+        }
        'skip': 'HTTP Error 404: Not Found',
    }, {
        # removed video
        'url': 'http://www.pornhd.com/videos/1962/sierra-day-gets-his-cum-all-over-herself-hd-porn-video',
-        'md5': '1b7b3a40b9d65a8e5b25f7ab9ee6d6de',
+        'md5': '956b8ca569f7f4d8ec563e2c41598441',
        'info_dict': {
            'id': '1962',
            'display_id': 'sierra-day-gets-his-cum-all-over-herself-hd-porn-video',
            'ext': 'mp4',
-            'title': 'md5:98c6f8b2d9c229d0f0fde47f61a1a759',
+            'title': 'Sierra loves doing laundry',
            'description': 'md5:8ff0523848ac2b8f9b065ba781ccf294',
            'thumbnail': r're:^https?://.*\.jpg',
            'view_count': int,
            'like_count': int,
            'age_limit': 18,
        },
        'skip': 'Not available anymore',
    }]
    def _real_extract(self, url):
@ -61,13 +61,7 @@ class PornHdIE(InfoExtractor):
            r"(?s)sources'?\s*[:=]\s*(\{.+?\})",
            webpage, 'sources', default='{}')), video_id)
        info = {}
        if not sources:
            entries = self._parse_html5_media_entries(url, webpage, video_id)
            if entries:
                info = entries[0]
        if not sources and not info:
            message = self._html_search_regex(
                r'(?s)<(div|p)[^>]+class="no-video"[^>]*>(?P<value>.+?)</\1',
                webpage, 'error message', group='value')
@ -86,29 +80,23 @@ class PornHdIE(InfoExtractor):
                'format_id': format_id,
                'height': height,
            })
-        if formats:
+        self._sort_formats(formats)
            info['formats'] = formats
        self._sort_formats(info['formats'])
        description = self._html_search_regex(
-            (r'(?s)<section[^>]+class=["\']video-description[^>]+>(?P<value>.+?)</section>',
+            r'<(div|p)[^>]+class="description"[^>]*>(?P<value>[^<]+)</\1',
-             r'<(div|p)[^>]+class="description"[^>]*>(?P<value>[^<]+)</\1'),
+            webpage, 'description', fatal=False, group='value')
            webpage, 'description', fatal=False,
            group='value') or self._html_search_meta(
            'description', webpage, default=None) or self._og_search_description(webpage)
        view_count = int_or_none(self._html_search_regex(
            r'(\d+) views\s*<', webpage, 'view count', fatal=False))
        thumbnail = self._search_regex(
            r"poster'?\s*:\s*([\"'])(?P<url>(?:(?!\1).)+)\1", webpage,
-            'thumbnail', default=None, group='url')
+            'thumbnail', fatal=False, group='url')
        like_count = int_or_none(self._search_regex(
-            (r'(\d+)</span>\s*likes',
+            (r'(\d+)\s*</11[^>]+>(?:&nbsp;|\s)*\blikes',
             r'(\d+)\s*</11[^>]+>(?:&nbsp;|\s)*\blikes',
             r'class=["\']save-count["\'][^>]*>\s*(\d+)'),
            webpage, 'like count', fatal=False))
-        return merge_dicts(info, {
+        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
@ -118,4 +106,4 @@ class PornHdIE(InfoExtractor):
            'like_count': like_count,
            'formats': formats,
            'age_limit': 18,
-        })
+        }
--- a/youtube_dl/extractor/pornhub.py
+++ b/youtube_dl/extractor/pornhub.py
@ -17,8 +17,6 @@ from ..utils import (
    determine_ext,
    ExtractorError,
    int_or_none,
    merge_dicts,
    NO_DEFAULT,
    orderedSet,
    remove_quotes,
    str_to_int,
@ -53,21 +51,20 @@ class PornHubIE(PornHubBaseIE):
    _VALID_URL = r'''(?x)
                    https?://
                        (?:
-                            (?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net))/(?:(?:view_video\.php|video/show)\?viewkey=|embed/)|
+                            (?:[^/]+\.)?(?P<host>pornhub\.(?:com|net))/(?:(?:view_video\.php|video/show)\?viewkey=|embed/)|
                            (?:www\.)?thumbzilla\.com/video/
                        )
                        (?P<id>[\da-z]+)
                    '''
    _TESTS = [{
        'url': 'http://www.pornhub.com/view_video.php?viewkey=648719015',
-        'md5': 'a6391306d050e4547f62b3f485dd9ba9',
+        'md5': '1e19b41231a02eba417839222ac9d58e',
        'info_dict': {
            'id': '648719015',
            'ext': 'mp4',
            'title': 'Seductive Indian beauty strips down and fingers her pink pussy',
            'uploader': 'Babes',
            'upload_date': '20130628',
            'timestamp': 1372447216,
            'duration': 361,
            'view_count': int,
            'like_count': int,
@ -84,8 +81,8 @@ class PornHubIE(PornHubBaseIE):
            'id': '1331683002',
            'ext': 'mp4',
            'title': '重庆婷婷女王足交',
            'uploader': 'Unknown',
            'upload_date': '20150213',
            'timestamp': 1423804862,
            'duration': 1753,
            'view_count': int,
            'like_count': int,
@ -123,7 +120,6 @@ class PornHubIE(PornHubBaseIE):
        'params': {
            'skip_download': True,
        },
        'skip': 'This video has been disabled',
    }, {
        'url': 'http://www.pornhub.com/view_video.php?viewkey=ph557bbb6676d2d',
        'only_matching': True,
@ -152,9 +148,6 @@ class PornHubIE(PornHubBaseIE):
    }, {
        'url': 'https://www.pornhub.net/view_video.php?viewkey=203640933',
        'only_matching': True,
    }, {
        'url': 'https://www.pornhubpremium.com/view_video.php?viewkey=ph5e4acdae54a82',
        'only_matching': True,
    }]
    @staticmethod
@ -172,13 +165,6 @@ class PornHubIE(PornHubBaseIE):
        host = mobj.group('host') or 'pornhub.com'
        video_id = mobj.group('id')
        if 'premium' in host:
            if not self._downloader.params.get('cookiefile'):
                raise ExtractorError(
                    'PornHub Premium requires authentication.'
                    ' You may want to use --cookies.',
                    expected=True)
        self._set_cookie(host, 'age_verified', '1')
        def dl_webpage(platform):
@ -202,10 +188,10 @@ class PornHubIE(PornHubBaseIE):
        # http://www.pornhub.com/view_video.php?viewkey=1331683002), not relying
        # on that anymore.
        title = self._html_search_meta(
-            'twitter:title', webpage, default=None) or self._html_search_regex(
+            'twitter:title', webpage, default=None) or self._search_regex(
-            (r'(?s)<h1[^>]+class=["\']title["\'][^>]*>(?P<title>.+?)</h1>',
+            (r'<h1[^>]+class=["\']title["\'][^>]*>(?P<title>[^<]+)',
-             r'<div[^>]+data-video-title=(["\'])(?P<title>(?:(?!\1).)+)\1',
+             r'<div[^>]+data-video-title=(["\'])(?P<title>.+?)\1',
-             r'shareTitle["\']\s*[=:]\s*(["\'])(?P<title>(?:(?!\1).)+)\1'),
+             r'shareTitle\s*=\s*(["\'])(?P<title>.+?)\1'),
            webpage, 'title', group='title')
        video_urls = []
@ -241,13 +227,12 @@ class PornHubIE(PornHubBaseIE):
        else:
            thumbnail, duration = [None] * 2
-        def extract_js_vars(webpage, pattern, default=NO_DEFAULT):
+        if not video_urls:
-            assignments = self._search_regex(
+            tv_webpage = dl_webpage('tv')
                pattern, webpage, 'encoded url', default=default)
            if not assignments:
                return {}
-            assignments = assignments.split(';')
+            assignments = self._search_regex(
                r'(var.+?mediastring.+?)</script>', tv_webpage,
                'encoded url').split(';')
            js_vars = {}
@ -269,35 +254,11 @@ class PornHubIE(PornHubBaseIE):
                assn = re.sub(r'var\s+', '', assn)
                vname, value = assn.split('=', 1)
                js_vars[vname] = parse_js_value(value)
            return js_vars
-        def add_video_url(video_url):
+            video_url = js_vars['mediastring']
-            v_url = url_or_none(video_url)
+            if video_url not in video_urls_set:
-            if not v_url:
+                video_urls.append((video_url, None))
-                return
+                video_urls_set.add(video_url)
            if v_url in video_urls_set:
                return
            video_urls.append((v_url, None))
            video_urls_set.add(v_url)
        if not video_urls:
            FORMAT_PREFIXES = ('media', 'quality')
            js_vars = extract_js_vars(
                webpage, r'(var\s+(?:%s)_.+)' % '|'.join(FORMAT_PREFIXES),
                default=None)
            if js_vars:
                for key, format_url in js_vars.items():
                    if any(key.startswith(p) for p in FORMAT_PREFIXES):
                        add_video_url(format_url)
            if not video_urls and re.search(
                    r'<[^>]+\bid=["\']lockedPlayer', webpage):
                raise ExtractorError(
                    'Video %s is locked' % video_id, expected=True)
        if not video_urls:
            js_vars = extract_js_vars(
                dl_webpage('tv'), r'(var.+?mediastring.+?)</script>')
            add_video_url(js_vars['mediastring'])
        for mobj in re.finditer(
                r'<a[^>]+\bclass=["\']downloadBtn\b[^>]+\bhref=(["\'])(?P<url>(?:(?!\1).)+)\1',
@ -315,16 +276,10 @@ class PornHubIE(PornHubBaseIE):
                    r'/(\d{6}/\d{2})/', video_url, 'upload data', default=None)
                if upload_date:
                    upload_date = upload_date.replace('/', '')
-            ext = determine_ext(video_url)
+            if determine_ext(video_url) == 'mpd':
            if ext == 'mpd':
                formats.extend(self._extract_mpd_formats(
                    video_url, video_id, mpd_id='dash', fatal=False))
                continue
            elif ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
                    video_url, video_id, 'mp4', entry_protocol='m3u8_native',
                    m3u8_id='hls', fatal=False))
                continue
            tbr = None
            mobj = re.search(r'(?P<height>\d+)[pP]?_(?P<tbr>\d+)[kK]', video_url)
            if mobj:
@ -341,10 +296,10 @@ class PornHubIE(PornHubBaseIE):
        video_uploader = self._html_search_regex(
            r'(?s)From:&nbsp;.+?<(?:a\b[^>]+\bhref=["\']/(?:(?:user|channel)s|model|pornstar)/|span\b[^>]+\bclass=["\']username)[^>]+>(.+?)<',
-            webpage, 'uploader', default=None)
+            webpage, 'uploader', fatal=False)
        view_count = self._extract_count(
-            r'<span class="count">([\d,\.]+)</span> [Vv]iews', webpage, 'view')
+            r'<span class="count">([\d,\.]+)</span> views', webpage, 'view')
        like_count = self._extract_count(
            r'<span class="votesUp">([\d,\.]+)</span>', webpage, 'like')
        dislike_count = self._extract_count(
@ -359,11 +314,7 @@ class PornHubIE(PornHubBaseIE):
            if div:
                return re.findall(r'<a[^>]+\bhref=[^>]+>([^<]+)', div)
-        info = self._search_json_ld(webpage, video_id, default={})
+        return {
        # description provided in JSON-LD is irrelevant
        info['description'] = None
        return merge_dicts({
            'id': video_id,
            'uploader': video_uploader,
            'upload_date': upload_date,
@ -379,7 +330,7 @@ class PornHubIE(PornHubBaseIE):
            'tags': extract_list('tags'),
            'categories': extract_list('categories'),
            'subtitles': subtitles,
-        }, info)
+        }
 class PornHubPlaylistBaseIE(PornHubBaseIE):
@ -422,7 +373,7 @@ class PornHubPlaylistBaseIE(PornHubBaseIE):
 class PornHubUserIE(PornHubPlaylistBaseIE):
-    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net))/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/?#&]+))(?:[?#&]|/(?!videos)|$)'
+    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?pornhub\.(?:com|net)/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/?#&]+))(?:[?#&]|/(?!videos)|$)'
    _TESTS = [{
        'url': 'https://www.pornhub.com/model/zoe_ph',
        'playlist_mincount': 118,
@ -490,7 +441,7 @@ class PornHubPagedPlaylistBaseIE(PornHubPlaylistBaseIE):
 class PornHubPagedVideoListIE(PornHubPagedPlaylistBaseIE):
-    _VALID_URL = r'https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net))/(?P<id>(?:[^/]+/)*[^/?#&]+)'
+    _VALID_URL = r'https?://(?:[^/]+\.)?(?P<host>pornhub\.(?:com|net))/(?P<id>(?:[^/]+/)*[^/?#&]+)'
    _TESTS = [{
        'url': 'https://www.pornhub.com/model/zoe_ph/videos',
        'only_matching': True,
@ -605,7 +556,7 @@ class PornHubPagedVideoListIE(PornHubPagedPlaylistBaseIE):
 class PornHubUserVideosUploadIE(PornHubPagedPlaylistBaseIE):
-    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net))/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/]+)/videos/upload)'
+    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?(?P<host>pornhub\.(?:com|net))/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/]+)/videos/upload)'
    _TESTS = [{
        'url': 'https://www.pornhub.com/pornstar/jenny-blighe/videos/upload',
        'info_dict': {
--- a/youtube_dl/extractor/prosiebensat1.py
+++ b/youtube_dl/extractor/prosiebensat1.py
@ -11,13 +11,12 @@ from ..utils import (
    determine_ext,
    float_or_none,
    int_or_none,
    merge_dicts,
    unified_strdate,
 )
 class ProSiebenSat1BaseIE(InfoExtractor):
-    _GEO_BYPASS = False
+    _GEO_COUNTRIES = ['DE']
    _ACCESS_ID = None
    _SUPPORTED_PROTOCOLS = 'dash:clear,hls:clear,progressive:clear'
    _V4_BASE_URL = 'https://vas-v4.p7s1video.net/4.0/get'
@ -40,18 +39,14 @@ class ProSiebenSat1BaseIE(InfoExtractor):
        formats = []
        if self._ACCESS_ID:
            raw_ct = self._ENCRYPTION_KEY + clip_id + self._IV + self._ACCESS_ID
-            protocols = self._download_json(
+            server_token = (self._download_json(
                self._V4_BASE_URL + 'protocols', clip_id,
                'Downloading protocols JSON',
                headers=self.geo_verification_headers(), query={
                    'access_id': self._ACCESS_ID,
                    'client_token': sha1((raw_ct).encode()).hexdigest(),
                    'video_id': clip_id,
-                }, fatal=False, expected_status=(403,)) or {}
+                }, fatal=False) or {}).get('server_token')
            error = protocols.get('error') or {}
            if error.get('title') == 'Geo check failed':
                self.raise_geo_restricted(countries=['AT', 'CH', 'DE'])
            server_token = protocols.get('server_token')
            if server_token:
                urls = (self._download_json(
                    self._V4_BASE_URL + 'urls', clip_id, 'Downloading urls JSON', query={
@ -176,7 +171,7 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
                        (?:
                            (?:beta\.)?
                            (?:
-                                prosieben(?:maxx)?|sixx|sat1(?:gold)?|kabeleins(?:doku)?|the-voice-of-germany|advopedia
+                                prosieben(?:maxx)?|sixx|sat1(?:gold)?|kabeleins(?:doku)?|the-voice-of-germany|7tv|advopedia
                            )\.(?:de|at|ch)|
                            ran\.de|fem\.com|advopedia\.de|galileo\.tv/video
                        )
@ -194,14 +189,10 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
            'info_dict': {
                'id': '2104602',
                'ext': 'mp4',
-                'title': 'CIRCUS HALLIGALLI - Episode 18 - Staffel 2',
+                'title': 'Episode 18 - Staffel 2',
                'description': 'md5:8733c81b702ea472e069bc48bb658fc1',
                'upload_date': '20131231',
                'duration': 5845.04,
                'series': 'CIRCUS HALLIGALLI',
                'season_number': 2,
                'episode': 'Episode 18 - Staffel 2',
                'episode_number': 18,
            },
        },
        {
@ -305,9 +296,8 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
            'info_dict': {
                'id': '2572814',
                'ext': 'mp4',
-                'title': 'The Voice of Germany - Andreas Kümmert: Rocket Man',
+                'title': 'Andreas Kümmert: Rocket Man',
                'description': 'md5:6ddb02b0781c6adf778afea606652e38',
                'timestamp': 1382041620,
                'upload_date': '20131017',
                'duration': 469.88,
            },
@ -316,7 +306,7 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
            },
        },
        {
-            'url': 'http://www.fem.com/videos/beauty-lifestyle/kurztrips-zum-valentinstag',
+            'url': 'http://www.fem.com/wellness/videos/wellness-video-clip-kurztripps-zum-valentinstag.html',
            'info_dict': {
                'id': '2156342',
                'ext': 'mp4',
@ -338,6 +328,19 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
            'playlist_count': 2,
            'skip': 'This video is unavailable',
        },
        {
            'url': 'http://www.7tv.de/circus-halligalli/615-best-of-circus-halligalli-ganze-folge',
            'info_dict': {
                'id': '4187506',
                'ext': 'mp4',
                'title': 'Best of Circus HalliGalli',
                'description': 'md5:8849752efd90b9772c9db6fdf87fb9e9',
                'upload_date': '20151229',
            },
            'params': {
                'skip_download': True,
            },
        },
        {
            # title in <h2 class="subtitle">
            'url': 'http://www.prosieben.de/stars/oscar-award/videos/jetzt-erst-enthuellt-das-geheimnis-von-emma-stones-oscar-robe-clip',
@ -414,6 +417,7 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
        r'<div[^>]+id="veeseoDescription"[^>]*>(.+?)</div>',
    ]
    _UPLOAD_DATE_REGEXES = [
        r'<meta property="og:published_time" content="(.+?)">',
        r'<span>\s*(\d{2}\.\d{2}\.\d{4} \d{2}:\d{2}) \|\s*<span itemprop="duration"',
        r'<footer>\s*(\d{2}\.\d{2}\.\d{4}) \d{2}:\d{2} Uhr',
        r'<span style="padding-left: 4px;line-height:20px; color:#404040">(\d{2}\.\d{2}\.\d{4})</span>',
@ -443,21 +447,17 @@ class ProSiebenSat1IE(ProSiebenSat1BaseIE):
        if description is None:
            description = self._og_search_description(webpage)
        thumbnail = self._og_search_thumbnail(webpage)
-        upload_date = unified_strdate(
+        upload_date = unified_strdate(self._html_search_regex(
-            self._html_search_meta('og:published_time', webpage,
+            self._UPLOAD_DATE_REGEXES, webpage, 'upload date', default=None))
                                   'upload date', default=None)
            or self._html_search_regex(self._UPLOAD_DATE_REGEXES,
                                       webpage, 'upload date', default=None))
-        json_ld = self._search_json_ld(webpage, clip_id, default={})
+        info.update({
        return merge_dicts(info, {
            'id': clip_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'upload_date': upload_date,
-        }, json_ld)
+        })
        return info
    def _extract_playlist(self, url, webpage):
        playlist_id = self._html_search_regex(
--- a/youtube_dl/extractor/puhutv.py
+++ b/youtube_dl/extractor/puhutv.py
@ -82,6 +82,17 @@ class PuhuTVIE(InfoExtractor):
        urls = []
        formats = []
        def add_http_from_hls(m3u8_f):
            http_url = m3u8_f['url'].replace('/hls/', '/mp4/').replace('/chunklist.m3u8', '.mp4')
            if http_url != m3u8_f['url']:
                f = m3u8_f.copy()
                f.update({
                    'format_id': f['format_id'].replace('hls', 'http'),
                    'protocol': 'http',
                    'url': http_url,
                })
                formats.append(f)
        for video in videos['data']['videos']:
            media_url = url_or_none(video.get('url'))
            if not media_url or media_url in urls:
@ -90,9 +101,12 @@ class PuhuTVIE(InfoExtractor):
            playlist = video.get('is_playlist')
            if (video.get('stream_type') == 'hls' and playlist is True) or 'playlist.m3u8' in media_url:
-                formats.extend(self._extract_m3u8_formats(
+                m3u8_formats = self._extract_m3u8_formats(
                    media_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                    m3u8_id='hls', fatal=False))
+                    m3u8_id='hls', fatal=False)
                for m3u8_f in m3u8_formats:
                    formats.append(m3u8_f)
                    add_http_from_hls(m3u8_f)
                continue
            quality = int_or_none(video.get('quality'))
@ -114,6 +128,8 @@ class PuhuTVIE(InfoExtractor):
                format_id += '-%sp' % quality
            f['format_id'] = format_id
            formats.append(f)
            if is_hls:
                add_http_from_hls(f)
        self._sort_formats(formats)
        creator = try_get(
--- a/youtube_dl/extractor/redbulltv.py
+++ b/youtube_dl/extractor/redbulltv.py
@ -1,8 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_HTTPError
 from ..utils import (
@ -12,7 +10,7 @@ from ..utils import (
 class RedBullTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?redbull(?:\.tv|\.com(?:/[^/]+)?(?:/tv)?)(?:/events/[^/]+)?/(?:videos?|live|(?:film|episode)s)/(?P<id>AP-\w+)'
+    _VALID_URL = r'https?://(?:www\.)?redbull(?:\.tv|\.com(?:/[^/]+)?(?:/tv)?)(?:/events/[^/]+)?/(?:videos?|live)/(?P<id>AP-\w+)'
    _TESTS = [{
        # film
        'url': 'https://www.redbull.tv/video/AP-1Q6XCDTAN1W11',
@ -31,8 +29,8 @@ class RedBullTVIE(InfoExtractor):
            'id': 'AP-1PMHKJFCW1W11',
            'ext': 'mp4',
            'title': 'Grime - Hashtags S2E4',
-            'description': 'md5:5546aa612958c08a98faaad4abce484d',
+            'description': 'md5:b5f522b89b72e1e23216e5018810bb25',
-            'duration': 904,
+            'duration': 904.6,
        },
        'params': {
            'skip_download': True,
@ -46,15 +44,11 @@ class RedBullTVIE(InfoExtractor):
    }, {
        'url': 'https://www.redbull.com/us-en/events/AP-1XV2K61Q51W11/live/AP-1XUJ86FDH1W11',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/films/AP-1ZSMAW8FH2111',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/episodes/AP-1TQWK7XE11W11',
        'only_matching': True,
    }]
-    def extract_info(self, video_id):
+    def _real_extract(self, url):
        video_id = self._match_id(url)
        session = self._download_json(
            'https://api.redbull.tv/v3/session', video_id,
            note='Downloading access token', query={
@ -111,119 +105,24 @@ class RedBullTVIE(InfoExtractor):
            'subtitles': subtitles,
        }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        return self.extract_info(video_id)
 class RedBullEmbedIE(RedBullTVIE):
    _VALID_URL = r'https?://(?:www\.)?redbull\.com/embed/(?P<id>rrn:content:[^:]+:[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12}:[a-z]{2}-[A-Z]{2,3})'
    _TESTS = [{
        # HLS manifest accessible only using assetId
        'url': 'https://www.redbull.com/embed/rrn:content:episode-videos:f3021f4f-3ed4-51ac-915a-11987126e405:en-INT',
        'only_matching': True,
    }]
    _VIDEO_ESSENSE_TMPL = '''... on %s {
      videoEssence {
        attributes
      }
    }'''
    def _real_extract(self, url):
        rrn_id = self._match_id(url)
        asset_id = self._download_json(
            'https://edge-graphql.crepo-production.redbullaws.com/v1/graphql',
            rrn_id, headers={'API-KEY': 'e90a1ff11335423998b100c929ecc866'},
            query={
                'query': '''{
  resource(id: "%s", enforceGeoBlocking: false) {
    %s
    %s
  }
 }''' % (rrn_id, self._VIDEO_ESSENSE_TMPL % 'LiveVideo', self._VIDEO_ESSENSE_TMPL % 'VideoResource'),
            })['data']['resource']['videoEssence']['attributes']['assetId']
        return self.extract_info(asset_id)
 class RedBullTVRrnContentIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?redbull\.com/(?P<region>[a-z]{2,3})-(?P<lang>[a-z]{2})/tv/(?:video|live|film)/(?P<id>rrn:content:[^:]+:[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
+    _VALID_URL = r'https?://(?:www\.)?redbull(?:\.tv|\.com(?:/[^/]+)?(?:/tv)?)/(?:video|live)/rrn:content:[^:]+:(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
    _TESTS = [{
        'url': 'https://www.redbull.com/int-en/tv/video/rrn:content:live-videos:e3e6feb4-e95f-50b7-962a-c70f8fd13c73/mens-dh-finals-fort-william',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/tv/video/rrn:content:videos:a36a0f36-ff1b-5db8-a69d-ee11a14bf48b/tn-ts-style?playlist=rrn:content:event-profiles:83f05926-5de8-5389-b5e4-9bb312d715e8:extras',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/tv/film/rrn:content:films:d1f4d00e-4c04-5d19-b510-a805ffa2ab83/follow-me',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        region, lang, rrn_id = re.search(self._VALID_URL, url).groups()
+        display_id = self._match_id(url)
        rrn_id += ':%s-%s' % (lang, region.upper())
        return self.url_result(
            'https://www.redbull.com/embed/' + rrn_id,
            RedBullEmbedIE.ie_key(), rrn_id)
        webpage = self._download_webpage(url, display_id)
-class RedBullIE(InfoExtractor):
+        video_url = self._og_search_url(webpage)
    _VALID_URL = r'https?://(?:www\.)?redbull\.com/(?P<region>[a-z]{2,3})-(?P<lang>[a-z]{2})/(?P<type>(?:episode|film|(?:(?:recap|trailer)-)?video)s|live)/(?!AP-|rrn:content:)(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://www.redbull.com/int-en/episodes/grime-hashtags-s02-e04',
        'md5': 'db8271a7200d40053a1809ed0dd574ff',
        'info_dict': {
            'id': 'AA-1MT8DQWA91W14',
            'ext': 'mp4',
            'title': 'Grime - Hashtags S2E4',
            'description': 'md5:5546aa612958c08a98faaad4abce484d',
        },
    }, {
        'url': 'https://www.redbull.com/int-en/films/kilimanjaro-mountain-of-greatness',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/recap-videos/uci-mountain-bike-world-cup-2017-mens-xco-finals-from-vallnord',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/trailer-videos/kings-of-content',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/videos/tnts-style-red-bull-dance-your-style-s1-e12',
        'only_matching': True,
    }, {
        'url': 'https://www.redbull.com/int-en/live/mens-dh-finals-fort-william',
        'only_matching': True,
    }, {
        # only available on the int-en website so a fallback is need for the API
        # https://www.redbull.com/v3/api/graphql/v1/v3/query/en-GB>en-INT?filter[uriSlug]=fia-wrc-saturday-recap-estonia&rb3Schema=v1:hero
        'url': 'https://www.redbull.com/gb-en/live/fia-wrc-saturday-recap-estonia',
        'only_matching': True,
    }]
    _INT_FALLBACK_LIST = ['de', 'en', 'es', 'fr']
    _LAT_FALLBACK_MAP = ['ar', 'bo', 'car', 'cl', 'co', 'mx', 'pe']
    def _real_extract(self, url):
        region, lang, filter_type, display_id = re.search(self._VALID_URL, url).groups()
        if filter_type == 'episodes':
            filter_type = 'episode-videos'
        elif filter_type == 'live':
            filter_type = 'live-videos'
        regions = [region.upper()]
        if region != 'int':
            if region in self._LAT_FALLBACK_MAP:
                regions.append('LAT')
            if lang in self._INT_FALLBACK_LIST:
                regions.append('INT')
        locale = '>'.join(['%s-%s' % (lang, reg) for reg in regions])
        rrn_id = self._download_json(
            'https://www.redbull.com/v3/api/graphql/v1/v3/query/' + locale,
            display_id, query={
                'filter[type]': filter_type,
                'filter[uriSlug]': display_id,
                'rb3Schema': 'v1:hero',
            })['data']['id']
        return self.url_result(
-            'https://www.redbull.com/embed/' + rrn_id,
+            video_url, ie=RedBullTVIE.ie_key(),
-            RedBullEmbedIE.ie_key(), rrn_id)
+            video_id=RedBullTVIE._match_id(video_url))
--- a/youtube_dl/extractor/redtube.py
+++ b/youtube_dl/extractor/redtube.py
@ -4,7 +4,6 @@ import re
 from .common import InfoExtractor
 from ..utils import (
    determine_ext,
    ExtractorError,
    int_or_none,
    merge_dicts,
@ -15,7 +14,7 @@ from ..utils import (
 class RedTubeIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:\w+\.)?redtube\.com/|embed\.redtube\.com/\?.*?\bid=)(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://(?:(?:www\.)?redtube\.com/|embed\.redtube\.com/\?.*?\bid=)(?P<id>[0-9]+)'
    _TESTS = [{
        'url': 'http://www.redtube.com/66418',
        'md5': 'fc08071233725f26b8f014dba9590005',
@ -31,9 +30,6 @@ class RedTubeIE(InfoExtractor):
    }, {
        'url': 'http://embed.redtube.com/?bgcolor=000000&id=1443286',
        'only_matching': True,
    }, {
        'url': 'http://it.redtube.com/66418',
        'only_matching': True,
    }]
    @staticmethod
@ -47,21 +43,14 @@ class RedTubeIE(InfoExtractor):
        webpage = self._download_webpage(
            'http://www.redtube.com/%s' % video_id, video_id)
-        ERRORS = (
+        if any(s in webpage for s in ['video-deleted-info', '>This video has been removed']):
-            (('video-deleted-info', '>This video has been removed'), 'has been removed'),
+            raise ExtractorError('Video %s has been removed' % video_id, expected=True)
            (('private_video_text', '>This video is private', '>Send a friend request to its owner to be able to view it'), 'is private'),
        )
        for patterns, message in ERRORS:
            if any(p in webpage for p in patterns):
                raise ExtractorError(
                    'Video %s %s' % (video_id, message), expected=True)
        info = self._search_json_ld(webpage, video_id, default={})
        if not info.get('title'):
            info['title'] = self._html_search_regex(
-                (r'<h(\d)[^>]+class="(?:video_title_text|videoTitle|video_title)[^"]*">(?P<title>(?:(?!\1).)+)</h\1>',
+                (r'<h(\d)[^>]+class="(?:video_title_text|videoTitle)[^"]*">(?P<title>(?:(?!\1).)+)</h\1>',
                 r'(?:videoTitle|title)\s*:\s*(["\'])(?P<title>(?:(?!\1).)+)\1',),
                webpage, 'title', group='title',
                default=None) or self._og_search_title(webpage)
@ -81,7 +70,7 @@ class RedTubeIE(InfoExtractor):
                    })
        medias = self._parse_json(
            self._search_regex(
-                r'mediaDefinition["\']?\s*:\s*(\[.+?}\s*\])', webpage,
+                r'mediaDefinition\s*:\s*(\[.+?\])', webpage,
                'media definitions', default='{}'),
            video_id, fatal=False)
        if medias and isinstance(medias, list):
@ -89,12 +78,6 @@ class RedTubeIE(InfoExtractor):
                format_url = url_or_none(media.get('videoUrl'))
                if not format_url:
                    continue
                if media.get('format') == 'hls' or determine_ext(format_url) == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
                        format_url, video_id, 'mp4',
                        entry_protocol='m3u8_native', m3u8_id='hls',
                        fatal=False))
                    continue
                format_id = media.get('quality')
                formats.append({
                    'url': format_url,
--- a/Show more
+++ b/Show more