Merge master, fix readme conflicts

This commit is contained in:
Chris Howell 2021-03-02 07:48:17 -08:00
commit baadc2f7f0
6 changed files with 507 additions and 1 deletions

247
README.md
View File

@ -310,3 +310,250 @@ youtube_transcript_api <first_video_id> <second_video_id> --cookies /path/to/you
If this project makes you happy by reducing your development time, you can make me happy by treating me to a cup of coffee :) If this project makes you happy by reducing your development time, you can make me happy by treating me to a cup of coffee :)
[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url) [![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url)
# YouTube Transcript/Subtitle API (including automatically generated subtitles and subtitle translations)
[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url) [![Build Status](https://travis-ci.com/jdepoix/youtube-transcript-api.svg)](https://travis-ci.com/jdepoix/youtube-transcript-api) [![Coverage Status](https://coveralls.io/repos/github/jdepoix/youtube-transcript-api/badge.svg?branch=master)](https://coveralls.io/github/jdepoix/youtube-transcript-api?branch=master) [![MIT license](http://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat)](http://opensource.org/licenses/MIT) [![image](https://img.shields.io/pypi/v/youtube-transcript-api.svg)](https://pypi.org/project/youtube-transcript-api/) [![image](https://img.shields.io/pypi/pyversions/youtube-transcript-api.svg)](https://pypi.org/project/youtube-transcript-api/)
This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles, supports translating subtitles and it does not require a headless browser, like other selenium based solutions do!
## Install
It is recommended to [install this module by using pip](https://pypi.org/project/youtube-transcript-api/):
```
pip install youtube_transcript_api
```
If you want to use it from source, you'll have to install the dependencies manually:
```
pip install -r requirements.txt
```
You can either integrate this module [into an existing application](#api), or just use it via an [CLI](#cli).
## API
The easiest way to get a transcript for a given video is to execute:
```python
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript(video_id)
```
This will return a list of dictionaries looking somewhat like this:
```python
[
{
'text': 'Hey there',
'start': 7.58,
'duration': 6.13
},
{
'text': 'how are you',
'start': 14.08,
'duration': 7.58
},
# ...
]
```
You can also add the `languages` param if you want to make sure the transcripts are retrieved in your desired language (it defaults to english).
```python
YouTubeTranscriptApi.get_transcripts(video_ids, languages=['de', 'en'])
```
It's a list of language codes in a descending priority. In this example it will first try to fetch the german transcript (`'de'`) and then fetch the english transcript (`'en'`) if it fails to do so. If you want to find out which languages are available first, [have a look at `list_transcripts()`](#list-available-transcripts)
To get transcripts for a list of video ids you can call:
```python
YouTubeTranscriptApi.get_transcripts(video_ids, languages=['de', 'en'])
```
`languages` also is optional here.
### List available transcripts
If you want to list all transcripts which are available for a given video you can call:
```python
transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)
```
This will return a `TranscriptList` object which is iterable and provides methods to filter the list of transcripts for specific languages and types, like:
```python
transcript = transcript_list.find_transcript(['de', 'en'])
```
By default this module always picks manually created transcripts over automatically created ones, if a transcript in the requested language is available both manually created and generated. The `TranscriptList` allows you to bypass this default behaviour by searching for specific transcript types:
```python
# filter for manually created transcripts
transcript = transcript_list.find_manually_created_transcript(['de', 'en'])
# or automatically generated ones
transcript = transcript_list.find_generated_transcript(['de', 'en'])
```
The methods `find_generated_transcript`, `find_manually_created_transcript`, `find_generated_transcript` return `Transcript` objects. They contain metadata regarding the transcript:
```python
print(
transcript.video_id,
transcript.language,
transcript.language_code,
# whether it has been manually created or generated by YouTube
transcript.is_generated,
# whether this transcript can be translated or not
transcript.is_translatable,
# a list of languages the transcript can be translated to
transcript.translation_languages,
)
```
and provide the method, which allows you to fetch the actual transcript data:
```python
transcript.fetch()
```
### Translate transcript
YouTube has a feature which allows you to automatically translate subtitles. This module also makes it possible to access this feature. To do so `Transcript` objects provide a `translate()` method, which returns a new translated `Transcript` object:
```python
transcript = transcript_list.find_transcript(['en'])
translated_transcript = transcript.translate('de')
print(translated_transcript.fetch())
```
### By example
```python
# retrieve the available transcripts
transcript_list = YouTubeTranscriptApi.list_transcripts('video_id')
# iterate over all available transcripts
for transcript in transcript_list:
# the Transcript object provides metadata properties
print(
transcript.video_id,
transcript.language,
transcript.language_code,
# whether it has been manually created or generated by YouTube
transcript.is_generated,
# whether this transcript can be translated or not
transcript.is_translatable,
# a list of languages the transcript can be translated to
transcript.translation_languages,
)
# fetch the actual transcript data
print(transcript.fetch())
# translating the transcript will return another transcript object
print(transcript.translate('en').fetch())
# you can also directly filter for the language you are looking for, using the transcript list
transcript = transcript_list.find_transcript(['de', 'en'])
# or just filter for manually created transcripts
transcript = transcript_list.find_manually_created_transcript(['de', 'en'])
# or automatically generated ones
transcript = transcript_list.find_generated_transcript(['de', 'en'])
```
## CLI
Execute the CLI script using the video ids as parameters and the results will be printed out to the command line:
```
youtube_transcript_api <first_video_id> <second_video_id> ...
```
The CLI also gives you the option to provide a list of preferred languages:
```
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en
```
You can also specify if you want to exclude automatically generated or manually created subtitles:
```
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --exclude-generated
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --exclude-manually-created
```
If you would prefer to write it into a file or pipe it into another application, you can also output the results as json using the following line:
```
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --json > transcripts.json
```
Translating transcripts using the CLI is also possible:
```
youtube_transcript_api <first_video_id> <second_video_id> ... --languages en --translate de
```
If you are not sure which languages are available for a given video you can call, to list all available transcripts:
```
youtube_transcript_api --list-transcripts <first_video_id>
```
## Proxy
You can specify a https/http proxy, which will be used during the requests to YouTube:
```python
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript(video_id, proxies={"http": "http://user:pass@domain:port", "https": "https://user:pass@domain:port"})
```
As the `proxies` dict is passed on to the `requests.get(...)` call, it follows the [format used by the requests library](http://docs.python-requests.org/en/master/user/advanced/#proxies).
Using the CLI:
```
youtube_transcript_api <first_video_id> <second_video_id> --http-proxy http://user:pass@domain:port --https-proxy https://user:pass@domain:port
```
## Cookies
Some videos are age restricted, so this module won't be able to access those videos without some sort of authentication. To do this, you will need to have access to the desired video in a browser. Then, you will need to download that pages cookies into a text file. You can use the Chrome extension [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg?hl=en) or the Firefox extension [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/).
Once you have that, you can use it with the module to access age-restricted videos' captions like so.
```python
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript(video_id, cookies='/path/to/your/cookies.txt')
YouTubeTranscriptApi.get_transcripts([video_id], cookies='/path/to/your/cookies.txt')
```
Using the CLI:
```
youtube_transcript_api <first_video_id> <second_video_id> --cookies /path/to/your/cookies.txt
```
## Warning
This code uses an undocumented part of the YouTube API, which is called by the YouTube web-client. So there is no guarantee that it won't stop working tomorrow, if they change how things work. I will however do my best to make things working again as soon as possible if that happens. So if it stops working, let me know!
## Donation
If this project makes you happy by reducing your development time, you can make me happy by treating me to a cup of coffee :)
[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url)

View File

@ -5,6 +5,7 @@ from ._errors import (
NoTranscriptFound, NoTranscriptFound,
CouldNotRetrieveTranscript, CouldNotRetrieveTranscript,
VideoUnavailable, VideoUnavailable,
TooManyRequests,
NotTranslatable, NotTranslatable,
TranslationLanguageNotAvailable, TranslationLanguageNotAvailable,
NoTranscriptAvailable, NoTranscriptAvailable,

View File

@ -37,7 +37,12 @@ class CouldNotRetrieveTranscript(Exception):
class VideoUnavailable(CouldNotRetrieveTranscript): class VideoUnavailable(CouldNotRetrieveTranscript):
CAUSE_MESSAGE = 'The video is no longer available' CAUSE_MESSAGE = 'The video is no longer available'
class TooManyRequests(CouldNotRetrieveTranscript):
CAUSE_MESSAGE = ("YouTube is receiving too many requests from this IP and now requires solving a captcha to continue. One of the following things can be done to work around this:\n\
- Manually solve the captcha in a browser and export the cookie. Read here how to use that cookie with youtube-transcript-api: https://github.com/jdepoix/youtube-transcript-api#cookies\n\
- Use a different IP address\n\
- Wait until the ban on your IP has been lifted")
class TranscriptsDisabled(CouldNotRetrieveTranscript): class TranscriptsDisabled(CouldNotRetrieveTranscript):
CAUSE_MESSAGE = 'Subtitles are disabled for this video' CAUSE_MESSAGE = 'Subtitles are disabled for this video'

View File

@ -14,6 +14,7 @@ import re
from ._html_unescaping import unescape from ._html_unescaping import unescape
from ._errors import ( from ._errors import (
VideoUnavailable, VideoUnavailable,
TooManyRequests,
NoTranscriptFound, NoTranscriptFound,
TranscriptsDisabled, TranscriptsDisabled,
NotTranslatable, NotTranslatable,
@ -38,6 +39,8 @@ class TranscriptListFetcher():
splitted_html = html.split('"captions":') splitted_html = html.split('"captions":')
if len(splitted_html) <= 1: if len(splitted_html) <= 1:
if 'class="g-recaptcha"' in html:
raise TooManyRequests(video_id)
if '"playabilityStatus":' not in html: if '"playabilityStatus":' not in html:
raise VideoUnavailable(video_id) raise VideoUnavailable(video_id)

View File

@ -0,0 +1,239 @@
<!DOCTYPE html>
<html>
<head>
<title>YouTube</title>
<script
src="https://www.google.com/recaptcha/api.js?hl=es"
async
defer
nonce="upPiqA/fwe4T7vXFBXo5Gw"
></script>
<style nonce="upPiqA/fwe4T7vXFBXo5Gw">
html {
height: 100%;
margin: 0;
padding: 0;
border: 0;
font-size: 100%;
background: transparent;
}
body {
margin: 0;
padding: 0;
border: 0;
font: 12px "YouTube Noto", Roboto, arial, sans-serif;
background: #f1f1f1;
height: 100%;
}
#captcha-page {
text-align: center;
height: 100%;
white-space: nowrap;
word-break: normal;
}
.g-recaptcha {
display: inline-block;
}
#captcha-page-content {
vertical-align: middle;
display: inline-block;
}
#captcha-page-vertical-align {
height: 100%;
display: inline-block;
vertical-align: middle;
}
p {
margin: auto;
color: #333;
font-size: 16px;
white-space: normal;
text-shadow: 0 0 0 transparent, 0 1px 1px #fff;
}
</style>
<link
rel="shortcut icon"
href="https://www.youtube.com/img/favicon.ico"
type="image/x-icon"
/>
<link
rel="icon"
href="https://www.youtube.com/img/favicon_32.png"
sizes="32x32"
/>
<link
rel="icon"
href="https://www.youtube.com/img/favicon_48.png"
sizes="48x48"
/>
<link
rel="icon"
href="https://www.youtube.com/img/favicon_96.png"
sizes="96x96"
/>
<link
rel="icon"
href="https://www.youtube.com/img/favicon_144.png"
sizes="144x144"
/>
</head>
<body>
<div id="captcha-page">
<div id="captcha-page-content">
<p>
Perdón por la interrupción. Hemos recibido un gran número de
solicitudes de tu red.
</p>
<p>
Para seguir disfrutando de YouTube, rellena el siguiente formulario.
</p>
<form action="/das_captcha?fw=1" method="POST">
<div
class="g-recaptcha"
data-sitekey="6Lf39AMTAAAAALPbLZdcrWDa8Ygmgk_fmGmrlRog"
></div>
<br /><input
type="hidden"
name="action_recaptcha_verify2"
value="1"
/><input
type="hidden"
name="next"
value="/watch?v=GJLlxj_dtq8"
/><input type="submit" value="Submit" />
</form>
<style nonce="upPiqA/fwe4T7vXFBXo5Gw">
#yt-masthead {
margin: 15px auto;
width: 440px;
margin-top: 25px;
}
#logo-container {
margin-right: 5px;
float: left;
cursor: pointer;
text-decoration: none;
}
.logo {
background: no-repeat
url("//www.gstatic.com/youtube/img/branding/youtubelogo/1x/youtubelogo_30.png");
width: 125px;
height: 30px;
cursor: pointer;
display: inline-block;
}
#masthead-search {
display: flex;
margin-top: 3px;
max-width: 650px;
overflow: hidden;
padding: 0;
position: relative;
}
.search-button {
border-left: 0;
-moz-border-radius-topleft: 0;
border-top-left-radius: 0;
-moz-border-radius-bottomleft: 0;
border-bottom-left-radius: 0;
float: right;
height: 29px;
padding: 0;
border: solid 1px transparent;
border-color: #d3d3d3;
background: #f8f8f8;
color: #333;
cursor: pointer;
}
.search-button:hover {
border-color: #c6c6c6;
background: #f0f0f0;
box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1);
}
.search-button-content {
border: none;
display: block;
opacity: 0.6;
padding: 0;
text-indent: -10000px;
background: no-repeat
url(//www.gstatic.com/youtube/src/web/htdocs/img/search.png);
background-size: auto;
width: 15px;
height: 15px;
box-shadow: none;
margin: 0 25px;
}
#masthead-search-terms-border {
flex: 1 1 auto;
border: 1px solid #ccc;
box-shadow: inset 0 1px 2px #eee;
background-color: #fff;
font-size: 14px;
height: 29px;
line-height: 30px;
margin: 0 0 2px;
overflow: hidden;
position: relative;
-moz-box-sizing: border-box;
box-sizing: border-box;
-webkit-transition: border-color 0.2s ease;
transition: border-color 0.2s ease;
}
#masthead-search-terms {
background: transparent;
border: 0;
font-size: 16px;
height: 100%;
left: 0;
margin: 0;
outline: none;
padding: 2px 6px;
position: absolute;
width: 100%;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
</style>
<div id="yt-masthead">
<a
id="logo-container"
href="https://www.youtube.com/"
title="Página de inicio de YouTube"
><span class="logo" title="Página de inicio de YouTube"></span
><span class="content-region">ES</span></a
>
<form
id="masthead-search"
class="search-form"
action="https://www.youtube.com/results"
onsubmit="if (document.getElementById('masthead-search-terms').value == '') return false;"
>
<div id="masthead-search-terms-border" dir="ltr">
<input
id="masthead-search-terms"
autocomplete="off"
onkeydown="if (!this.value &amp;&amp; (event.keyCode == 40 || event.keyCode == 32 || event.keyCode == 34)) {this.onkeydown = null; this.blur();}"
name="search_query"
value=""
type="text"
placeholder="Buscar"
title="Buscar"
aria-label="Buscar"
/>
</div>
<button
class="search-button"
type="submit"
onclick="if (document.getElementById('masthead-search-terms').value == '') return false; document.getElementById('masthead-search').submit(); return false;"
dir="ltr"
>
<span class="search-button-content">Buscar</span>
</button>
</form>
</div>
</div>
<span id="captcha-page-vertical-align"></span>
</div>
</body>
</html>

View File

@ -12,6 +12,7 @@ from youtube_transcript_api import (
TranscriptsDisabled, TranscriptsDisabled,
NoTranscriptFound, NoTranscriptFound,
VideoUnavailable, VideoUnavailable,
TooManyRequests,
NoTranscriptAvailable, NoTranscriptAvailable,
NotTranslatable, NotTranslatable,
TranslationLanguageNotAvailable, TranslationLanguageNotAvailable,
@ -134,6 +135,16 @@ class TestYouTubeTranscriptApi(TestCase):
with self.assertRaises(VideoUnavailable): with self.assertRaises(VideoUnavailable):
YouTubeTranscriptApi.get_transcript('abc') YouTubeTranscriptApi.get_transcript('abc')
def test_get_transcript__exception_if_youtube_request_limit_reached(self):
httpretty.register_uri(
httpretty.GET,
'https://www.youtube.com/watch',
body=load_asset('youtube_too_many_requests.html.static')
)
with self.assertRaises(TooManyRequests):
YouTubeTranscriptApi.get_transcript('abc')
def test_get_transcript__exception_if_transcripts_disabled(self): def test_get_transcript__exception_if_transcripts_disabled(self):
httpretty.register_uri( httpretty.register_uri(
httpretty.GET, httpretty.GET,