关于TikTok Scraper

TikTok Scraper是一款针对TikTok的数据收集工具，该工具可以帮助广大用户从TikTok快速收集和下载各种有用的信息，其中包括视频、趋势、标签、音乐、feed和URL等元数据。
值得一提的是，作为一个纯数据爬取工具，该工具不需要进行登录或设置密码，因为TikTok Scraper使用了TikTok Web API来收集媒体信息和相关元数据。

注意：当前版本的TikTok Scraper不支持无水印下载视频。

功能介绍

1、从用户、标签、趋势或音乐Id页面下载的帖子元数据（不限量）
2、将帖子元数据存储为JSON/CSV文件
3、下载媒体数据，并保存为ZIP文件
4、从CLI下载单个视频文件
5、使用签名URL向TikTok API发送自定义请求
6、从用户、标签和单个视频页面提取元数据
7、保存之前的爬取进度，只下载以前没有下载过的新视频。此功能仅在CLI中有效，并且仅在“下载”标志处于启在CLI中查看和管理以前下载的帖子历史记录用状态时有效。
8、在CLI中查看和管理以前下载的帖子历史记录
9、以批处理模式爬取并下载文件中指定的用户、标签、音乐feed和单个视频

工具安装

TikTok Scraper要求本地设备安装并配置好Node.js v10+环境。

通过NPM安装

npm i -g tiktok-scraper

通过YARN安装

yarn global add tiktok-scraper

工具使用

命令行终端

$ tiktok-scraper --help

Usage: tiktok-scraper <command> [options]

Commands:
  tiktok-scraper user [id]     Scrape videos from username. Enter only username
  tiktok-scraper hashtag [id]  Scrape videos from hashtag. Enter hashtag without #
  tiktok-scraper trend         Scrape posts from current trends
  tiktok-scraper music [id]    Scrape posts from a music id number
  tiktok-scraper history       View previous download history
  tiktok-scraper from-file [file] [async]  Scrape users, hashtags, music, videos mentioned
                                in a file. 1 value per 1 line

Options:
  --version            Show version number                             [boolean]
  --session            Set session cookie value. Sometimes session can be
                       helpful when scraping data from any method  [default: ""]
  --session-file       Set path to the file with list of active sessions. One
                       session per line!                           [default: ""]
  --timeout            Set timeout between requests. Timeout is in Milliseconds:
                       1000 mls = 1 s                               [default: 0]
  --number, -n         Number of posts to scrape. If you will set 0 then all
                       posts will be scraped                        [default: 0]
  --since              Scrape no posts published before this date (timestamp).
                       If set to 0 the filter is deactived          [default: 0]
  --proxy, -p          Set single proxy                            [default: ""]
  --proxy-file         Use proxies from a file. Scraper will use random proxies
                       from the file per each request. 1 line 1 proxy.
                                                                   [default: ""]
  --download, -d       Download video posts to the folder with the name input
                       [id]                           [boolean] [default: false]
  --asyncDownload, -a  Number of concurrent downloads               [default: 5]
  --hd                 Download video in HD. Video size will be x5-x10 times
                       larger and this will affect scraper execution speed. This
                       option only works in combination with -w flag
                                                      [boolean] [default: false]
  --zip, -z            ZIP all downloaded video posts [boolean] [default: false]
  --filepath           File path to save all output files.
      [default: "/Users/karl.wint/Documents/projects/javascript/tiktok-scraper"]
  --filetype, -t       Type of the output file where post information will be
                       saved. 'all' - save information about all posts to the`
                       'json' and 'csv'
                               [choices: "csv", "json", "all", ""] [default: ""]
  --filename, -f       Set custom filename for the output files    [default: ""]
  --store, -s          Scraper will save the progress in the OS TMP or Custom
                       folder and in the future usage will only download new
                       videos avoiding duplicates     [boolean] [default: false]
  --historypath        Set custom path where history file/files will be stored
                   [default: "/var/folders/d5/fyh1_f2926q7c65g7skc0qh80000gn/T"]
  --remove, -r         Delete the history record by entering "TYPE:INPUT" or
                       "all" to clean all the history. For example: user:bob
                                                                   [default: ""]
  --webHookUrl         Set webhook url to receive scraper result as HTTP
                       requests. For example to your own API       [default: ""]
  --method             Receive data to your webhook url as POST or GET request
                                      [choices: "GET", "POST"] [default: "POST"]
  --help               Show help                                       [boolean]

工具使用演示

tiktok-scraper user USERNAME -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper trend -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper hashtag HASHTAG_NAME -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper music MUSIC_ID -d -n 50 --session sid_tt=dae32131231
  tiktok-scraper video https://www.tiktok.com/@tiktok/video/6807491984882765062 -d
  tiktok-scraper history
  tiktok-scraper history -r user:bob
  tiktok-scraper history -r all
  tiktok-scraper from-file BATCH_FILE ASYNC_TASKS -d

工具使用样例

命令行终端使用

从用户{USERNAME}爬取300条视频帖子，并将帖子元数据保存至CSV（-t csv）文件中：

tiktok-scraper user USERNAME -n 300 -t csv --session sid_tt=asdasd13123123123adasda

Output:
CSV path: /bla/blah/USERNAME_1552945544582.csv

从哈希标签{HASHTAG_NAME}爬取100条帖子数据，下载（-d）并保存为ZIP（-z）文档，将帖子元数据保存为JSON和CSV文件（-t all）：

tiktok-scraper hashtag HASHTAG_NAME -n 100 -d -z -t all  --session sid_tt=asdasd13123123123adasda

Output:
ZIP path: /bla/blah/HASHTAG_NAME_1552945659138.zip
JSON path: /bla/blah/HASHTAG_NAME_1552945659138.json
CSV path: /bla/blah/HASHTAG_NAME_1552945659138.csv

从趋势部分爬取50条帖子，下载（-d）并保存为ZIP（-z）文档，将帖子元数据保存为CSV文件（-t csv）：

tiktok-scraper trend -n 50 -d -z -t csv --session sid_tt=asdasd13123123123adasda


Output:
ZIP path: /bla/blah/trend_1552945659138.zip
CSV path: /bla/blah/tend_1552945659138.csv

从指定音乐ID爬取100条帖子，下载（-d）并保存为ZIP（-z）文档，将帖子元数据保存为CSV文件（-t csv）：

tiktok-scraper music MUSICID -n 100 -d -z -t csv --session sid_tt=asdasd13123123123adasda

Output:
ZIP path: /bla/blah/music_1552945659138.zip
CSV path: /bla/blah/music_1552945659138.csv

从用户{USERNAME}下载最新的20条发布视频，保存下载进度防止重复下载：

tiktok-scraper user USERNAME -n 20 -d -s --session sid_tt=asdasd13123123123adasda


Output:
Folder Path: /User/Bob/Downloads/USERNAME

管理下载历史

我们可以通过命令行工具查看历史数据：

tiktok-scraper history

删除单条历史记录：

tiktok-scraper history -r TYPE:INPUT
tiktok-scraper history -r user:tiktok
tiktok-scraper history -r hashtag:summer
tiktok-scraper history -r trend

设置好自定义路径后，每次运行你都需要告诉工具文件路径：

tiktok-scraper hashtag summer -s -d -n 10 --historypath /Blah/Blah/Blah

删除所有的记录：：

tiktok-scraper history -r all

批处理爬取并下载内容

## User feed by username <---- this is just a comment and hence it is not important
tiktok
charlidamelio
sam
bob

## User feed by user id
id:12312312312

## Hashtag feed
#love
#summer
#story

## Music feed
music:3242234234
music:46646
music:23423424234

## Single Videos. Each video will be downloaded without the watermark
https://www.tiktok.com/@shalisavdlaan/video/6788483055796391173
https://www.tiktok.com/@officialsaarx/video/6785180623263911174
https://www.tiktok.com/@dominos_nl/video/6786669305623842053
https://www.tiktok.com/@jessiejikki/video/6620697278451551493
https://www.tiktok.com/@.one_man_army/video/6798822211307310338

输出文件样例

JSON输出样例

视频Feed

{
    headers: {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36',
        referer: 'https://www.tiktok.com/',
        cookie: 'tt_webid_v2=689854141086886123'
    },
    collector:[{
        id: 'VIDEO_ID',
        text: 'CAPTION',
        createTime: '1583870600',
        authorMeta:{
            id: 'USER ID',
            name: 'USERNAME',
            following: 195,
            fans: 43500,
            heart: '1093998',
            video: 3,
            digg: 95,
            verified: false,
            private: false,
            signature: 'USER BIO',
            avatar:'AVATAR_URL'
        },
        musicMeta:{
            musicId: '6808098113188120838',
            musicName: 'blah blah',
            musicAuthor: 'blah',
            musicOriginal: true,
            playUrl: 'SOUND/MUSIC_URL',
        },
        covers:{
            default: 'COVER_URL',
            origin: 'COVER_URL',
            dynamic: 'COVER_URL'
        },
        imageUrl:'IMAGE_URL',
        videoUrl:'VIDEO_URL',
        videoUrlNoWaterMark:'VIDEO_URL_WITHOUT_THE_WATERMARK',
        videoMeta: { width: 480, height: 864, ratio: 14, duration: 14 },
        diggCount: 2104,
        shareCount: 1,
        playCount: 9007,
        commentCount: 50,
        mentions: ['@bob', '@sam', '@bob_again', '@and_sam_again'],
        hashtags:
        [{
            id: '69573911',
            name: 'PlayWithLife',
            title: 'HASHTAG_TITLE',
            cover: [Array]
        }...],
        downloaded: true
    }...],
    //If {filetype} and {download} options are enbabled then:
    zip: '/{CURRENT_PATH}/user_1552963581094.zip',
    json: '/{CURRENT_PATH}/user_1552963581094.json',
    csv: '/{CURRENT_PATH}/user_1552963581094.csv'
}

getUserProfileInfo

{
    secUid: 'MS4wLjABAAAAv7iSuuXDJGDvJkmH_vz1qkDZYo1apxgzaxdBSeIuPiM',
    userId: '107955',
    isSecret: false,
    uniqueId: 'tiktok',
    nickName: 'TikTok',
    signature: 'Make Your Day',
    covers: ['COVER_URL'],
    coversMedium: ['COVER_URL'],
    following: 490,
    fans: 38040567,
    heart: '211522962',
    video: 93,
    verified: true,
    digg: 29,
}

getHashtagInfo

{
    challengeId: '4231',
    challengeName: 'love',
    text: '',
    covers: [],
    coversMedium: [],
    posts: 66904972,
    views: '194557706433',
    isCommerce: false,
    splitTitle: ''
}

getVideoMeta

{
    headers: {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36',
        referer: 'https://www.tiktok.com/',
        cookie: 'tt_webid_v2=689854141086886123'
    },
    collector:[{
        id: '6807491984882765062',
        text: 'We’re kicking off the #happyathome live stream series today at 5pm PT!',
        createTime: '1584992742',
        authorMeta: { id: '6812221792183403526', name: 'blah' },
        musicMeta:{
            musicId: '6822233276137213677',
            musicName: 'blah',
            musicAuthor: 'blah'
        },
        imageUrl: 'IMAGE_URL',
        videoUrl: 'VIDEO_URL',
        videoUrlNoWaterMark: 'VIDEO_URL_WITHOUT_THE_WATERMARK',
        videoMeta: { width: 480, height: 864, ratio: 14, duration: 14 },
        covers:{
            default: 'COVER_URL',
            origin: 'COVER_URL'
        },
        diggCount: 49292,
        shareCount: 339,
        playCount: 614678,
        commentCount: 4023,
        downloaded: false,
        hashtags: [],
    }]
}

getMusicInfo

{
    music: {
        id: '6882925279036066566',
        title: 'doja x calabria',
        playUrl: 'dfdfdfdf',
        coverThumb:
            'dfdfdf',
        coverMedium:
            'dfdfdf',
        coverLarge:
            'fdfdf',
        authorName: 'bryce',
        original: true,
        playToken:
            'ffdfdf',
        keyToken: 'dfdfdfd',
        audioURLWithcookie: false,
        private: false,
        duration: 46,
        album: '',
    },
    author: {
        id: '6835300004094166021',
        uniqueId: 'mashupsbybryce',
        nickname: 'bryce',
        avatarThumb:
            'dfdfd',
        avatarMedium:
            'dfdfdf',
        avatarLarger:
            'dfdfdf',
        signature: 'hi ily :)\n70k sounds cool tbh\nfollow my soundcloud & insta',
        verified: false,
        secUid: 'MS4wLjABAAAA1_5bjLAamayD4rv3q49qJGa_7dZ5jzExTO0ozOybqIwwhw5TAg_iM25lkO94DM3K',
        secret: false,
        ftc: false,
        relation: 0,
        openFavorite: false,
        commentSetting: 0,
        duetSetting: 0,
        stitchSetting: 0,
        privateAccount: false,
    },
    stats: { videoCount: 361700 },
    shareMeta: {
        title: 'bryceyouloser | ♬ doja x calabria | on TikTok',
        desc: '361.0k videos - Watch awesome short ' + 'videos created with ♬ doja x calabria',
    },
};

许可证协议

本项目的开发与发布遵循MIT开源许可证协议。

项目地址

TikTok Scraper：【GitHub传送门】

参考资料

https://www.tiktok.com/
https://nodejs.org/
https://github.com/drawrowfly/tiktok-scraper/tree/master/examples/CLI/Examples.md
https://github.com/drawrowfly/tiktok-scraper/tree/master/examples/CLI/DownloadHistory.md
https://github.com/drawrowfly/tiktok-scraper/tree/master/examples/CLI/BatchDownload.md