강희주 / Music_Recommendation_Website

node-crawler ChangeLog

1.3.0

• #367 add http2 functionality (@BeijingProtoHuman)
• #364 Fix some typos (@pzmarzly)
• #363 Remove stale vendored jQuery version (@pzmarzly)

1.2.2

• #353 Release automate (@mike442144)
• #338 #comment Adding support for Https socks5. Agent is imported directly … (@djpavlovic)
• #336 Update README.md (@DanielHabenicht)
• #329 add support for removeRefererHeader request option to preserve referer during redirects (@petskratt)
• #314 docs: fix typo (@Jason-Cooke)

1.2.1

• #310 Upgrade dependencies' version(@mike442144)
• #303 Update seenreq to v3(@mike442144)
• #304 Replacement of istanbul with nyc (@kossidts)
• #300 Add formData arg to requestArgs (@humandevmode)
• #280 20180611 updatetestwithnock (@Dong-Gao)

1.2.0

• #278 Added filestream require to download section (@swosko)
• Use nock to mock testing instead of httpbin
• Replace jshint by eslint
• Fix code to pass eslint rules

1.1.4

• Tolerate incorrect Content-Type header #270, #193
• Added examples #272, 267
• Fixed "skipDuplicates" and "retries" config incompatible bug #261
• Fix typo in README #268

1.1.3

• Upgraded request.js and lodash

1.1.2

• Recognize all XML MIME types to inject jQuery #245
• Allow options to specify the Agent for Request #246
• Added logo

1.1.1

• added a way to replace the global options.headers keys by queuing options.headers #241
• fix bug of using last jar object if current options doesn't contain jar option #240
• fix bug of encoding #233
• added seenreq options #208
• added preRequest, setLimiterProperty, direct request functions

1.0.5

• fix missing debugging messages #213
• fix bug of 'drain' never called #210

1.0.4

• fix bug of charset detecting #203
• keep node version up to date in travis scripts

1.0.3

• fix bug, skipDuplicate and rotateUA don't work even if set true

1.0.0

• upgrade jsdom up to 9.6.x
• remove 0.10 and 0.12 support #170
• control dependencies version using ^ and ~ #169
• remove node-pool
• notify bottleneck until a task is completed
• replace bottleneck by bottleneckp, which has priority
• change default log function
• use event listener on request and drain instead of global function #144
• default set forceUTF8 to true
• detect ESOCKETTIMEDOUT instead of ETIMEDOUT when timeout in test
• add done function in callback to avoid async trap
• do not convert response body to string if encoding is null #118
• add result document #68 #116
• add event schedule which is emitted when a task is being added to scheduler
• in callback, move $ into res because of weird API
• change rateLimits to rateLimit

0.7.5

• delete entity in options before copy, and assgin after, jar is one of the typical properties which is an Entity wich functions #177
• upgrade request to version 2.74.0

0.7.4

• change debug option to instance level instead of options
• update README.md to detail error handling
• call onDrain with scope of this
• upgrade seenreq version to 0.1.7

0.7.0

• cancel recursion in queue
• upgrade request version to v2.67.0

0.6.9

• use bottleneckConcurrent instead of maxConnections, default 10000
• add debug info

0.6.5

• fix a deep and big bug when initializing Pool, that may lead to sequence execution. #2
• print log of Pool status

0.6.3

• you could also get result.options from callback even when some errors ouccurred #127 #86
• add test for bottleneck

0.6.0

• add bottleneck to implement rate limit, one can set limit for each connection at same time.

0.5.2

• you can manually terminate all the resources in your pool, when onDrain called, before their timeouts have been reached
• add a read-only property queueSize to crawler #148 #76 #107

0.5.1

• remove cache feature, it's useless
• add localAddress, time, tunnel, proxyHeaderWhiteList, proxyHeaderExclusiveList properties to pass to request #155

0.5.0

• parse charset from content-type in http headers or meta tag in html, then convert
• big5 charset is avaliable as the iconv-lite has already supported it
• default enable gzip in request header
• remove unzip code in crawler since request will do this
• body will return as a Buffer if encoding is null which is an option in request
• remove cache and skip duplicate request for GET, POST(only for type urlencode), HEAD
• add log feature, you can use winston to set logger:winston, or crawler will output to console
• rotate user-agent in case some sites ban your requests