Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Development #21

Merged
merged 2 commits into from
Nov 23, 2023
Merged

Development #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8abf133
web log control
lizongying Nov 23, 2023
f3b1ed8
remove query
lizongying Nov 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README_CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1111,8 +1111,8 @@ curl https://github.com/lizongying/go-crawler -x http://localhost:8082 --cacert
* extra
* request context
* total
* log
* status
* cdp

```shell
go get -u github.com/lizongying/go-css@latest
Expand Down
68 changes: 35 additions & 33 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,35 +3,34 @@ module github.com/lizongying/go-crawler
go 1.21.0

require (
github.com/andybalholm/brotli v1.0.4
github.com/aws/aws-sdk-go-v2 v1.18.1
github.com/aws/aws-sdk-go-v2/credentials v1.13.26
github.com/aws/aws-sdk-go-v2/service/s3 v1.36.0
github.com/go-rod/rod v0.114.2
github.com/andybalholm/brotli v1.0.6
github.com/aws/aws-sdk-go-v2 v1.23.1
github.com/aws/aws-sdk-go-v2/credentials v1.16.4
github.com/aws/aws-sdk-go-v2/service/s3 v1.44.0
github.com/go-rod/rod v0.114.5
github.com/go-rod/stealth v0.4.9
github.com/go-sql-driver/mysql v1.7.1
github.com/google/uuid v1.3.0
github.com/google/uuid v1.4.0
github.com/lizongying/cron v1.0.0
github.com/lizongying/go-css v0.0.0-20230906104236-b58497123e0f
github.com/lizongying/go-gua64 v0.1.0
github.com/lizongying/go-json v0.0.0-20230906102344-7915011b6304
github.com/lizongying/go-query v0.0.0-20230720084938-e0776701ba5f
github.com/lizongying/go-re v0.0.0-20230906094757-1b48356fa6db
github.com/lizongying/go-uid v0.0.0-20231109082237-2794e92dcc84
github.com/lizongying/go-xpath v0.0.0-20230906104639-f46eb0e080c4
github.com/mattn/go-sqlite3 v1.14.17
github.com/mattn/go-sqlite3 v1.14.18
github.com/pkg/errors v0.9.1
github.com/redis/go-redis/v9 v9.0.5
github.com/refraction-networking/utls v1.3.2
github.com/segmentio/kafka-go v0.4.40
github.com/stretchr/testify v1.8.1
github.com/redis/go-redis/v9 v9.3.0
github.com/refraction-networking/utls v1.5.4
github.com/segmentio/kafka-go v0.4.45
github.com/stretchr/testify v1.8.4
github.com/temoto/robotstxt v1.1.2
github.com/tidwall/gjson v1.16.0
go.mongodb.org/mongo-driver v1.11.3
go.uber.org/fx v1.19.2
golang.org/x/crypto v0.12.0
golang.org/x/text v0.13.0
golang.org/x/time v0.3.0
github.com/tidwall/gjson v1.17.0
go.mongodb.org/mongo-driver v1.13.0
go.uber.org/fx v1.20.1
golang.org/x/crypto v0.15.0
golang.org/x/text v0.14.0
golang.org/x/time v0.4.0
gopkg.in/yaml.v3 v3.0.1
)

Expand All @@ -40,25 +39,28 @@ require (
github.com/andybalholm/cascadia v1.3.1 // indirect
github.com/antchfx/htmlquery v1.3.0 // indirect
github.com/antchfx/xpath v1.2.4 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 // indirect
github.com/aws/smithy-go v1.13.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.4 // indirect
github.com/aws/smithy-go v1.17.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cloudflare/circl v1.3.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/gaukas/godicttls v0.0.3 // indirect
github.com/gaukas/godicttls v0.0.4 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.1 // indirect
github.com/klauspost/compress v1.15.15 // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/quic-go/quic-go v0.37.4 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
Expand All @@ -71,10 +73,10 @@ require (
github.com/ysmood/gson v0.7.3 // indirect
github.com/ysmood/leakless v0.8.0 // indirect
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/dig v1.16.1 // indirect
go.uber.org/dig v1.17.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
go.uber.org/zap v1.23.0 // indirect
golang.org/x/net v0.14.0 // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/sys v0.11.0 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sync v0.2.0 // indirect
golang.org/x/sys v0.14.0 // indirect
)
Loading