Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
V
VenJob
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Mai Hoang Thai Ha
VenJob
Commits
9a32f409
Commit
9a32f409
authored
Jul 19, 2021
by
Mai Hoang Thai Ha
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fixed ARGV
parent
8ab617be
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
15 additions
and
25 deletions
+15
-25
lib/tasks/web_crawler.rake
+15
-25
No files found.
lib/tasks/web_crawler.rake
View file @
9a32f409
require
'open-uri'
require
'csv'
require
'zip'
require
"rails/all"
namespace
:crawler
do
desc
'crawler from CareerBuilder'
task
jobs: :environment
do
ARGV
.
each
{
|
a
|
task
a
.
to_sym
{
;
}
}
total_pages
=
0
if
ARGV
.
length
<=
1
ARGV
.
each
do
|
a
|
task
a
.
to_sym
{
;
}
case
a
when
'TEST'
total_pages
=
1
when
'ALL'
first_page
=
Nokogiri
::
HTML
(
HTTParty
.
get
(
'https://careerbuilder.vn/viec-lam/tat-ca-viec-lam-vi.html'
).
body
)
jobs_per_page
=
first_page
.
css
(
'div.job-item'
).
count
total_jobs
=
first_page
.
css
(
'.search-result-list .job-found p'
).
text
.
split
(
' '
).
first
.
gsub
(
','
,
''
).
to_i
total_pages
=
(
total_jobs
.
to_f
/
jobs_per_page
).
round
else
exit
end
end
elsif
# exit
if
ARGV
.
length
==
1
&&
ARGV
[
0
]
==
'TEST'
total_pages
=
1
elsif
ARGV
.
length
==
1
&&
ARGV
[
0
]
==
'ALL'
first_page
=
Nokogiri
::
HTML
(
HTTParty
.
get
(
'https://careerbuilder.vn/viec-lam/tat-ca-viec-lam-vi.html'
).
body
)
jobs_per_page
=
first_page
.
css
(
'div.job-item'
).
count
total_jobs
=
first_page
.
css
(
'.search-result-list .job-found p'
).
text
.
split
(
' '
).
first
.
gsub
(
','
,
''
).
to_i
total_pages
=
(
total_jobs
.
to_f
/
jobs_per_page
).
round
else
exit
end
(
1
..
total_pages
).
each
do
|
page
|
...
...
@@ -38,14 +28,13 @@ namespace :crawler do
# title - company
title
=
job_page
.
css
(
'div.job-desc h1.title'
).
text
company
=
job_page
.
css
(
'div.job-desc a.job-company-name'
).
text
Company
.
create!
(
name:
company
)
# info box
info_box_item
=
job_detail
.
css
(
'.detail-box ul li'
)
# city, update_at, industry, type, salary, experience, level, expiration_date
job_industries
=
[]
update_at
,
job_type
,
salary
,
experience
,
level
,
expiration_date
=
''
job_cities
=
[]
job_detail
.
css
(
'.detail-box .map a'
).
each
do
|
part
|
job_detail
.
css
(
'.detail-box .map
p
a'
).
each
do
|
part
|
city
=
part
.
text
job_cities
<<
city
end
...
...
@@ -96,6 +85,7 @@ namespace :crawler do
other_info_list
<<
info
end
company
=
Company
.
find_or_create_by
(
name:
company
)
job
=
Job
.
find_or_create_by
(
title:
title
,
job_type:
job_type
,
...
...
@@ -108,7 +98,7 @@ namespace :crawler do
requirement:
requirement
,
other_info:
other_info_list
.
each
{
|
info
|
}
)
Company
.
find_or_initialize_by
(
name:
company
)
.
jobs
<<
job
company
.
jobs
<<
job
job_industries
.
each
do
|
industry
|
industry_id
=
Industry
.
find_or_create_by
(
name:
industry
)
job
.
industries
<<
industry_id
...
...
@@ -132,7 +122,7 @@ namespace :crawler do
industry_list
<<
industry
end
industry_list
.
each
do
|
industry
|
Industry
.
create
!
(
name:
industry
)
Industry
.
create
(
name:
industry
)
end
end
...
...
@@ -156,7 +146,7 @@ namespace :crawler do
city_list
<<
city
end
city_list
.
each
do
|
city
|
City
.
create
!
(
City
.
create
(
name:
city
[
:name
],
region:
city
[
:region
]
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment