Toggle navigation
Toggle navigation
This project
Loading...
Sign in
graykode
/
commit-autosuggestions
Go to a project
Toggle navigation
Toggle navigation pinning
Projects
Groups
Snippets
Help
Project
Activity
Repository
Pipelines
Graphs
Issues
0
Merge Requests
0
Wiki
Snippets
Network
Create a new issue
Builds
Commits
Issue Boards
Authored by
graykode
2020-11-01 16:27:11 +0900
Browse Files
Options
Browse Files
Download
Email Patches
Plain Diff
Commit
3d6b29a66fec4fc6f0ee286e82935dbe009f2e9d
3d6b29a6
1 parent
f552b834
(fixed) blank space is added in code tokens
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
2 deletions
gitparser.py
gitparser.py
View file @
3d6b29a
...
...
@@ -59,7 +59,7 @@ def jobs(repo, args):
for
line
,
code
in
mod
.
diff_parsed
[
"deleted"
]:
deleted
.
extend
(
args
.
tokenizer
.
tokenize
(
code
))
if
len
(
added
)
+
len
(
deleted
)
<=
args
.
max_source_length
:
if
added
and
deleted
and
len
(
added
)
+
len
(
deleted
)
<=
args
.
max_source_length
-
3
:
with
jsonlines
.
open
(
args
.
output_file
,
mode
=
"a"
)
as
writer
:
writer
.
write
(
{
...
...
@@ -105,7 +105,7 @@ def main(args):
)
write_jsonl
(
data
[
int
(
n_data
*
0.9
):
int
(
n_data
*
0.95
)],
path
=
args
.
output_dir
,
mode
=
'valid
ation
'
path
=
args
.
output_dir
,
mode
=
'valid'
)
write_jsonl
(
data
[
int
(
n_data
*
0.95
):],
...
...
Please
register
or
login
to post a comment