yogsototh/clj-ml
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Filter out junk from text fields.

Browse source
This commit is contained in:
Joshua Eckroth 2013-07-11 00:24:24 -04:00
parent 8b53ee681c
commit 0da42ca0ea
1 changed files with 3 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
src/clj_ml/data.clj
View file

@ -523,8 +523,9 @@ split immediately you can use do-split-dataset."
(for [doc docs-keep-n]
(let [fulltext (str/replace (.text (Jsoup/parse (or (:fulltext doc) (:extracted doc) "")))
#"\s+" " ")
fulltext-sub (subs fulltext 0 (min (count fulltext) 10000))
title (:title doc "")
fulltext-sub (str/replace (subs fulltext 0 (min (count fulltext) 10000))
#"[^ \w\d]" "")
title (str/replace (:title doc "") #"[^ \w\d]" "")
has-tid? (some #(= term-name %) (get-in doc [:terms vocab-name]))]
[(if has-tid? :yes :no) title fulltext-sub])))
ds-title (let [f (filters/make-filter