Блог

Я пытаюсь использовать этот подход

 library(quanteda)

dataset1 <- data.frame( anumber = c(1,2,3), text = c("Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.","It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum", "Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source."))

myDfm <- dataset1 %>%
corpus() %>%
tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
dfm()%>%                         


   dfm_trim(min_termfreq = 1)
        
tstat_dist <- textstat_simil(myDfm, method = "cosine")

# hiarchical clustering the distance object
pres_cluster <- hclust(as.dist(tstat_dist))
# label with document names
pres_cluster$labels <- docnames(myDfm)
# plot as a dendrogram
plot(pres_cluster, xlab = "", sub = "", main = "Cosine Distance on Token Frequency")
  

извлечь словарный кластер для слов, но на конечном графике я получаю название документа, которое соответствует каждой имеющейся у меня строке. Возможно ли внести какие-либо изменения, чтобы получать слова текста, а не названия документов в кластере?

Я ожидал бы увидеть эти слова:

 textstat_frequency(myDfm, n = 5)
  

   feature frequency rank docfreq group
1     the        10    1       3   all
2      of         7    2       3   all
3   lorem         6    3       3   all
4   ipsum         6    3       3   all
5       a         5    5       2   all
  

Да — вам нужен margin = "features" аргумент при вычислении расстояния. (И вы можете удалить назначение меток.) Итак, последняя часть вашего кода должна быть:

 # compute the distance on features, not documents
tstat_dist <- textstat_simil(myDfm, method = "cosine", margin = "features")
# hiarchical clustering the distance object
pres_cluster <- hclust(as.dist(tstat_dist))
# plot as a dendrogram
plot(pres_cluster, xlab = "", sub = "", main = "Cosine Distance on Token Frequency")
  

Однако для вычисления иерархической кластеризации вам следует вычислять метрику расстояния, а не косинусное подобие.