--- title: "Author keywords and references" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Author keywords and references} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") # Render data frames and tibbles as tables rather than console text. local({ kp <- function(x, ...) { if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x)) knitr::knit_print(knitr::kable(x)) } for (cls in c("data.frame", "tbl_df", "tbl")) { registerS3method("knit_print", cls, kp, envir = asNamespace("knitr")) } }) ``` Keyword co-occurrence and citation-network analysis both need something the Search API alone does not return: a document's author-supplied keywords, and its own reference list. This walks through retrieving both, what each costs, and what your 'Scopus' entitlement needs to cover. The API calls below are shown exactly as you would run them with a configured key. No key is available when the site and the package are built, and Elsevier's terms do not permit redistributing retrieved records in any case, so each call is paired with its output assembled offline from one representative example, the 2015 *Nature* review "Deep learning". The corpus of real articles that the other articles run on carries neither author keywords nor reference lists, the two fields this one is about, so it cannot stand in here. The vignette therefore renders the same with or without a key, and `scopus_has_key()` is the switch you would use to run the calls for real. ```{r setup} library(scopusflow) ``` ## Author keywords from a search The Search API's `COMPLETE` view carries an `authkeywords` field alongside the usual title, DOI and date. Requesting it costs no extra request beyond `COMPLETE`'s own smaller page size (25 records per page, against 200 for `STANDARD`), which already means more requests, and so more quota, for the same number of records. ```{r, eval = FALSE} recs <- scopus_fetch("DOI(10.1038/nature14539)", view = "COMPLETE") recs$authkeywords ``` ```{r, include = FALSE} # Representative COMPLETE-view result, assembled offline. authkeywords is the # single string scopus_fetch() returns under view = "COMPLETE", one element # per record, in Scopus' own " | "-delimited form. The bibliographic fields are # the document's own; the 'Scopus' identifier and citation count are left NA # rather than guessed at, since nothing on the page prints them. recs <- tibble::tibble( entry_number = 1L, scopus_id = NA_character_, doi = "10.1038/nature14539", title = "Deep learning", authors = "LeCun Y.; Bengio Y.; Hinton G.", year = 2015L, date = "2015-05-28", publication = "Nature", citations = NA_integer_, authkeywords = "deep learning | neural networks | representation learning | backpropagation" ) ``` ```{r} recs$authkeywords ``` In development, this field came back `NA` even on a live, otherwise fully-entitled key, for documents that do carry author keywords in 'Scopus' itself, which points to an entitlement gap specific to this one field rather than the documents genuinely having none. If your own keywords come back all `NA`, it is worth raising with your Scopus/Elsevier account contact. ## References via Abstract Retrieval The reference list is not available from Search under any view; it needs Abstract Retrieval's `FULL` or `REF` view, an entitlement separate from ordinary abstract access and from Search access. This is a per-document endpoint, so retrieving references for *n* documents costs *n* requests against Abstract Retrieval's own, smaller weekly quota, separate from Search's. Each row of the returned `references` data frame is one cited work. ```{r, eval = FALSE} ab <- scopus_abstract( "10.1038/nature14539", view = "FULL", include = c("references", "keywords") ) ab$references[[1]][, c("title", "authors", "source", "year")] ``` ```{r, include = FALSE} # Representative Abstract Retrieval result: one row carrying a `references` # list-column (a data frame of cited works, in scopus_abstract()'s schema) and # the same n_requests / quota attributes the function attaches. refs <- tibble::tibble( position = as.character(1:4), # The 'Scopus' identifier of each cited work is left NA, as it is whenever # the API does not resolve one, rather than invented for the illustration. id = NA_character_, doi = c("10.1109/5.726791", "10.1038/nature14236", NA, NA), title = c( "Gradient-based learning applied to document recognition", "Human-level control through deep reinforcement learning", "ImageNet classification with deep convolutional neural networks", "Learning representations by back-propagating errors" ), authors = c( "LeCun Y.; Bottou L.; Bengio Y.; Haffner P.", "Mnih V.; Kavukcuoglu K.; Silver D.", "Krizhevsky A.; Sutskever I.; Hinton G.", "Rumelhart D.; Hinton G.; Williams R." ), source = c( "Proceedings of the IEEE", "Nature", "Advances in Neural Information Processing Systems", "Nature" ), year = c(1998L, 2015L, 2012L, 1986L), # NA is what view = "FULL" actually returns for this column; only view = # "REF" populates a cited work's own citation count. citedbycount = NA_integer_ ) ab <- tibble::tibble( id = "10.1038/nature14539", doi = "10.1038/nature14539", title = "Deep learning", year = 2015L, # scopus_abstract() joins keywords "; ", like its authors column; only # Search's COMPLETE view uses the " | " form shown above. authkeywords = "deep learning; neural networks; representation learning; backpropagation", references = list(refs) ) attr(ab, "n_requests") <- 1L attr(ab, "quota") <- list(remaining = 24999L, reset = NA_character_) ``` ```{r} ab$references[[1]][, c("title", "authors", "source", "year")] ``` `view = "FULL"` is the recommended default: in development, it returned a complete, correctly counted reference list for every document tried, while `view = "REF"` returned an inconsistent, sometimes-truncated subset, on an otherwise identical request made moments apart. `scopus_abstract()` warns when the number of references returned does not match the document's own reported count, rather than returning a partial list silently. The number of requests spent and the remaining Abstract Retrieval quota are attached as attributes, since this endpoint draws on a separate, smaller quota that is easy to exhaust unnoticed: ```{r} attr(ab, "n_requests") # requests spent so far attr(ab, "quota")$remaining # Abstract Retrieval quota left ``` A key or subscription tier that does not cover the requested view raises a `scopus_error_forbidden` condition naming the view, rather than a generic HTTP failure, and stops the whole batch rather than repeating the same failure for every remaining identifier, since entitlement is an account-level property, not a per-document one. For more than a handful of identifiers, pass `cache_dir` so an interrupted or quota-limited batch resumes without re-spending quota already spent: ```r ab <- scopus_abstract( dois, view = "FULL", include = c("references", "keywords"), cache_dir = "abstract-cache" ) ``` ## A minimal, cross-tool corpus `scopus_corpus()` combines a search result with this Abstract Retrieval step, returning a minimal shape close to what OpenAlex's `works` API already returns: `id`, `title`, `year`, `keywords` (a list-column of character vectors) and `references` (a list-column of data frames), rather than bibliometrix's semicolon-joined citation strings. It does not replace [`as_bibliometrix()`], which keeps its own field-mapping convention for users who want that instead. ```{r, eval = FALSE} recs <- scopus_fetch("DOI(10.1038/nature14539)", max_results = 1) corpus <- scopus_corpus(recs, view = "FULL") corpus$keywords[[1]] nrow(corpus$references[[1]]) ``` ```{r, include = FALSE} # scopus_corpus() pairs each record with its Abstract Retrieval references and # splits scopus_abstract()'s "; "-joined authkeywords string into a character # vector per record. corpus <- tibble::tibble( id = "10.1038/nature14539", title = "Deep learning", year = 2015L, keywords = list(trimws(strsplit(ab$authkeywords, ";", fixed = TRUE)[[1]])), references = list(refs) ) ``` ```{r} corpus$keywords[[1]] nrow(corpus$references[[1]]) ``` The keywords list-column is ready for the co-occurrence analysis this article is named for, unnested and tallied into a per-keyword document count across the corpus: ```{r} sort(table(unlist(corpus$keywords)), decreasing = TRUE) ``` This costs one Abstract Retrieval request per record in `recs`, on top of whatever retrieved `recs` in the first place.