Semanticlab Weblog

Big, Linked Geospatial Data and Its Application in Earth Observation

2018-03-09T00:00:00+01:00

Integrating earth observation data with linked open data would pave the way for easy reuse and integration of these datasets. The article discusses how knowledge discovery and data mining can be used to enrich image metadata and GIS data with domain concepts that have been formalized in domain ontologies. It then presents tools used to convert, store and visualize these data.

Tools

Strabon is a geospatial database system build upon RDF4j supporting spatial and temporal features such as selects, joins, etc.
GeoTriples provides means for publishing geospatial data in formats such as shapefiles, spatially enabled DBMS, KML and GeoJSON in RDF.
Sextant enables visualization, exploration and interaction with time-evolving linked geospatial data

Employment relations: a data driven analysis of job markets using online job boards and online professional networks

2017-11-21T00:00:00+01:00

Career websites contain valuable data on employees, their skill sets and, employment history. This article uses k-means clustering on keywords describing skill sets that have been transformed using either

term frequency inverse document frequency (TF-IDF) or
t-distributed stochastic neighbor embedding (TSNE).

A third experiment performs the clustering after 20 keywords have been selected using Latent Dirichlet Allocation.

In addition the authors extract the chronological information about positions to visualize potential career paths.

Insights

clustering jobs per title yields job titles commonly used for different kind of work (e.g. web developer, business intelligence, oracle development, etc.)
the network generated from the chronological information shows (i) typical career paths and (ii) identifies positions with high in- and out-degrees.

Suffix array

2017-11-20T00:00:00+01:00

The suffix array is a memory-efficient alternative to the suffix tree which provides a sorted list of string indices indicating the string’s suffixes.

Creation

Provided that the sentinel character $ has the smallest value the suffix array for the string barbapapa is determined as follows:

determine all possible suffixes and the corresponding start indices (the index one indicates the first position).

suffix	start position in string
barbapapa$	1
arbapapa$	2
rbapapa$	3
bapapa$	4
apapa$	5
papa$	6
apa$	7
pa$	8
a$	9
$	10

sort all suffixes

suffix	start position in string
$	10
a$	9
apa$	7
apapa$	5
arbapapa$	2
barbapapa$	1
bapapa$	4
pa$	8
papa$	6
rbapapa$	3

the suffix array only records the start positions, i.e. contains the values [10, 9, 7, 5, 2, 1, 4, 8, 6, 3] in the example above.

Suffix arrays can be derived from suffix trees in $\mathcal{O}(n^2 log\quad n)$ although more efficient algorithms that are able to build the array in $\mathcal{O}(n)$ exist.

Applications:

searching: e.g. does the suffix papa occur in the string? Do a binary search in the array and lookup the corresponding string value to determine whether a suffix is present.
counting: e.g. how often does the suffix apa occur?. Do a binary search and count the neighboring suffixes starting with apa.
K-mer counting: compute all substrings of up to size k and the number of times they occur in the string.

Dynamic feature scaling for online learning of binary classifiers

2017-11-10T00:00:00+01:00

This article describes and evaluates different online feature scaling approaches and their impact on the performance of binary classifiers.

online feature scaling requires features to be adapted on the fly, i.e. before all properties of the underlying distribution are known.
the evaluations in the article suggest, that the rather simple unsupervised dynamic scaling approach performs exceptionally well when used with the average weight vector for training and testing, to reduce the impact of the last features encountered in the learning step.

Method

The feature value $x_j$ for feature $j$ is transformed using the medium $\mu_j$ and standard deviation $\sigma_j$ to the scaled value $x_j'$:
\[x_j' = \frac{x_j - \mu_j}{\sigma_j}\]
estimations of the kth update of the mean and standard deviation for the jth feature are obtained from:
\[\begin{align} \mu^k_j &= \mu^{k-1}_j + \frac{x_j^k - \mu_j^{k-1}}{k} \\ s^k_j &= s^{k-1}_j + (x^k_j - \mu_j^{k-1}) (x_j^k - \mu_j^k) \text{ with}\\ \sigma^k_j &= \sqrt{s^k_j/(k-1)} \end{align}\]

40 years of suffix trees

2017-11-03T00:00:00+01:00

Suffix trees are used in text searching, indexing, statistics. This article describes the history, construction, current developments and applications of suffix trees.

Of special interest are applications that use suffix tries in conjunction with the Burrows-Wheeler transformation for compression and structures such as the succinct suffix tree and the suffix array that aim at reducing the space requirements of the tree.

Data sketching

2017-10-27T00:00:00+02:00

This article introduces three popular data structures that efficiently handle and summarize large data sets.

Bloom filters are basically sets which answer the question of whether an item is part of the set (i.e. has been seen by the filter) with either (i) “the item is definitely not a part of the set”, or (ii) “the item might be part of the set.
The Count-Min Sketch method is a probabilistic method for counting the number of times items of a certain type have been observed. When queried the structure returns an estimation which is considered an upper bound for the corresponding count.
The HyperLogLog method counts the number of different items seen in a large set of individuals without keeping count of every single individual.

Rich Data, Poor Fields

2016-11-23T14:27:21+01:00

This article shows how handheld devices and big data technology may multiply field yields and make farming more environmentally friendly.

GPS enabled precision leveling techniques and soil sampling reduce water and fertilizer consumption while increasing the farmer’s yield.
data mining and maschine learning help in analyzing satelity data, ground surveys, and past data on the climate, relief, geology and vegetation, supporting farmers in deciding (i) what to grow on and (ii) how to treat the soil;
hyperspectral imaging using satellites, drones and handheld devices recognizes diseases at an early stage
hand-held analytics devices help analyzing irregularities with the crop

Borg, Omega and Kubernetes

2016-11-23T14:03:56+01:00

Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Commun. ACM, 59(5), 50—57. https://doi.org/10.1145/2890784

Summary

In this article five Google engineers discuss lessons learned from over a decade of container management at Google such as

the transformation from machine-oriented to application oriented (machine and operation system agnostic) deployment
- containers are the only entity supported by Google's infrastructure
- log collection, telemetry (cAdvisor) and configuration is done on a per application level
bundling of application containers into pods. A pod may, for instance, contain containers for (i) the main application, (ii) log rotation, etc. This design increases modularity and allows multiple teams to develop different pieces of the service.
Reduction of complexity by standardizing APIs and metadata. Kubernetes for instance provide the following three fields for every object:
- ObjectMetadata: information on objects (name, UID, etc.)
- Specification: the object's desired state
- Status: the object's current state
recommendations for best practices
- don't let the container system manage port numbers
- provide labels and metadata with the containers
- don't expose raw state (e.g. force service access through a centralized API that hides implementation details and provides services for object validation, defaulting and versioning).

Social-media-based public policy informatics: Sentiment and network analyses of U.S. Immigration and border security

2016-08-15T08:48:03+02:00

Chung, W., & Zeng, D. (2016). Social-media-based public policy informatics: Sentiment and network analyses of U.S. Immigration and border security. Journal of the Association for Information Science and Technology, 67(7), 1588—1606.

Summary

This article presents a case study from the domain of policy informatics - i.e. the field of finding effective ways to use information technology to understand and tackle complex problems of society - which describes how Web intelligence has been successfully applied to assess the public sentiment towards the U.S. immigration and border security.

Method

a framework for collecting relevant Tweets from Twitter
sentiment analysis using eight emotional categories defined in Plutchik (1980): anger, disgust, fear, sadness, surprise, anticipation, joy and trust and the emotion lexicon developed by Mohammad & Turney 2013.
social network analysis using different metrics such as most followed users, most befriended users, and betweenness centrality for identifying key players
examination of differences in emotion and network between three major phases of the discussion using the Welch two sample t-test.

Conclusions

news often coincide with changes in public sentiment
Tweet posters and their relationships change considerably over time
different influence metrics yield significantly different rankings

Polarity Shift Detection, Elimination and Ensemble: A Three-Stage Model for Document-Level Sentiment Analysis

2016-08-12T10:55:28+02:00

Xia, Rui, Feng Xu, Jianfei Yu, Yong Qi, and Erik Cambria. Polarity Shift Detection, Elimination and Ensemble: A Three-Stage Model for Document-Level Sentiment Analysis. Information Processing & Management, Emotion and Sentiment in Social and Expressive Media, 52, no. 1 (January 2016): 36—45.

Introduction

This article presents an approach for handling explicit polarity shifts due to (i) negation (I don't like this movie), and (ii) contrast (Fairly good, but not my style) as well as implicit shifts due to sentiment inconsistency that appears frequently if people express different opinions towards different aspects of a product or event.

Negation usually shifts the sentiment of the affected parts,
Contrast can shift the polarity of neighboring sentences or subsentences. If the polarity shift is not considered, the impact of the shifted part should be decreased relatively to the unshifted part.
Sentiment inconsistency can be viewed as a type of implicit contrast - i.e. the impact of the inconsistent parts should be weakened.

Method

The authors deploy a three stage model to address polarity shifts:

Rules identify negation as well as forward and backward contrasts. A statistical method that draws upon the weighted log-likelihood ratio (WLLR) detects statistical inconsistencies in reviews (i.e. cases where one sentence sentiment is inconsistent with the overall document sentiment).
Negation polarity shift elimination by replacing negated terms with their antonyms. The antonyms are, again, determined based on the WLLR metric obtained for terms in the training documents. For example, the most positive term would be replaced with the most negative term according to WLLR (these substitutions correspond to the relative sentiment strength rather than to the actual meaning of the term). The relevance of a term for positive/negative sentiment is computed as outlined below. $$\text{relevance}(t_i, +) = p(t_i | +) log \frac{p(t_i |+)}{p(t_i | -)}$$ $$\text{relevance}(t_i, -) = p(t_i | -) log \frac{p(t_i |-)}{p(t_i | +)}$$

The polarity shift ensemble model is trained based on three components: (i) sentences for which negations have been eliminated, (ii) sentences containing contrast, and (iii) sentences with sentiment inconsistency as well as a base classifier for sentences without polarity shifts.

Evaluation

The evaluation draws upon the Multi-domain sentiment datasets by Blitzer, Dredze and Pereira (2007) that contains four datasets of 1000 positive and negative Amazon reviews each. The authors use linear SVM (LibSVM), logistic regression (LibLinear) and Naive Bayes (OpenPR-NB) to evaluate their approach against the following four methods:

Baseline (bag of words without negation detection)
Das (2001): negated words are marked with the suffix "-NOT" prior to training and classification.
REV: sentiment words in the scope of negation are reversed to their antonyms.
Li et al. (2010): text is separated into a polarity-shifted and polarity-unshifted fraction based on which two classifiers are trained.
the presented approach

The evaluation demonstrates that the presented approach outperforms all methods used in the experiments.