Matthias Traub, Rene Kaiser and Wolfgang Weiss
Institute of Information and Communication Technologies
JOANNEUM RESEARCH
Graz, Austria
matthias.traub@student.tugraz.at, rene.kaiser@joanneum.at, wolfgang.weiss@joanneum.at


ABSTRACT
As information technology has advanced, the usage of online social networks rapidly increased at the same time, and social interaction shifted onto these platforms. Online social networks have gained significant popularity and are now among the most popular sites on the Web. Likewise, sophisticated video communication systems, which in the past mainly concentrated on the business videoconferencing market, are shifting into the private domain. In the realm of these developments, we are aiming for integration of online social networks with intelligent video communication systems. Relevant information extracted from massive amounts of online social network data could inform intelligent behaviour within video communication systems, and therefore needs to be extracted. This article presents a selection of taxonomies of relationships as basis for multidimensional social networks that we consider relevant for such purposes. As users might not necessarily be very active, this selection also considers how well it could work with sparse network layers. Based on a literature study, we conclude that the combination of layers with sparse information onto a multidimensional network usually increases the usefulness of the data for its analysis.


1 INTRODUCTION
Social networks data is mainly composed out of social relationships. These relations are created by either by users, or by the online social network platform itself automatically. People who interact with each other, e.g. by sharing common activities or being member of a common group, form a social network. Users can establish groups of common interests, interact and collaborate with one another and create conscious (explicit) or unconscious (implicit) relationships. The social network makes the ways in which users are related explicit, and thereby allows to analyse how they are connected, based on one or several relation types [4]. Implicit relations can be extracted from online social network sites with state-of-the art algorithms.

The relationships in an online social network can be categorized in many different ways, based upon different taxonomies that reflect certain aspects of the relation. Each of these categories represents additional information that can be exploited to analyse the relations’ characteristics and their dynamics. These taxonomies must be carefully examined regarding the purpose of the analysis. Once the data is extracted, the categorization creates a consistent and overarching view on each relationship. Exploiting this additional information about each relationship, in contrast to utilizing only the basic information about the relation (e.g. actor, object, relation type, weight), can be a big advantage in terms of network analysis.

A social network analysis algorithm could focus for example on communication between users, and as a first step, it would create a social network based on direct relations between users. On the other hand the analysis could also be based on common activities like sharing or liking an image, or commenting on the same blog post, which would allow extracting indirect relationships. Merging multiple such layers, possibly both more topically-related and socially-related links, into a multidimensional social network can be more useful than examining online social networks containing only sparse information [5].

The analysis’ purpose that serves as context for the considerations of this article is to extract useful information from an online social network that can feed into the social network’s video communication system. The more the latter knows about the participants of the current session, the better it can satisfy the individual needs and preferences of its users.

Which pairs of users of the videoconferencing session have the strongest social ties? Which of them have a common interest about the topic that is currently being discussed (speech to text based analysis)? Which users within the group have a special static role, like the teacher in a session together with pupils? Knowing about the participants’ communication behaviour regarding text-based interaction in the social network (cp. [13]), could we predict their behaviour in video communication, e.g. who is shy and who will dominate the conversation by speaking all the time?

One aspect in video communication systems that could utilize such information is communication orchestration [14, 10]. This concept refers to an automatic process that decides which of the available video streams in a group video communication session are displayed for each particular users, and how. Especially in larger groups or setups with multiple cameras, orchestration can help users to see (and hear) what’s most relevant to them at each point in time, supporting the users’ individual communication needs. Putting the right video streams on screen in proper size, based automatic understanding on verbal and non-verbal interaction, enables to follow the dynamic conversation and to read facial expressions and subtle communication cues. In that sense, communication orchestration can exploit answers to the example questions in the previous paragraph.

A concrete example: if the videoconferencing system has access to all profiles of the session’s participants, it could derive static roles like “teacher” and “pupils” in this setup. The visual presentation could hence adapt, supporting the teacher with a good overview of the group in terms of camera selection and presentation, and supporting the pupils by giving the teacher a special, more important role, keeping him or her visible on screen most of the time, even when another pupil would be speaking. For further details on communication orchestration, please refer to [10] and [14].

This article is structured as follows. In Section II we discuss related work from the field of social network analysis, in particular such with connection to communication behaviour in online social networks and such that is generating social networks out of different feature layers. Then, we highlight basic information about the taxonomies of relationships and the specification of multidimensional social networks. We conclude a literature research by proposing and discussing a selection of relevant taxonomies with the use-case of informing communication orchestration in mind.


2 RELATED WORK
Social network analysis has long been an interesting area of study in the fields of human computer interaction, sociology and computer supported collaborative work. In general, online social networks display a rich internal structure where users can create relations of various types and mostly decide which and how many relations to create.

Those networks have existed since the beginning of the Internet. The graph derived from email communication can be used to form a social network. These communication networks have been studied by Diesner [11] in 2005 where email traffic of 21,000 employees was analysed. Dynamics of the structure and properties of the organizational communication network were explored, as well as the dynamics of the characteristics and patterns of communicative behaviour of the employees from different organizational levels.

Social networks illustrate the relation and connection between users and, as mentioned before, these connections or ties between users can be based on to either one or several relations. If those relations are weighted, the network can be treated as a fuzzy network [8].

Multimedia sharing sites like Flickr already have been subject of many social network studies. New relations emerge from the users’ common tags. Direct links between contact lists were also studied. Based on a Flickr dataset, Kazienko [9] defined nine types of relations between users. These relations were extracted from the contact lists, tags, groups of items and their authors, favourite pictures and comments to pictures. Some of them, like favourites and comments, were then split into three sub layers, e.g. author-commentator, commentator-author and commentator-commentator.


3 TAXONOMY OF RELATIONSHIPS
A relationship in an online social network can be clustered in many different ways and based on various characteristics. In this section we give a short overview of selected taxonomies we propose to utilize for supporting intelligent video conferencing tools that are integrated with online social networks. These taxonomies are useful for the observation of the relations between two users. In certain cases, the relation between two users is defined by a common relation to a third object, which we refer to as meeting object in the following. The following is based on previous work by the authors of [1], [2], [3], [6], and [7].

A Relationship creation

An obvious classification is based on the active subject that created a certain relationship. It can be divided into three main types:

• Created by the users themselves, e.g. by adding another user to a follower list, or by exchanging messages with another user.
•  Established by the system, e.g. by profile matching, also called demographic filtering.
• A combination of the previous, e.g. when the system suggests a new relation and the user confirms it.

Further, relationship dynamics like durability and intensity of a relationship can be predicted to a certain degree based on the relationship origin. Determining the creator of a relationship leads to a classification of users who are very active in creating new relations [1].

B Awareness of involved users

Another aspect of social relationships is the awareness of the users involved within a relation:

• Both users are aware of the relation, e.g. by exchanging messages directly with each other.
• Only one user is aware of the relation, e.g. user A is referring to user B in messages or posts.
• No user is aware of the relation. These relationships are usually based on profile matching, a process automatically performed by the system in the background. Usually, these are not directly visible to the users.

This awareness typically correlates with the level of activity within the network. If the relations of a network are constructed mainly out of relationships where both users are aware of it, chances are higher that these networks survive. In terms of relationship dynamics, the awareness factor is also important [1]. Do relationships behave differently when they start off created by the system or when both users are aware of the relation right away?

C Direction of relationship

As another relevant taxonomy, the direction and mutuality of the connection between users can be divided into three types:

• Asymmetrical, e.g. user A follows user B but there is no connection from B to A.
• Symmetrical, i.e. a relation exists when a mutual relationship exists between users or when users share a common activity, e.g. liking the same comment.
• Reflexive, e.g. connections from users back to themselves, commonly excluded from consideration for analysis purposes.

Changes in the directions of relationships can be used to analyse the relationship dynamics, e.g. to what extent and when do asymmetrical relations change to symmetrical, and vice versa [9].

D Data used to create relationships

This taxonomy is based on the type of data used for relationship creation. Splitting relationships into categories based on the type of data used to create those relations, adds additional granularity level to the analysis. Based on [4], we divided the relationships resting upon the following three compound data types:

• Mutual communication, e.g. direct messaging, email exchange and instant messaging.
• Common activities, e.g. commenting, liking, sharing the same (media) object.
• Profile matching data, e.g. contact lists, geo information or user profile information.

This categorization enables the investigation of the correlation between these features and type of data used to create a relationship. Considering the observation of just a single activity, e.g. the commenting of a photo, the information has limited usefulness for the analysis of the whole network.  For example the relation was created because two users commented the same photo but after that they just exchange direct messages. Only larger sets of data bring insights into the relationship dynamics (cp. [1]).

E Directness of relationships

The directness of the relationship provides more useful information about the users and their interaction than other categories. There are three kinds of directness of a relationship [1]:

• Direct relationship (Fig 1). It connects two entities with a direct connector. This direct connector is related to a specific feature, e.g. communication or activities. For example, sending a message to another user is typically creating a direct relationship.

Fig. 1.    Direct relation with unidirectional connections. There could also be direct bidirectional relations.

• Quasi-direct relationship. The relation between the users is based on the connection to a certain object, e.g. posting, photo, and video conference session. There is a communication meeting object between users. The roles of a user within such a relationship can be equal or different regarding to the meeting object.

• Quasi-direct relation with equal roles (Fig 2). This means that the users A and B build up a relation through the meeting object and their role in relation to this object is the same. This usually occurs when participating in a common activity related to the meeting object, e.g. two users like or create a comment on the same meeting object.

Fig. 2.    Quasi-direct relation with eaual roles.

• Quasi-direct relation with different roles (Fig 3). User A and B build up a relation through a meeting object but they act within different roles, e.g. user A creates a post or picture and acts as the creator of the meeting object and user B comments or likes the meeting object.
  

Fig. 3.    Quasi-direct relation with different roles.

• Indirect relationships. Two users are connected by indirect link when their user profile information are similar to each other.

With the categorization of directness of a relationship, further characteristics can be analysed and exploited. Type of active subject, awareness of the users, mutuality of the connection and the data type used as meeting object can be defined.


4 MULTIDIMENSIONAL SOCIAL NETWORKS
A social network created out of the users’ communication behaviour consists of a single layer. However, a social network extracted from a typical online social network is based on a range of individual features available from users. The set of connections and relationships is derived directly from data about user activities, e.g. comments on blog posts, liking and sharing pictures, follower and followee links. Each feature may bind users in a different way and forms another kind of relation. Using the relationship taxonomies described above for every applicable feature creates a certain social network layer.

A connection between user A and B exists if at least one relation of any type is present. All of them combined create a strongly related group of users. Considering an online social network with only sparse information on a particular group of users, the observation of only one relationship feature would not give useful data to analyse [9].

For better understanding, Figure 4 shows a simple illustration of a multidimensional network and its combination of different layers. The bottom layer (resulting network) is a combination of three network layers based on different taxonomies and aspects. The figure contains five different users A to E displayed in bordered rectangles and meeting objects 1 to 3 displayed in double bordered green boxes. In the meeting object creator layer, the direct connection between meeting objects and their creator is illustrated. Since there are three meeting objects, there must be at least one, up to a maximum three, creators connected to them. The communication connection layer shows the user-to-user direct communication connection. In our example, user A follows user B, and E follows D. The meeting object commentator layer shows the connection between the meeting objects and the creation of comments. The combination of those three layers is shown in the bottom layer.

The combined multidimensional layer gives a complete overview of the whole network. It contains all the previous connections and enables more complex analysis methods. For example, in none of the other layers a connection from user A to D was clearly evident. Combining layers with sparse information can create a network with dense connections and serves as a basis for typical network analysis methods [1].

Fig. 4.    Combination of feature extraction layers to a combined multidimensional social network layer. User entities are display in colored rectangles labelled A to E. Meeting objects are depicted as green double borderd rectangles. The arrows illustrate the relation between the entities and objects.

5 DISCUSSION
In this article, we discuss possibilities for information extraction out of an online social network environment. The use-case we had in mind was to extract information that could support a real-time video communication system [10, 14] which exploits users’ social media profiles and streams as background knowledge for orchestrating video communication. Especially information about the communication behaviour inside the network with the other participants of the video conference session could be interesting for orchestration purposes, as well as shared common activities and similarities of user profiles, such as common interests.

Information about the users’ connections inside the online social network can be used to inform communication orchestration and network optimization components of a video communication system. This could bias camera selection in order to preferably show other users that are strongly connected to the current user, which may especially support users in heated discussions of larger groups where it is difficult to keep the overview. General classification of user profiles can be used to specify communication preferences. Orchestration may even influence the conversation as a moderator through its actions, for example supporting shy participants to participate by giving them visual presence easier than others once they decide to actively communicate.

This work is part of an ongoing research activity. So far we have not managed to implement a dedicated social network analysis component that is integrated with our video communication system. Therefore, this article does not include any evaluation of the construction of a multidimensional social network and the usefulness of the presented taxonomies for our application.

The limitation of each of the selected taxonomies in isolation is that they can be used to analyse only one feature of the relationships between users. To create a complete description of a certain relationship, it is essential to define a category within each of the selected taxonomies. Utilizing this extra information implies benefits for the analysis. Only the compound set of relationships should be used for creating a layer for the multidimensional social network observation.

Our literature research has further confirmed that the combination of multiple relation types based on different social network features also gives more insight into the global social network, especially if the existing data is sparse.

Acknowledgment
This work was performed within Vconect - Video Communications for Networked Communities (grant agreement no 287760) and received funding from the European Commission under the Seventh Framework Program (http://www.vconect-project.eu).

REFERENCES
[1] Musial K., Kazienko P., Social networks on the Internet, World Wide Web, 2012, Vol. 16, pp. 31–72.

[2] Newman M.E.J., Girvan M., Finding and evaluating community structure in networks. Phys. Rev. E 69(2), 026113, 2004.

[3] Vaidyanathan A., Shore M., Billinghurst, M., Data in Social Network Analysis, 2009, pp. 134–149.

[4] Fuchs C., Towards a dynamic Theory of Virtual Communities, International Journal of Knowledge and Learning 3(4/5), 2007, pp. 372-403.

[5] Kazienko P., Bródka P., Musial K., and Gaworecki J., Multi- layered Social Network Creation Based on Bibliographic Data, SocialCom-10, The Second IEEE International Conference on Social Computing,SIN-10 Symposium on Social Intelligence and Networking, IEEE Computer Society Press, 2010, pp. 407-412.

[6] Garton L., Haythorntwaite C., and Wellman B., Studying Online Social Networks, Journal of Computer-Mediated Communication, 3 (1), 1997.

[7] Musiał K., Kazienko P., and Kajdanowicz T., Multirelational Social Networks in Multimedia Sharing Systems, Chapter in N.T.Nguyen, G.Kołaczek, B.Gabryś (eds.) Knowledge Processing and Reasoning for

Information Society, Academic Publishing House EXIT, Warsaw, 2008, pp. 275-292.

[8] Brunelli M., A Fuzzy Approach to Social Network Analysis Social Network Analysis and Mining, 20-22 July 2009, pp. 225 - 230.

[9] Kazienko P., Musiał K., Kajdanowicz T.: Multidimensional social network and its application to the social recommender system. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 41(4), 746–759 (2011).

[10] Kaiser R.,Weiss W., Falelakis M., Michalakopoulos S., and Ursu M. F., A rule-based Virtual Director enhancing group communication. In ICME Workshops, IEEE (2012), 187–192.

[11] Diesner J. et al., Communication Networks from the Enron Email Corpus “It's Always About the People. Enron is no Different”, Computational & Mathematical Organization Theory October 2005, Volume 11, Issue 3, pp 201-228.

[12] Borgatti S.P., et. al, Network Analysis in the Social Sciences, Feb 2009, Sciencemag vol 323, pp. 892-895.

[13] Johannes Schantl, Claudia Wagner, Rene Kaiser and Markus Strohmaier, The Utility of Social and Topical Factors in Anticipating Repliers in Twitter Conversations, ACM Web Science, Paris, France, 2013.

[14]     Ursu, M. F., Groen, M., Falelakis, M., Frantzis, M., Zsombori, V. and Kaiser, R. (2013). Orchestration: tv-like mixing grammars applied to video-communication for social groups. In ACM MM 2013, pp. 333-342.

Matthias Traub is currently working on his master thesis in Telematics at the University of Technology in Graz. He received his Bachelor's degree in Telematics in 2012 and specialized in Multimedia Information systems, Technical Informatics and Pervasive Computing. He works in the scope of the EU-funded FP7 project Vconect (www.vconect-project.eu) at JOANNEUM RESEARCH. His research interests include social media analysis and its application, conversation analysis in real-time video communication system.​

Rene Kaiser is key researcher at JOANNEUM RESEARCH, Graz, Austria. His main research expertise and interests are in the fields of content personalization, media semantics, knowledge representation, event processing, social computing, videoconferencing and telepresence, and non-linear video production, with particular focus on “Virtual Director” technology for automatic framing and selection of live video streams. Rene studied Software Engineering at FH Hagenberg, is a PhD student at TU Graz, member of the Wissensmanagement Forum Graz, and co-organizer of the yearly Barcamp Graz. He contributed to European research projects NM2, Clinicip, Aposdle, TA2, FascinatE and Vconect. Rene is an active member of IEEE STCSN and contributes as E-Letter chair. To get in touch, please see contact information at http://about.me/rene.kaiser.

Wolfgang Weiss is a researcher at JOANNEUM RESEARCH, Institute for Information and Communication Technologies. His research fields are Semantic Web, temporal reasoning and event processing, and Virtual Directors. The activities of Virtual Directors are in the field of video mediated communication and live event broadcast systems. Wolfgang works in the scope of the EU funded project VCONECT (http://www.vconect-project.eu/) and gained experience in previous EU funded projects TA2 (http://www.ta2-project.eu/) and FascinatE (http://www.fascinate-project.eu/).