Skip to content

Tech Talk: Tom Hughes-Croucher (Joyent) — “Node.js at Scale”

Node.js at Scale
Tom Hughes-Croucher (Joyent)
Thursday, August 4, 2011

ABSTRACT

When we talk about performance what do we mean? There are many metrics that matter in different scenarios but it’s difficult to measure them all. Tom Hughes-Croucher looks at what performance is achievable with Node today, which metrics matter and how to pick the ones that most matter to you. Most importantly he looks at why metrics don’t matter as much as you think and the critical decision making involved in picking a programming language, a framework, or even just the way you write code.

BIOGRAPHY

Tom Hughes-Croucher is the Chief Evangelist at Joyent, sponsors of the Node.js project. Tom mostly spends his days helping companies build really exciting projects with Node and seeing just how far it will scale. Tom is also the author of the O’Reilly book “Up and running with Node.js”. Tom has worked for many well known organizations including Yahoo, NASA and Tesco.

Tagged
2011 08 11 Sam tech talks Comments (0) Permalink

Tech Talk: Neha Narkhede (LinkedIn) — Kafka, LinkedIn’s open-source distributed pub-sub messaging system

Kafka
Neha Narkhede (LinkedIn)
Wednesday, July 27, 2011

ABSTRACT

Kafka is a distributed publish-subscribe messaging system aimed at providing a scalable, high-throughput, low latency solution for log aggregation and activity stream processing for LinkedIn. Built on Apache Zookeeper in Scala, Kafka aims at providing a unified stream for both real-time and offline consumption. We provide a mechanism for parallel data load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines. Kafka combines the benefits of traditional log aggregators and messaging systems and has been used successfully in production for 8 months. It provides API similar to that of a messaging system and allows applications to consume log events in real-time.

Written by the SNA team at LinkedIn, Kafka is open sourced under the Apache 2.0 License and is an Apache incubator project. In this presentation, we will highlight the core design principles for this system, and how this system fits into LinkedIn’s data ecosystem as well as some of the products and monitoring applications it supports in our usage.

BIOGRAPHY

Neha Narkhede is a Senior Software Engineer in the Search, Network and Analytics Team at LinkedIn, focusing on Distributed Systems. She is one of the initial contributors to Project Kafka. In the past she’s worked on search systems in large scale databases and has been an active contributor to several projects LinkedIn has open sourced, including Voldemort, Bobo and Zoie.

Tagged ,
2011 08 11 Sam tech talks Comments (0) Permalink

Tech Talk: Rajat Paharia (Bunchball) — “Game Dynamics”

Game Dynamics
Rajat Paharia (Bunchball)
Monday, July 25th, 2011

ABSTRACT

Status, achievement, reward, competition, self-expression: by addressing these fundamental human needs and desires, designers can make experiences both compelling and satisfying. Game designers, in particular, have known for years how to incent and motivate players by addressing these needs through the use of mechanics like points, levels, leaderboards, virtual goods, challenges, and real-time feedback. We’ll cover the migration of these mechanics out of the gaming world and into the world at large, including destination sites, devices, productivity applications, corporate intranets, and LinkedIn.

BIOGRAPHY

Rajat Paharia is the founder and Chief Product Officer of Bunchball. He founded the company in 2005, and released the industry’s first gamification platform, Nitro, in 2007. Rajat works closely with Bunchball’s Fortune 500 clients to craft gamification strategies that engage customers, partners and employees, including companies like NBC, Warner Brothers, Comcast and Hewlett-Packard. Rajat’s skill set combines a unique expertise in technology, design and human behavior developed during a four-year career at design firm IDEO and through degrees in computer science and human-computer interaction.

Tagged
2011 07 28 Sam tech talks Comments (0) Permalink

Tech Talk: Michael Stack (StumbleUpon) — “State of HBase”

State of HBase
Michael Stack (StumbleUpon)
Monday, July 18th, 2011

ABSTRACT

Attendees will learn about the current state of the HBase project. We’ll review what the community is contributing, some of the more interesting production installs, killer apps on HBase, the on-again, off-again HBase+HDFS love affair, and what the near-future promises. A familiarity with BigTable concepts and Hadoop is presumed.

BIOGRAPHY

Michael Stack works at StumbleUpon and is Chief Janitor on the HBase Project and a member of the Hadoop PMC.

Tagged
2011 07 26 Sam tech talks Comments (0) Permalink

Project Voldemort 0.90 release

I’m thrilled to announce that we have finally cut off a branch and are ready to do the 0.90 open source release for Project Voldemort. For folks still enchanted by the Potter mania, no, I am not talking about Harry’s nemesis. Project Voldemort is an open-source distributed key-value store being used here at LinkedIn and at various other companies. This is one of our biggest open-source releases and contains features that we have worked on and deployed at LinkedIn in the last one year.

Over the past year our user base internally has grown tremendously. About 3 years ago Voldemort was deployed as a small cluster to serve our first product Who viewed my profile. Over time we have grown to multiple clusters, some of which now span over LinkedIn’s two data-centers. At its crux a single cluster serves either one of the two types of applications – those dealing with read-only static data (computed offline in Hadoop after some number crunching) or read-write data. Now with around 200 stores overall in production we have been serving various critical user facing features. Some examples of applications that use us include People you may know, Jobs you may be interested in, Skills, LinkedIn Share Button, Referral Engine, Company Follow, LinkedIn Today and more. With new users ramping in nearly every other week our clusters have collectively started doing over a billion queries per day.

picture-48_01

What’s new?

One of the most important upgrades we have done in production recently has been switching all our clients and servers from the legacy thread-per-socket blocking I/O approach to the new non-blocking implementation which multiplexes using just a fixed number of threads (usually set in proportion to the number of CPU cores on the machine). This is good from an operations perspective on the server because we no longer have to manually keep bumping up the maximum number of threads when new clients are added. From the client’s perspective we now won’t need to worry about thread pool exhaustion due to slow responses from slow servers.

While upgrading the client side logic, we also redesigned the routing layer to model it as a pipelined finite state machine (we call it the pipeline routed store). For example, a put() request on the client is now modeled as a series of following states:

  1. Generate list of nodes to put to
  2. Put sequentially till first success
  3. Put in parallel to the rest of the nodes
  4. Increment vector clock.

Designing every client request as states with transitions makes it easy to extend the pipeline and add new features (states) with minimum hassle. This enabled us to add support for a long-standing feature request – hinted-handoff, an additional consistency mechanism that handles cases of transient failures by using other live nodes as a backup system. Also in preparation for our new datacenter we were able to quickly plug in a new topology aware routing strategy that we call zone aware routing. For this strategy we cluster nodes into logical groups called zones (in our case a zone = data-center). Our routing strategy is then just a simple extension of Amazon Dynamo’s partitioning algorithm with special constraints on how we jump the ring.

The other important feature that we use at LinkedIn is the read-only stores pipeline. Our largest read-only stores cluster powers most of the recommendation features and fetches around 3 TB of data every day from Hadoop. Some of the initial work we did in this area was to make the fetch + swap pipeline more efficient by migrating off the old servlet based tool to a new administrative service based tool. This gave us better visibility into progress of data transfer (fetch phase) along with more control for swaps. Besides optimizing the data-transfer pipeline, we also spent some time iterating on the underlying storage format and data layout. The final storage format that we have come up with has a better memory footprint, supports iterators and has been tweaked for making rebalancing of read-only stores as simple as possible.

Besides working a lot on our Java client, we also updated some clients relying on Voldemort’s server-side routing. In particular the Python and Ruby clients have gone through some iterations. Many of our internal customers use these for quick prototyping of their ideas during our monthly Hackdays. In fact one of the by-products of this inDay has resulted in Voldemort having a nice GUI, more about which you can read here.

What’s next?

With some of the core pieces of the system in place now, our road-map is to pay more emphasis on the operation / management aspects. For example, we want to provide better tools to make ‘migration’ or ‘rebalancing’ of clusters as easy as pressing a button. Doing so would allow us to have larger clusters thereby decreasing operational overhead of maintaining many small clusters. We’re also slowly changing our old philosophy of adding new clusters for every new application and instead plan to add better support for multi-tenancy.

Fork / Download Voldemort, read all about updating to the new version, find an interesting new sub-project, submit a patch and come join the Death Eaters.

2011 07 19 rsumbaly announcements Comments (0) Permalink

Tech Talk: Andy Twigg (Acunu) — Stratified B-Tree and Versioned Dictionaries

Stratified B-Tree and Versioned Dictionaries
Andy Twigg (Acunu)
Monday, June 20, 2011

ABSTRACT

A classic versioned data structure in storage and computer science is the copy-on-write (CoW) B-tree — it underlies many of today’s file systems and databases, including WAFL, ZFS, Btrfs and more. Unfortunately, it doesn’t inherit the B-tree’s optimality properties; it has poor space utilization, cannot offer fast updates, and relies on random IO to scale. Yet, nothing better has been developed since. We describe the `stratified B-tree’, which beats all known semi-external memory versioned B-trees, including the CoW B-tree. In particular, it is the first versioned dictionary to achieve optimal tradeoffs between space, query and update performance.

BIOGRAPHY

Andy Twigg is Chief Scientist at Acunu, a UK startup building a ground-up platform for Big Data, initially by rewriting the storage stack. His PhD from Cambridge on compact routing was nominated for the British Computer Society Best Dissertation Award, and he is a Junior Research Fellow at St John’s College, Oxford and member of the CS Department.

Tagged
2011 07 08 Sam tech talks Comments (0) Permalink

Tech Talk: Michael Deerkoski (Flickr) — “Continuous Deployment at Flickr”

Continuous Deployment at Flickr
Michael Deerkoski (Flickr)
Wednesday, May 25, 2011

ABSTRACT

Flickr is almost certainly the best online photo management and sharing application in the world. The small, efficient, development team uses a process called Continious Deployment. There are several technical tools in place to make this happen, but the most important aspect to make it work is the culture of individual accountability. We’ll dive into that process, how it works, and the pros and cons.

BIOGRAPHY

Mike Deerkoski is the Senior Director of Engineering at Flickr. Before Flickr, among other roles, Mike was the technical lead for display ad serving and targeting for Yahoo!, including behavioral targeting, delivering 6 billion ad impressions daily. Also Mike was the engineering lead for the My Excite Start page, one of the first consumer-facing java web applications and the largest to date.

Tagged
2011 06 01 Sam tech talks Comments (0) Permalink

Tech Talk: Anil Madan (eBay) — “Hadoop at eBay”

Hadoop at eBay
Anil Madan (eBay)
Monday, May 23, 2011

ABSTRACT

The talk will illustrate how Hadoop has become a critical center piece of infrastructure for eBay, running on thousands of servers. I will also discuss how it fuels our derived data pipeline which in turn affects just about all our services. Attendees will understand how we have integrated Hadoop into our existing data warehouse and how we are leveraging components of the ecosystem like HBase, Pig, and Hive for different research and production use cases.

BIOGRAPHY

Anil Madan is the Director of Engineering at eBay responsible for Hadoop platform development. Prior to Hadoop at eBay, he has built different ecommerce platforms namely User Behavior Tracking, Experimentation, Personalization, Advertising & Click/Impression Filtering.

Tagged
2011 06 01 Sam tech talks Comments (0) Permalink

Kafka v0.6 release

We are pleased to announce the release of Project Kafka version 0.6. This is the first release after we open-sourced the project towards the end of 2010. Our philosophy is to successfully deploy any big feature in production at LinkedIn, before making it available in an open-source release. Although this ensures a stable release, it could take several months to get the next version out. We are now working on developing frequent and regular release cycles for Kafka.

In this release, we focused on enhancing the producer component. In addition to this, several important bug fixes and enhancements have made it into this release.

Here is a description of the producer features that we developed for v0.6 -

Automatic load balancing

In v0.6, we introduced built-in automatic load balancing between the producers and the brokers in Kafka. Currently, in our own usage we publish from a large number of heterogeneous machines and so it is desirable that the publisher not need any explicit knowledge of the cluster topology. We rely on a hardware load balancer to distribute the producer load across multiple brokers. An advantage of using the hardware load balancer is the “healthcheck” functionality that detects if a broker is down and forwards the producer request to another healthy broker. In v0.6, this “healthcheck” feature is provided in the cluster-aware producer. Producers discover the available brokers in a cluster and the number of partitions on each, by registering watchers in zookeeper. Since the number of broker partitions is configurable per topic, zookeeper watchers are registered on the following events -

  • a new broker comes up
  • a broker goes down
  • a new topic is registered
  • a broker gets registered for an existing topic

Internally, the producer maintains an elastic pool of connections to the brokers, one per broker. This pool is kept updated to establish/maintain connections to all the live brokers, through the zookeeper watcher callbacks. When a producer request for a particular topic comes in, a broker partition is picked by the partitioner (see section on Semantic partitioning). The available producer connection is used from the pool to send the data to the selected  broker partition.

Asynchronous send

Asynchronous non-blocking operations are fundamental to scaling messaging systems. In Kafka, the producer provides an option to use asynchronous dispatch of produce requests (producer.type=async). This allows buffering of produce requests in an in-memory queue and batch sends that are triggered by a time interval or a pre-configured batch size. Since data is typically published from set of heterogenous machines producing data at variable rates, this asynchronous buffering helps generate uniform traffic to the brokers, leading to better network utilization and higher throughput.

Semantic partitioning

Consider an application that would like to maintain an aggregation of the number of profile visitors for each user. It would like to send all profile visit events for a member to a particular partition and, hence, have all updates for a member to appear in the same stream for the same consumer thread. In v0.6, we added the capability to the cluster aware producer to be able to semantically map messages to the available kafka nodes and partitions. This allows partitioning the stream of messages with some semantic partition function based on some key in the message, to spread them over broker machines. The partitioning function can be customized by providing an implementation of the kafka.producer.Partitioner interface, default being the random partitioner. For the example above, the key would be member_id and the an example of a partitioning function would be hash(member_id)%num_partitions.

To check out the new functionality, download the new version of Kafka here. To get started with the code and get an example working, see our quickstart.

2011 05 25 Neha Narkhede announcements Comments (0) Permalink

Tech Talk: Xavier Amatriain (Telefonica) — “The Science and Magic of User and Expert Feedback for Improving Recommendations”

The Science and Magic of User and Expert Feedback for Improving Recommendations
Dr. Xavier Amatriain (Telefonica)
Thursday, March 3, 2011

ABSTRACT

Recommender systems are playing a key role in the next web revolution as a practical alternative to traditional search for information access and filtering. Most of these systems use Collaborative Filtering techniques in which predictions are solely based on the feedback of the user and similar peers. Although this approach is considered relatively effective, it has reached some practical limitations such as the so-called Magic Barrier. Many of these limitations strive from the fact that explicit user feedback in the form of ratings is considered the ground truth. However, this feedback has a non-negligible amount of noise and inconsistencies. Furthermore, in most practical applications, we lack enough explicit feedback and would be better off using implicit feedback or usage data.

In the first part of my talk, I will present our studies in analyzing natural noise in explicit feedback and finding ways to overcome it to improve recommendation accuracy. I will also present our study of user implicit feedback and an approach to relate both kinds of information. In the second part, I will introduce a radically different approach to recommendation that is based on the use of the opinions of experts instead of regular peers. I will show how this approach addresses many of the shortcomings of traditional Collaborative Filtering, generates recommendations that are better perceived by the users, and allows for new applications such as fully-privacy preserving recommendations.

BIOGRAPHY

Xavier Amatriain (PhD) is Research Scientist in Telefonica (Barcelona, Spain), where his current focus is on Recommender Systems, User Modeling, Data Mining, Social Networks, and neighboring Web Science areas. Previous to this, he was Research Director at the University of California Santa Barbara, where he worked on Multimedia and Immersive Systems and led the Allosphere project. During his PhD at the Universitat Pompeu Fabra, he was lead developer and coordinator of the award-winning CLAM project.

Xavier has authored more than 50 publications, holds several patents, and has been lecturing on Software Engineering, Information Retrieval, and Multimedia Systems in several universities for over 10 years. He currently teaches Information Systems for Managers at EADA Business School. He is also interested in software development and research management, and in particular in models of technology transfer and agile methods.

Tagged
2011 04 13 Sam tech talks Comments (0) Permalink