Senior Hadoop Systems Engineer
Klout - San Francisco, CA

This job posting is no longer available on 37Signals. Find similar jobs: Senior Hadoop System Engineer jobs - Klout jobs

We're looking for a seasoned engineer to help us create cutting edge systems and tools to manage, maintain and monitor all aspects of the Klout production, staging, and development environments with a focus on our Big Data systems. Our environment is an intense combination of modern software and tried and true technologies including and leveraging Hadoop, HBase, Hive, ElasticSearch, Scala, Storm, and Node.js. We'll challenge your scalability chops with our high volume consumer site traffic coupled with the Big Data challenges of processing over 12 billion data signals a day for over 400 million people on multiple petabyte clusters.

Here are the values we look for in a team member:
  • You believe in automating everything. If you are repeating the same task more than once, you believe it should be automated
  • You believe configuration management isn't just a good idea but a must
  • You prove your ideas with testing and metrics, no guesswork
  • No pointing the finger or blame -- we're all in this together and everyone owns site up
  • We're here to serve, both our customers and our team members -- we provide tools to help everyone succeed

Here's what you need to have:
  • 5+ years of experience in systems or database administration experience on Linux-based systems
  • 2+ years of experience installing, configuring, monitoring, and administering large Hadoop clusters, preferably on Cloudera CDH3 and CDH4
  • Experience or a strong understanding of Hadoop architecture including HDFS, Hadoop MapReduce, Hive, Oozie, HBase, Hue, ZooKeeper, and ElasticSearch
  • Experience with current technologies like HAProxy, memcached, and Redis
  • Strong understanding of best practices and standards for Hadoop application design and implementation
  • Strong knowledge of Linux, OS level file system and network troubleshooting
  • Experience with Java, Python, and Bash (Ruby, Scala, and C++ a plus plus)
  • Expertise in at least one large-scale RDBMS
  • Experience running a monitoring system -- and participating in an on-call rotation
  • Klout score of 10 or higher

The things you'll do:
  • Perform HBase maintenance, splitting regions and performing compactions as necessary
  • Monitor and tune Hadoop clusters for performance
  • Monitor and report data usage across the clusters
  • Monitor Hadoop clusters through Nagios and Graphite
  • Support development, QA, and operations during new system deployments
  • Evaluate and propose new tools and technologies to meet the needs of the organization
  • Backup and recover NameNode and Hive metadata
  • Perform system health checks and monitor logs to ensure data integrity and fix block corruption
  • Develop automated tools and scripts to perform routine cluster maintenance
  • Deploy, configure, monitor, and adminster different systems like Hue, Hive, and ElasticSearch in an automated fashion
  • Patch, test, and deploy proprietary builds of different open source systems, including HBase and Hive
  • Document configs and upgrade procedures
  • Create runbooks for troubleshooting, cluster recovery, and routine cluster maintenance

Here’s what to expect:
  • Competitive pay, 100% paid health premium benefits and 50% dependent premiums
  • 401k plan
  • Stock options
  • 3 weeks vacation
  • Catered lunches, snacks, drinks
  • Passionate people building great products
  • A fun environment where people get the job done
  • Appreciation of exceptional science and engineering
  • Doing something no one has ever done before

Klout began with a very simple idea: Everyone has influence—the ability to drive action. Klout built on this idea to show anyone how he or she can influence the world and its future.

Klout was founded in 2008 to empower everyone to unlock their influence. We come to work every day inspired to help people understand the power of their voices and democratize influence.

Klout is a privately held company based in San Francisco, Calif. We’re a growing startup backed by Kleiner Perkins, among others.

37Signals - 17 months ago - save job