As a Systems Engineer on our BFD Team (Big Fast Data), you will be responsible for designing, developing and scaling our real-time data ingestion platform. This will involve designing and developing high performance, scalable and robust systems for real time distributed data-stream processing and storage. This will also involve implementing large-scale stream acquisition platforms to bring in social media and transactional firehoses for processing and will involve leveraging and contributing to open source software whenever possible (Cassandra, Gearman, Hadoop, etc).
Challenges include: distributed processing and messaging, robust distributed storage, data-stream computational pipelines, real-time indexing, multi-level caching, capacity planning, and performance tuning.
- help invent the next generation of ecommerce; integrated experiences that leverage the store, the web and mobile, with social identity being the glue
- work with the world’s best engineers to design, prototype, and develop high performance distributed systems that ingest, process, store, and manage terabytes of real-time streams of social and commerce data – the Twitter firehose, Walmart transaction firehoses, and many more
- be at the forefront of solutions for distributed processing and messaging, robust distributed storage, data-stream computational pipelines, real-time indexing, multi-level caching, capacity planning, and performance tuning
- work with and further develop open-source technologies including Hadoop, Cassandra, and Gearman; create Hadoop-like data streaming applications
- BS/MS/PhD in Computer Science, Computer Engineering, Electrical Engineering or related field
- 2-5+ years of Programming experience in C/C++/Java/Perl/Python
- Extensive Linux systems programming experience
- Expertise in distributed/scalable systems and algorithms with awareness of time and space complexity
- Strong background in process design for reliable systems; strong “big picture” awareness of systems
- Good generalist experience, ideally working with all layers in the tech stack; experience working with operations and creating software libraries/frameworks
- Working knowledge of standard tools for optimizing and testing code; plus to have experience building tools/apps and/or working with custom-built tools
- Past experience handling large amounts of data on distributed platforms is also a huge plus
- Bachelor's Degree and 6 years experience building scalable e-commerce applications
Staff Software Engineer - WEC
Walmart Stores - 19 months ago
Wal-Mart Stores, Inc., branded as Walmart since 2008 and Wal-Mart before then, is an American public multinational corporation that runs...