Avatar

Mortar Blog

@mortardata / mortardata.tumblr.com

News, updates, and technical posts on Mortar, data science, data engineering, and the Hadoop ecosystem.
Avatar

Mortar Is Joining Datadog

We have some big news to share: Mortar is joining Datadog, the leading SaaS-based monitoring service for cloud applications. We will be putting our team and our technology to work inside Datadog to expand their analytics capabilities and generate new insights for their users. Datadog processes hundreds of billions of data points every day from servers and applications, and we are thrilled to have the opportunity to apply what we have built at Mortar on that scale.

This is a big change for us, and although we are working hard to limit the impacts on our customers, it will be a change for them, too. As we transition into Datadog, we will be winding down the public Mortar service over the next few months. If you’re currently a Mortar user, you will have continued access to your account until May 15, giving you time to transition to new tools and to transfer or back up your Mortar code.

We have created guides on how to transfer your code, whether it is stored in Web projects or in GitHub-backed Mortar projects. And we’ve created a guide to running Mortar code outside the Mortar service using Luigi, the workflow engine developed and open-sourced by Spotify. Ensuring code portability for our customers is one of the main reasons why we built the Mortar platform using the best available open technologies and languages.

We're incredibly grateful to all of you who have been our customers, who have helped spread the word about Mortar, or who have offered invaluable advice as we built this business over the past 3.5 years. We hope you have enjoyed using Mortar as much as we enjoyed building it. We could not have done it without your support, feedback, and encouragement.

We also hope you’ll keep an eye on what we’re doing in our new digs at Datadog. It ought to be a very exciting 2015.

Avatar

Hadoop Weekly - February 9, 2015

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

Avatar

Hadoop Weekly - February 2, 2015

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

Avatar

Hadoop Weekly - January 26, 2015

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

It was a busy week of announcements and releases—Google and Hortonworks announced a new integration for HDP on Google Compute Engine, Google and Cloudera announced a joint project to bring a Spark backend to the Google Dataflow SDK, Netflix announced a new open-source project, and Apache Flink 0.8.0 was released. In addition, there are articles on machine learning from PayPal and Databricks as well as several other high quality posts on Kafka, HBase, and more.

Avatar

Spark as a Service

As of today, Apache Spark has been integrated into the Mortar platform and is available for use by customers on request.

We always strive to deliver the best data technologies in a platform that is rock-solid, scalable, and easy to use. Our customers have already done incredible things with our existing technologies—Pig, Luigi, Python, R, Java—and we are excited to see what they can achieve with the addition of Spark.

Avatar

Hadoop Weekly - January 19, 2015

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

We have a lot of great technical posts this week covering security in Hadoop, recent improvements to Spark, HDFS’ new heterogenous disk support, and the relatively new Apache incubator project NiFi. In addition, Apache Flink graduated from the incubator, Etsy open-sourced a new tool for profiling Hadoop jobs, and much more.

Avatar

Hadoop Weekly - January 12, 2015

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

The first full week of 2015 has been fruitful for Hadoop-related content (especially technical). There are a number of great posts, including several from the Data Day Texas conference as well as coverage of Drill, Spark, Storm, and more.

Avatar

Building Data Pipelines using Luigi, with Erik Bernhardsson of Spotify [Meetup Video, Slides, Transcript]

John Matson

At our most recent NYC Data Science Meetup, we heard a great talk by Erik Bernhardsson from Spotify. Erik gave a comprehensive overview of Luigi, the workflow engine that he, Elias Freider, and other Spotify engineers built and open-sourced. Spotify uses Luigi to run thousands of Hadoop jobs a day, and to automate all kinds of complex data pipelines involving multiple systems. It's also used by many other cutting-edge companies such as Asana, Buffer, and Stripe.

We at Mortar were especially keen to hear what Erik had to say, both because we are devoted Luigi users and because Luigi is one of the core technologies in our platform. With Luigi on Mortar, our customers can build multi-stage data pipelines with complex dependencies, execute them in the cloud, and schedule them to run as often as needed.

Check out the video and slides from Erik’s talk, embedded below, to see what Luigi is all about and why it’s so useful.

Avatar

Hadoop Weekly - January 5, 2015

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

The first issue of 2015 starts off the year relatively quiet, although there are a number good technical and industry news articles. Technical articles cover the Kite SDK, Kafka, HBase, Hive, and Sqoop; news articles include a number of year-end synopses. Based on these, it looks like 2015 will be an interesting year for the Hadoop ecosystem.

Avatar

Hadoop Weekly - December 29, 2014

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

For the last issue of Hadoop Weekly in 2014, we have a short but sweet edition. The theme for this week is the future of Hadoop—from the first technical post about Apache NiFi to posts on younger security projects (Apache Ranger and Apache Sentry) to several posts about the Hadoop industry in 2015.

Avatar

Hadoop Weekly - December 22, 2014

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

This week’s newsletter marks the milestone of issue 100, which goes out to over 4,000 recipients. The ecosystem has changed a lot in the last 100 issues, which you can see from this week’s coverage of Storm, Spark, Slider, and Kafka. Thanks to everyone that’s contributed great content to this newsletter in the past—please keep it coming!

Avatar

Hadoop Weekly - December 15, 2014

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

It was quite a busy week for funding news with the Hortonworks IPO and VC funding for Altiscale and Qubole. There were also quite a few new releases (Oozie, Phoenix, Samza) as well as several posts describing recent releases in more detail. And given that we’re halfway into December, the first of 2014-retrospective and 2015-predictions for Hadoop are in this week’s issue.

Avatar

Hadoop Weekly - December 8, 2014

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

There’s a lot to cover from the past two weeks, including several new releases: Apache Hadoop 2.6.0, Apache Ambari 1.7.0, and updates from Cloudera, MapR, and Hortonworks. Technical posts from LinkedIn, Pinterest, and Spotify all give a glimpse into the data infrastructure of those companies. There’s also coverage of the Hortonworks IPO and Apache Drill, which graduated from the Apache incubator this week.

Avatar

Hadoop Weekly - November 24, 2014

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

Lots of news out of Europe this week as both StrataConf Barcelona and ApacheCon EU took place. In addition to several interesting presentations from those conferences, this week’s issue contains several articles on Spark and YARN. Also, Stripe has open-sourced four Hadoop tools, and there were releases from open-source projects Apache Pig and Apache Chukwa as well as vendor releases by Splice Machine and HP’s Vertica. One quick editorial note—with the short week in the US for Thanksgiving, I’m going to skip next week’s issue and resume with issue #98 on December 7th.

Avatar

Hadoop Weekly - November 17, 2014

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer focused on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

Big news this week out of Palo Alto as Hortonworks has filed paperwork for an initial public offering. There were also a number of notable releases this week, including Apache Hive 0.14.0. Technical posts cover a large number of ecosystem topics, including Apache Sqoop, Apache Drill, and Apache Pig. There’s a lot of breadth in this issue, so there should be something for everyone!

You are using an unsupported browser and things might not work as intended. Please make sure you're using the latest version of Chrome, Firefox, Safari, or Edge.