The poor man's data pipeline
May 17, 2017
Tasked with building out a shiny new data pipeline in May of 2017, I devoured every bit of experience I could find from colleagues, meetups, and research. My criteria were low operational responsibility, high flexibility for future development, and that the thing be simple enough that I can implement it myself. If you know anything about the landscape of big data, you know solutions in this space aren't exactly known for their low maintenance and simplicity. Just read some of the blogs of the big players like Spotify, Netflix, and Yelp. Yes, these are complex in part because the companies are massive but I think you'll find every modern case study looks similar. I was convinced there was an easier way to get 80% of the functionality for 20% of the work.