Getting Started With Grafana

Grafana

Preface

When I first saw Grafana, I was astonished by its beauty immediately and I believe it should be the very tools for most dashboard / monitor use cases. This 10,000+ stared project provide a complete solution for metrics and analytics.

Check this live demo and you should feel the same pleasure as I do.

How does Grafana work

In short, Grafana is a metric solution which include UI Render / Query / Data Source features.
UI Render is amazing; Different types of Queries are supported and it provide a wide range of Data source (mostly time series DB) support.

So, to setup a Grafana for our own use, what should we do?

  • First we need to setup the data source.
  • Second, we setup Grafana server and connect it with the data source.
  • Thrid, we feed data into the data source

That’s it!

Setting up

The following article will cover how to set all the stuff up under Mac OSX. Although we have homebrew and pip, but it take more effort than it seems to be.

Graphite as Data Source

First of all, we are going to choose the data source first. Grafana provide a lot of DB support including: Graphite, Elasticsearch, CloudWatch, InfluxDB, OpenTSDB, KairosDB, Prometheus,

I will take Graphite as our primary data source, with the following reasons:

  • powerful data APIs
  • friendly render APIs with image accessing
  • using whisper file to store data, operation friendly
  • overall the design of Graphite is very clean, every layer of the design is scalable.

Read More

Unit Test 101

The importance of writing unit tests have been heavily discussed. However, reality is cruel. Many developers do not write UT, and even larger amount of developers write funny tests just to pass the code coverage bar, and some developers with ambition are not quite sure about how to write good unit tests. Most good UT are written by top developers.

I thought I am that kind of ambitious developer, so I spend two weekends to start to learn it. Most cases listed below are originated from or revised from the book “Practical Unit Testing with Testng and Mockito” by Tomek Kaczanowski. A really good book worth reading.


Concepts

Before we start, let’s visit the concepts of SUT and DOC.

Read More

Evolution of Metric System Architecture

Preface

In the past 2 years, I spent about 70% of my working time to build, to break, and to fix data products. This article is a brief retrospect of my understanding on building the whole systems, as well as what kind of tools could be plugged as components.

Goal of a data System

We use data to understand reality and improve our product. This is the primary goal of a data/metric system. A good data system answers question, a better data system identifies root causes, and an even better data system help improve the whole system directly.

Use cases

In Yahoo!, the data platform I am working on mainly support a Personalization System (Recommendation system). During the iteration of the recom system, we follow and forecast what would be the actual use cases for the team to understand or to improve the Recom system. The major use cases for our system includes:

  • Understand system performance with reports from different key metrics
  • Detect / identify metric abnormal / data pipeline failure
  • Collect user feedback data to improve system online in short cycle
  • Make it easy for PM/Dev/Scientist to play with data

For different stage, we focus on different aspect and use different tools / techniques to solve problems. Let me illustrate.

Read More

Git: under the basics


My boss told me that my goal in this quarter is to working on Continuous Integration for our current product, and all of a sudden I think there’s a lot of gap between the goal and my current skill. The first thing came into my mind is that: “Ohhh, I am still not quite familiar with Git”. After a short period of panic, I sit down to learn about git. And here’s my note.

If you think you could learn git with manual after you learn how to branch, commit and merge, then you might probably be dispointted. Git is very flexible but it do something in a more novel way, so certain understanding of it’s internal is necessary for mastering it, and would be helpful when you look for help in manual. For example, I hear about so many terms such as “HEAD”, “Index”, “Ref”, “Staging Area”, but I could not tell exactly what is that, and I don’t even know how git works. After some diving, I wrapped something very basic in this post.

Read More

Sync two Git remote repositories

In Yahoo we use Gerrit as our code review tool. Engineers commit code changes to Gerrit for review. After code has been reviewed by peers, Gerrit help push to Github. However sometimes bad thing happens.

For example, if a committer forget to setup Gerrit env and the code is committed and pushed to Github directly (because he has this permission to do so), he will notice this soon because future changes from Gerrit might break. Then he try to submit a review for the missing commit to Gerrit with an “git commit —amend” to generate a change-id (which is used by Gerrit). Because “—amend” generate different commit-id, so after the review is passed, even the content is the same, the commit-id in two remote repo (Gerrit and Github) is different, which leads to future reviews are still not able to pushed to Github from Gerrit and sometimes new review could not be submitted.

So how could we fix it? It would be a good idea to push changes from Github to Gerrit to get them in sync. Here are two options.

Read More

Recommended workflows in Alfred

Finally I purchase Alfred for workflows.

A Workflow is a combination of actions, and the killer feature in Alfred powerpack. In Alfred, we can import existing workflows or create our own workflows — to run a series of actions, which will dramatcially improve productivity.

My workflow lists

Before writing this post, I spent about half an hour on Alfred Workflows to go through all existing workflows based on the AlfredWorkflow repo by hzlzh and select following workflows as enhancement of Alfred.

There is another workflow repo by @zenorocha.

And here’s my lists.

Read More

Apache Pig in Practice 1


I write many pig script in the past few months and have explored some tricks with my buddies. hopes it could help someone.

Let’s focus on some interesting topics in this first article and get prepared for the later Pig rush.

IDE & Environment

Vim

I use Vim to write most script language and those are my favourite plugins to write Pig:

  • Pig Syntax Highlight. Latest update on Jun 2014, Pig 0.12 supported.
  • You complete me. Best auto-complete plugins ever. If you don’t use a MAC, Supertab is also a reasonable choice.
  • Tabularize Align and keep cleaness of the Pig codelet. Most common usage is :Tab/AS to align FOREACH ... GENERATE clause.

To improve debug efficiency, I like to run pig with short cut. Here are my simple approach: add the following in .vimrc for quick run with F5

1
2
3
4
5
6
7
8
9
10
11
12
13
14
map <F5> :call Compile_Run()<CR>
function Compile_Run()
if &filetype=="coffee"
:w
!coffee % 2>&1
elseif &filetype=="cpp"
:w
!g++ -g -o %< %; ./%<
elseif &filetype=="python"
:w
!python %
elseif &filetype=="pig"
:w
!./run_pig.sh %

revise run_pig.sh as you like. General idea is reduce redundant work and typo.

Read More