Tags: google

BigTable(next generation database led by Google) 1

By MD on Jun 13, 2008 | In Database | Send feedback »

BigTable. It could be pretty big as it sounds. According to Jeffry Dean, who is a fellow at Google, the biggest one today is up to 4000TB, spanning over thousands of servers..., only a table!!!

Have you ever heard about BigTable? Unless you're a database vendor or a Google infrastructure freak, I'm afraid you haven't.

According to the paper titled Bigtable: A Distributed Storage System for Structured Data, it is like:

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving).

Two points.

  • scale to a very large size: petabytes of data across thousands of commodity servers
  • both in terms of data size and latency requirements

How?

Performance matters

To reduce HDD seek time is a keen point to general performance of computer since I/O costs million times more than CPU's. So, it's wise to fetch as big chunk as possible at once.

Today, most of user files are getting larger and larger, ever larger. But still, a unit of HDD stays smaller, 512KB usually, and 512KB-4KB of filesystem on Linux.

Google File System, Google's underlying distributed filesystem, makes use of a huge chunk, 64MB in size.

The next thing to consider is layout. How data should be laid on a block? Contiguous data can be read/written from/to disk at once.

A database usually put one row on a contiguous space. So as long as you put all the data you require on a single record, you can get the best performance. Some databases provide another approach, column oriented.

BigTable is not a conventional table

It's more like a spreadsheet. And a map under the hood.

BigTable offers a new way both in performance and functionality. Next time, I will show you details.

Tags: bigtable, google

Visualize Your Data with Google Visualization API

By MD on May 30, 2008 | In Ease Of Use | Send feedback »

Today, here at Google I/O 2nd day, a new Google Visualization API was announced.

What's new?

1. events
2. gadget.draw()

Google Visualization Gadget supports selection events so that a developer can respond to end users.

With draw() method, a gadget can draw any tables only if a table supports pre-defined DataTable APIs.

Let's try the new draw() method with the JSON example.

Instead of writing tables with HTML tags, simply pass DataTable object to Table gadget.

Here's the code.

Code:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
  <head>
    <script type="text/javascript" src="http://www.google.com/jsapi"></script>
    <script type="text/javascript">
    
    google.load("visualization", "1", {packages:["table"]});
    
    var xhr;
    try {  xhr = new ActiveXObject('Msxml2.XMLHTTP');   }
    catch (e)
    {
        try {   xhr = new ActiveXObject('Microsoft.XMLHTTP');    }
        catch (e2)
        {
          try {  xhr = new XMLHttpRequest();     }
          catch (e3) {  xhr = false;   }
        }
     }
 
   xhr.open("GET", DATASOURCE_URL,  true);
   xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
   xhr.send(null);
  
   xhr.onreadystatechange  = function()
    {
         if(xhr.readyState  == 4)
         {
              if(xhr.status  == 200){
                  the_object = eval( "(" + xhr.responseText + ")" );
                  handleQueryResponse(the_object.table);
              }else{
                
              }
         }
    };
 
      // Query response handler function.
      function handleQueryResponse(table) {
        var data = new google.visualization.DataTable();
        
        // convert to Google.DataTable
        // column
        for (var col = 0; col < table.cols.length; col++) {
          data.addColumn('string', table.cols[col].label);
        }
        // row
        for (var row = 0; row < table.rows.length; row++) {
          data.addRow();
          for (var col = 0; col < table.cols.length; col++) {
            data.setCell(row, col, table.rows[row][col].v);
          }
        }
        
       var vis_table = new google.visualization.Table(document.getElementById('table_div'));
       vis_table.draw(data, {showRowNumber: false});
        
      }
 
    </script>
  </head>
 
  <body>
    <div id="table_div">Loading...</div>
  </body>
</html>

Thanks a lot, Google Visualization Team! Now that our own data can be easily integrated with Gadget!

Tags: google, json, restfull, visualization, web servise

Google?????????????

By MD on Apr 22, 2008 | In Database, ??? | Send feedback »

??4?22?????????????????????????????????????????????????Google?????????????????????????????????????????????????????????????

???????????????
?????????????????????????????????????2003??15?kwh(????????)??????

???????????????

?????????Google???????????????????????????????????????????????Google???????????????????????

2004??????5?????2007?????50????????????????????????????????????????????????Google????????????????????????Google????????????????????????

????1??????????????120w?24??????????????????????????????1?????????????????????????????

120w * 24 * 1.5 * 365 = ?1500kwh

????50?????

1500kwh * 500,000 = 788,400,000kwh

?8000?kwh!!!

2003??15?kwh?2010??????????20?kwh???????2007??17?kwh????????????????2007?????????????????Google?????????????

800 / 17,000 * 100 = 4.7%

5%??

?????????????????????????????????????????????????????????????Google?????????????????????????

So much data, relatively little space??????????????2003??5??????(exa?10???)????????IDC??????????2006???40????????????????????????????????????????????????3???10??????????????????3???8??????????????????????????????Google????????????????????????????????

IDC?????????????????????2007??161?????????2010???988?????????????????3???6?????????????Google??????300??????????5?kwh!!!????2010??????????????20?kwh????????25??Google???????????

???????????????????????????????????????????????????????????????????????????2015???2020???????????????Google??????????????????????

???????????????????????5%?Google?????????????????????????????????????10??????????0.5%?????????

Tags: google