{"id":603,"date":"2021-01-30T10:34:12","date_gmt":"2021-01-30T10:34:12","guid":{"rendered":"https:\/\/datascientists.info\/?p=603"},"modified":"2021-01-30T15:37:52","modified_gmt":"2021-01-30T15:37:52","slug":"keep-your-data-in-the-cloud","status":"publish","type":"post","link":"https:\/\/datascientists.info\/index.php\/2021\/01\/30\/keep-your-data-in-the-cloud\/","title":{"rendered":"Data Infrastructure in the Cloud"},"content":{"rendered":"\n

Having your data infrastructure in the cloud has become a real option for a lot of companies, especially since the big cloud providers have a lot of managed services available for a modern data architecture aside from just a database management system.<\/p>\n\n\n\n

In this article I will look into what the three big cloud providers offer in tools to support a modern data architecture.<\/p>\n\n\n\n

Modern Data Infrastructure Setup<\/h2>\n\n\n\n

A generic data infrastructure looks like the picture shown below. It contains one data entry point, where all data is collected as the general API for all producers of data. This data is pushed in single events, whenever possible and defined using a schema. All of the schemas are defined and stored in a schema registry.<\/p>\n\n\n\n

Following this entry point there is the possibility to do some ETL, like collecting several messages for one event into bigger chunks. This is useful for writing data out onto a storage or inserting into an analytical database. As these do not handle single row operations as well as bulk ones. Storing raw event data on a cloud storage is also possible and then access these files using SQL from the DWH system. This potentially saves more expensive storage in the DWH system.<\/p>\n\n\n\n

After the initial ETL transformations done on the events it seems sensible to do all the other transformations from the source using the compute power of the database system. As this can be done by using SQL, this offers the possibility to have people like analysts with SQL knowledge participate in the process. No other programming skills are needed from this point on.<\/p>\n\n\n\n

\"Modern
Modern Data Infrastructure<\/figcaption><\/figure>\n\n\n\n

Next we will look into the tools offered by the three largest cloud vendors that support this setup. <\/p>\n\n\n\n

Google Cloud Platform<\/h2>\n\n\n\n

The Google Cloud Platform<\/a> offers tools to implement this architecture, as seen in the image below.<\/p>\n\n\n\n

\"Modern
Modern Data Architecture in the Cloud Googles version<\/figcaption><\/figure>\n\n\n\n