Bigquery is a fully managed, serverless data warehouse. This warehouse enables scalable analysis over petabytes of data. It is a platform as a service (Paas) that supports querying using ANSI SQL. Now sprinkle can read from Bigquery warehouse.
Sprinkle supports a wide range of data sources. A list of data sources will be shown in the Create datasource tab. In this case, on selecting the Bigquery DB tab, it requires the user to name it.
Post naming, it routes the user to the configuration page.Now the user needs to fill in the credentials such as Project Id, Dataset before testing the connection and updating.
Optimising Incremental Ingestion in BigQuery Database
Also users can select Yes or No to Optimize Incremental Ingestion. If optimize is Yes, all the datasets will undergo full ingestion on every Sunday or every night. If optimize is No, data will be ingesting incrementally and it never goes under complete ingestion.
Connecting to Google Oauth
Users need to authenticate and provide permission to Sprinkle for accessing BigQuery.
In the Dataset page according to the Google Authentication the datasets will be seen.
In the Datasets page, the user can add tables either by table or query method, in table method the user is required to apply a table name and filter clause could also be applied whenever required.
On selecting Query, the user must provide a table name and apply SQL Query before creating a table.
After adding the dataset, clicking on Create the user will get redirected to the Ingestion job page. Here the concurrency (number of tables that can run in parallel, a maximum of 7) can be set preferentially before running the job. The status of the job will be updated in the tab below once it’s complete. The jobs can also be set to run automatically by enabling autorun. By default, the frequency is set to every night. Frequency can be changed by clicking on More --> Autorun-->Change Frequency.