PostgreSQL is an open-source database that is used to keep custom structured data. This open object-relational database system uses and extends the SQL language combined with many features that safely store and scale the most complicated data workloads. As Sprinkle is capable of integrating from any data sources, it pulls data from the PostgreSQL source database and loads it into the data warehouse of preference.
Sprinkle supports a wide range of data sources. A list of data sources will be shown in the Create data source tab. In this case, on selecting the PostgreSQL tab, it requires the user to name it. Post naming, it routes the user to the configuration page.
After naming the data source, the configure tab would require the user to fill in the credentials such as Username, Password, JDBC URL and select between connecting via SSH Host or not before testing the connection and updating.
Optimising Incremental Ingestion in PostgreSQL
Also users can select Yes or No to Optimize Incremental Ingestion. If optimize is Yes, all the datasets will undergo full ingestion on every Sunday or every night. If optimize is No, data will be ingesting incrementally and it never goes under complete ingestion.
In Add Tables, the user can add tables either by table or query method, in table method the user is required to apply a table name and filter clause could also be applied whenever required.
On selecting Query, the user must provide a table name and apply SQL Query before creating a table.
In the Run and Schedule tab, the concurrency (number of tables that can run in parallel, a maximum of 7) can be set preferentially before running the job. The status of the job will be updated in the tab below once it’s complete. The jobs can also be set to run automatically by enabling autorun. By default, the frequency is set to every night. Frequency can be changed by clicking on More --> Autorun-->Change Frequency.