Connecting Batch to Hightouch
Modern data architectures demand seamless integration between storage, processing, and actionable platforms. Connecting Batch to your Data Warehouse enables a unified, efficient, and scalable approach to leveraging customer data for engagement.
Hightouch is an excellent tool to send your data to Batch and works great with Snowflake, BigQuery, Redshift, Databricks, ClickHouse, and many more.
By doing so, you ensure keeping your Data Warehouse as the Single Source of Truth, you have complete control over your data update frequency and you can leverage huge data volumes.
Step-by-step guide
In this tutorial, we will send user data to Batch’s Profile API.
We will be sending attributes and events to each of your users profiles.
1. Create your data source in your Data Warehouse
The first step is to organize your data in order to have a table storing your user data, transformed and ready to be sent to Batch. NB : Views will work as well.
In our example, we will want to have a table with one row per user.
We will call it hightouch_batch_profile_base in this example.
We recommend you use a dedicated database and schema to store the data destined to be sent via Reverse ETL.
2. Connect Hightouch with your Data Warehouse
Make sure Hightouch has access to your Data Warehouse, see this Hightouch basics tutorial.
3. Create a model in Hightouch
Creating a model is basically telling Hightouch where to go fetch the data in your Warehouse.
We will call it batch_profile_base in this example.
A model could be made with a more complex query but we recommend you keep your complex logic in your Data Warehouse, with your usual transformations methods.
4. Create your Batch Destination in Hightouch
This is where you will configure the access to your Batch Project.
You will need API Keys available in your Batch Dashboard > Settings.
In Hightouch, create a Destination and use the Batch Profile API Update URL and headers :
Authorization → Your REST API Key (use the “Secret” toggle to keep your REST API Key secret)
Content-Type → application/json
X-Batch-Project → Your Project Key
Once this is done, deal with your internal permissions and your Batch Destination is ready!
5. Create your Sync in Hightouch
Syncs declare how you want your data to appear in Batch and on what frequency you want updates. This is the section where you will have to make choices.
Add Sync > Select your model > Select your destination
Here comes the mapping configuration. When you trigger a sync, Hightouch will identify all changes on your table. You will have to choose what to do with rows added, rows changed and rows removed.
For now, we will focus on rows added and rows changed. You will have to implement the same logic for both.
NB : Rows removed may be used to delete a user profile according to your policy. You can have a look at Batch GDPR API.
Second choice : You will have to choose between single row updates or batch updates with up to 10 000 rows. We recommend you use batch updates. You will still be able to debug your profiles one by one with the Hightouch logs.
Webhook endpoint : POST
Payload type : JSON (application/json)
There, three options again
Use JSON editor : You will have to paste the structure of a valid update payload and replace the values by the column names of your table. NB : column names are case sensitive. You will have to update the JSON payload in the sync configuration interface for every new attribute or event you want to sync.
Use one column from model : This option requires the JSON update payload to be stored in a column of your table. Maybe your data team will prefer this option. You will have to update the JSON payload in your data warehouse for every new attribute or event you want to sync.
Use multiple columns from model : You will have to map fields to recreate the JSON payload without having to write it. We prefer previous options.
JSON Payload example with batching (JSON editor option)
JSON Payload example with batching (JSON editor option)
[ {% for row in rows %} { "identifiers": { "custom_id": "{{ row.CUSTOM_ID }}" }, "attributes": { "$email_address": "{{ row.EMAIL }}", "my_first_attribute": "{{ row.MY_FIRST_ATTRIBUTE }}", "my_second_attribute": "{{ row.MY_SECOND_ATTRIBUTE }}" }, "events": [ { "name": "{{ row.MY_FIRST_EVENT }}" } ] }{% if forloop.last == false %},{% endif %} {% endfor %} ]
Rate Limit : Yes, 50 calls per second
Concurrent Requests : No input
The last three parameters to set are up to you. You may choose immediate retry or wait for the next sync run.
6. Test & Deploy
We recommend you test your sync with Hightouch testing module. Remember to test both added rows and changed rows. The API Response should guide you through debugging if needed.
Once your test is successful, schedule your sync, deploy it and enjoy having your profile base completely up to date on Batch.
You can use Batch’s Profile View to verify that the customer data has reached its destination, on the basis of a customer ID: How to find a user's profile on the dashboard?
You’re now ready to use your data to send personalised and engaging messages to your audience 🎉