Google recently announced preemptive support for the ability to transfer information via its Storage Transfer Service (STS), which allows users to move data from AWS S3 to Cloud Storage and copy data between multiple Cloud Storage buckets.

STS is a Google Cloud service that allows users to quickly and securely transfer data between object and file storage in Google Cloud, Amazon, Azure, on-premises, and more.
Additionally, the service now includes the ability to preview automatic transfers of data that has been added or updated in the source location. This type of transfer is event-driven, as the service listens for notifications about them to initiate the data transfer. Currently, these event-driven transfers are supported from AWS S3 or Cloud Storage to Cloud Storage.

STS relies on Pubsub and SQS to perform event-driven transfers. Customers need to set up the event notification and provide STS with access to this queue. Using a new field – “Event Stream” – in the transfer task, clients can set an event stream name and control when STS starts and stops listening for events from that stream

Once the transfer job is created, STS begins consuming object change notifications from the source. Any change or upload of an object will now trigger a change notification on which the service acts in real time to copy the object to the destination.

With event-driven transfer, automatic, ongoing replication of data between buckets can be set up with just a few mouse clicks. It comes with a number of supporting features including filtering based on prefixes, retry handling, checksums, detailed transfer logs via Cloud Logging and progress tracking via Cloud Monitoring. In addition to being easy to set up and use, it is extremely scalable and flexible, allowing data to be replicated between buckets within the same project, between different projects, or even between different regions and continents of Cloud Storage.

Event-driven transfers can be created between Cloud Storage buckets in three steps:

• Create a pubsub subscription that listens to changes in the Cloud Storage bucket
• Giving STS permission to copy data between buckets and listen to that pubsub subscription
• Create a transfer job with an event stream configuration

Event-driven transfer is an extremely powerful solution that can help users automate data transfer and processing tasks, saving time and resources.