Create Project Resources
Start the cloud scheduler job
Start the pipeline and launch dataflow job
Stream Processing with Cloud Pub/Sub and Dataflow: Qwik Start
Google Cloud Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes messages to a Cloud Pub/Sub topic. A consumer creates a subscription to that topic. Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable window of time.
Dataflow is a fully-managed service for transforming and enriching data in stream (real-time) and batch modes with equal reliability and expressiveness. It provides a simplified pipeline development environment using the Apache Beam SDK, which has a rich set of windowing and session analysis primitives as well as an ecosystem of source and sink connectors.
Pub/Sub is a scalable, durable event ingestion and delivery system. Dataflow compliments Pub/Sub's scalable, at-least-once delivery model with message deduplication and exactly-once, in-order processing if you use windows and buffering.
What You'll Do
Read messages published to a Pub/Sub topic
Window (or group) the messages by timestamp
Write the messages to Cloud Storage
Participe do Qwiklabs para ler o restante deste laboratório e muito mais!
- Receber acesso temporário a Console do Cloud.
- Mais de 200 laboratórios, do nível iniciante ao avançado.
- Tamanho compacto para que você possa aprender no seu próprio ritmo.