Stream Processing with Cloud Pub/Sub and Dataflow: Qwik Start




Create Project Resources

Start the cloud scheduler job

Start the pipeline and launch dataflow job

Stream Processing with Cloud Pub/Sub and Dataflow: Qwik Start

30 minutos 1 crédito


Google Cloud Self-Paced Labs


Google Cloud Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes messages to a Cloud Pub/Sub topic. A consumer creates a subscription to that topic. Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable window of time.

Dataflow is a fully-managed service for transforming and enriching data in stream (real-time) and batch modes with equal reliability and expressiveness. It provides a simplified pipeline development environment using the Apache Beam SDK, which has a rich set of windowing and session analysis primitives as well as an ecosystem of source and sink connectors.

Pub/Sub is a scalable, durable event ingestion and delivery system. Dataflow compliments Pub/Sub's scalable, at-least-once delivery model with message deduplication and exactly-once, in-order processing if you use windows and buffering.

What You'll Do

  1. Read messages published to a Pub/Sub topic

  2. Window (or group) the messages by timestamp

  3. Write the messages to Cloud Storage

Participe do Qwiklabs para ler o restante deste laboratório e muito mais!

  • Receber acesso temporário a Console do Cloud.
  • Mais de 200 laboratórios, do nível iniciante ao avançado.
  • Tamanho compacto para que você possa aprender no seu próprio ritmo.
Participe para iniciar este laboratório