PubSub BigQuery Subscription

Google Cloud has recently released a very neat feature to stream data directly from PubSub to BigQuery without needing to manage any pipelines in the middle. I've had some time to explore it in detail, and today I would like to share what I've done, the areas where it can be very useful but also some limitations can restrict the use cases of this new feature. Further reading - Slides: https://slides.com/richardhe/pubsub-bigquery-subscription - BigQuery Subscription: https://cloud.google.com/pubsub/docs/bigquery - Avro Logical Types: https://fastavro.readthedocs.io/en/latest/logical_types.html - Code repo: https://github.com/rocketechgroup/direct-pubsub-bq-stream