係屋企點練PySpark/Kafka

有無人知邊到可以搵到resource俾你practice?大學學過但實作經驗少，公司又唔點用

Kafka自己local就set到
係Spark難set d

zoomcamp self pace 做
https://github.com/DataTalksClub/data-engineering-zoomcamp
eg https://github.com/samsonafo/dtc_dezoomcamp_project

databricks 都有community version
以前edx / coursera mooc 玩過下

kafka 唔駛用confluent kafka 練啦

spark kafka 都可以放係kubernetes

唔知做乜topic 就NYC Taxi 做source data
medium searh NYC Taxi data project 一大堆

多謝巴打
咁啱呢排要用spark
跟住嚟學下先

lm

巴打仲有無多啲關於其他skill既正source share

git clone https://github.com/pingcap/tidb-docker-compose.git
cd tidb-docker-compose && docker-compose pull
sudo setenforce 0
docker-compose up -d
docker-compose exec tispark-master /opt/spark/bin/pyspark

其實Spark都有得Stream data，一直唔明點解出面市場要人識兩樣。

kafka 寫落碟 perpetual keep 住啲 data喎

Spark點解唔得？

https://databricks.com/blog/2017/01/19/real-time-streaming-etl-structured-streaming-apache-spark-2-1.html?_ga=2.126652040.408463605.1649528105-1193307410.1647889094

spark.read.schema

都可以keep到data

最簡單就係一直用開無人想轉， user 又無 request

公司點樣唔畀用？ 純粹唔想invest新tech stack 定legal 或者錢問題？

試下自己幫公司諗idea 跟住整個MVP睇下佢哋buy唔buy？