Scraping Mining of Cryptocurrencies
Investor clients need to have personalized reports about the monitoring of operations on the cryptocurrency platform. We are going to help with this case.
- Industry
- Other
- Solution
- Data Engineering
01 / 03
The challenge
Investor clients need to have personalized reports about the monitoring of operations on the cryptocurrency platform. The data to generate these reports is found in "Hive OS", "Hiveon Pool", but the extraction and consumption of information become more ocmplex:
- Not easily accesible
- There is no historical information
- Data is not centralized
02 / 03
What Tekne did
Objective
Extract the data from the "Hive OS" and "Hiveon Pool" platforms and make them available in a centralized databases, which serves as a supply for the preparation of personalized reportasthat allow visualizing the main indicators such as the income and status of the rigs used for crypto mining.
Implementation Process
1 APIs survey
2 External data sources:
- USDT/ETH Price
3 Python Script development:
- Incremental ETL
- Data upload to AWS
4 Automation and orchestration using Airflow: freq. 1 hour
03 / 03
Results
- Historical data of Hashrate, Payouts and Temperature with incremental load every 1h, centralized in BBDD of AWS
- Automatic mapping of new entries (workers / farms added)
- Handling of errors and alerts
