Big data analytics case study: Ethereum transactions, wallet activity and contract events, ready to query as they happen

The challenge: keeping up with Ethereum while preserving its history

The client wanted live insight into Ethereum transactions and other activity on the chain, room for detailed historical analysis, and an app that stayed easy to use as the chain's data kept growing. The platform needed to meet five requirements:

• Nodes that agree. Each node pairs an execution client with a consensus client. The two had to be configured to work together, so reading data didn't mean long delays, and several nodes had to stay in sync so the data stayed intact and reachable.
• A continuous stream. Transactions arrive constantly and in volume, and the platform had to follow them in real time.
• Blocks that change. When the chain reorganises, blocks already read can be replaced, and the data has to stay correct through the change.
• Fast answers from a growing history. Analysis needed fast queries over the chain's data, and storage that kept retrieval quick as the dataset grew.
• A manageable operating workload. Processing had to continue with minimal interruption, need hands-on fixes as rarely as possible, and keep setup and syncing straightforward for the team operating it.

What we built: big data analytics from synced nodes to fast queries

We built the node set-up, the pipeline, the storage, the analytics and the app. Six pieces carry it:

• Synced nodes. Geth (execution) and Prysm (consensus) run together, so the nodes stay coordinated, and Erigon nodes support further analytics. The same set-up pulls extra data from the chain for analysis.
• Queues for a constant stream. Redis and BullMQ run the queues that take in the chain's flow of data, so transactions are processed with few or no stops, and updates reach the app in real time.
• Storage that stays fast. Astra DB stores the indexed chain data and retrieves it quickly as the dataset grows, behind a Node.js back end.
• Models for chain behaviour. Mathematical models analyse what happens on the chain, from smart contract actions to reorganised blocks, and OpenAI's models are used for advanced analytics.
• Infrastructure as code. Terraform builds the infrastructure and sets up data replication, which cut manual work and made new data sources simpler to add.
• An app for large datasets. A Next.js and Material UI front end with a minimal interface makes big blockchain datasets easier to work with. Bugsnag reports errors.

Where the pipeline stops, and who takes over

The pipeline and its models turn the chain into data people can query. What that data means, and what to do about it, stays with the people using the platform.

Results

The platform's users see fresh data from Ethereum, from transaction history to the details of smart contract events, and the back end stays reliable as the dataset grows. The table says what changed rather than by how much.

Technologies and tools

Timeline

The build and ongoing development spanned 30 months.

If your AI work is waiting on data that arrives too late to be worth reading, our data engineering services design the pipeline backwards from whatever will read it, and ship a data contract and a quality gate with every dataset.