Data engineering
Data Engineering Services for AI Applications
Data engineering services for teams whose AI programme has stalled on access, quality or permissions rather than on models. We design the pipelines, warehouses and retrieval stores backwards from the agents and models that will read them, then ship a data contract and a quality gate with every dataset we hand over.
- 01Extractread on scheduleCRM, ERP
- 02Contract checkschema matches the contractpass
- 03Quality teststhe owner is alerted on a failurepass
- 04Lineageevery hop tracedrecorded
- 05Publishaccess rules appliedwarehouse
- First assessment
- 2–3 weeks
- Ships with a contract and a quality gate
- Every dataset
- AI engineers
- 30+
- Years building AI systems
- 6
The data stack
Platforms we build on
SnowflakeWarehouse
DatabricksLakehouse
BigQueryWarehouse
dbtTransformation
Apache AirflowOrchestration- SnowflakeWarehouse
- DatabricksLakehouse
- BigQueryWarehouse
- dbtTransformation
- Apache AirflowOrchestration
01 / Scope
What data engineering services cover, and where they stop
Data engineering services are the work of getting data out of the systems that produce it and into a shape something else can rely on: pipelines that run on a schedule and tell you when they fail, models that hold their meaning as the business changes them, stores that answer the query pattern the consumer actually has, and the permissions and lineage that make the result safe to act on.
Most AI projects that stall on data do not stall on volume. They stall because access is unclear, trust is low, permissions do not survive the trip into the index, or the data arrives too late to be useful. Those are the conditions our data engineering services address.
Where data engineering services stop: we build the layer, we do not build the model on top of it. Training, evaluation and deployment of models belong to machine learning development, and the agents and retrieval systems that read the layer belong to the agent and generative AI hubs. If your problem is the model, start there instead.
- Typical first engagement
- 2–3 weeks
- A consumer-first assessment: what will read the data, how fresh it has to be, and under whose permissions
- Platforms
- Snowflake, Databricks, BigQuery, Postgres
- Plus dbt, Airflow or Dagster for transformation and orchestration, and pgvector, Pinecone or Azure AI Search when retrieval is in scope
- Team on a build
- Named in scope
- Always includes an AI engineer on the consumer side, which is what makes the layer fit what reads it
- First deliverable
- Target architecture and data contracts
- Before any pipeline is written, so the shape is agreed while it is still cheap to change
02 / Services
Our data engineering services
Four of these data engineering services have a page of their own with its own cost and timeline. The rest are capabilities inside a larger build; the sections below this one explain how each is delivered.
- 01Data warehouse consultingModelling, performance and cost on Snowflake, Databricks or BigQuery, including rescuing a platform that grew organically.SnowflakeDatabricksBigQuerydbtSet after assessment
- 02Data analytics consultingSemantic layers, governed metrics and the dbt models underneath them, so two dashboards stop disagreeing about revenue.dbtLookerPower BISnowflakeSet after assessment
- 03Data migrationMoving off legacy ETL, on-prem warehouses or a platform you have outgrown, with a dual-running period and a rollback position.AirflowdbtFivetranSpark3–6 months
- 04Data governance consultingContracts, quality tests, lineage and access control, with an owner against every dataset and an action against every failure.Great ExpectationsOpenLineagedbt testsSet after assessment
- 05Data pipelinesBatch and streaming ingestion with tests, lineage and alerting, built so a failure wakes someone rather than going quiet.AirflowDagsterdbtSpark6–10 weeks
- 06AI-ready data layersFeature stores, semantic layers and vector indexes shaped around the agent or model that will query them.pgvectorPineconeAzure AI Search6–10 weeks
- 07Permission-aware retrievalDocument ingestion that carries the source system's access control all the way into the index, so a retrieval answer cannot leak.SharePointGoogle DriveElasticsearch6–10 weeks
- 08Streaming and eventsKafka or Kinesis pipelines feeding real-time scoring and agents that cannot wait for tomorrow's batch.KafkaKinesisPub/SubSet after assessment
03 / AI readiness
What an AI workload needs from your data that a dashboard never did
A dashboard tolerates data that is a day old, flattened, and readable by everyone who can open the report. An agent does not. It reads on demand, on behalf of a specific person, and acts on what it finds, so the same dataset has to answer three questions a dashboard never asked: how stale is this, who is allowed to see it, and what shape does it need to be in to be retrieved rather than scanned.
The table below is the check our data engineering services run in the first week of an engagement. Minimum is where a workload will function. Comfortable is where it stops being the thing that breaks.
Book a data assessment| Check | Minimum | Comfortable |
|---|---|---|
| Freshness at read time | Under 24 hours old when it is read | Under 15 minutes old when it is read |
| Permission granularity | Dataset level, with a filter applied at query time | Row and document level, carried from the source system into the index |
| Document shape | Text extractable, one document per record | Chunked on structure with headings and source anchors kept, so a citation points at a place |
| Lineage | You can name the source table for a figure | Column level, so a wrong answer is traced to the transform that caused it |
| Schema stability | Breaking changes are announced | A contract, with notice and a version, so the consumer breaks in staging and not in production |
| Identity resolution | A join key exists between the systems the workload reads | One resolved entity per customer, so the agent does not answer about half of them |
04 / Process
How our data engineering services work, stage by stage
Six stages, each with a gate that has to pass before the next one starts. The gate is a check, not a meeting: it either evaluates true or the stage is not finished.
- 01Consumer discoveryConsumers signed off
- 02Architecture and contractsContracts agreed with producers
- 03IngestionSource reconciles to target
- 04ModellingTests pass on every model
- 05ServingConsumer reads at required freshness
- 06OperateRunbook rehearsed with your team
Stage 011 week
Consumer discovery
We start at the other end from most data projects: with whatever is going to read the data. Agents, retrieval systems, models, analysts. Each one states what it needs, how fresh, under whose permissions, and what it does when the data is missing.
What gets built
- Consumer requirements, one per reader
- Source system inventory
- Access and permission map
- Fixed quote for the build
Gate
every_consumer.has(freshness, permissions, shape, failure_mode)
Skip it and
you build the warehouse the last project needed and discover the agent cannot use it.
WorkshopsSource profiling
Stage 021–2 weeks
Architecture and contracts
The target shape, written down before anything is built. Every dataset that will cross a team boundary gets a contract: the fields, their types, the rules they obey, and what the producer guarantees about freshness and notice.
What gets built
- Target architecture
- A data contract per shared dataset
- Permission model
- Cost envelope for the platform
Gate
contract.signed_by(producer) AND contract.accepted_by(consumer)
Skip it and
the producing team changes a column and the consumer discovers it only after an incorrect answer appears.
dbtOpenLineage
Stage 03Set in the project plan
Ingestion
Getting the data in, incrementally, with the tests that prove it arrived intact. Row counts and checksums are reconciled against the source before anything downstream is allowed to read it.
What gets built
- Batch and streaming ingestion
- Reconciliation tests
- Retry and backfill behaviour
- Alerting on failure
Gate
source.row_count == target.row_count AND checksum_matches
Skip it and
silent partial loads, which are worse than an outage because nothing looks broken.
AirflowDagsterFivetranKafka
Stage 04Set in the project plan
Modelling
Transformation into the shape the consumers asked for in stage one, with a test on every model. Uniqueness, referential integrity, accepted values and freshness, all of them in version control and all of them run on every build.
What gets built
- dbt models
- Tests per model
- Semantic layer or feature definitions
- Column level lineage
Gate
dbt build --fail-fast AND coverage(tests) >= 0.80
Skip it and
two dashboards disagree about revenue and the team cannot trace which definition is current.
dbtGreat Expectations
Stage 05Set in the project plan
Serving
Exposing the layer to the things that read it, in the access pattern each one needs: a warehouse query, a feature lookup, or a vector search whose results are filtered by the reader's permissions before they are ranked.
What gets built
- Query and retrieval endpoints
- Permission filters at read time
- Index build and refresh
- Load testing against the real pattern
Gate
p95_latency <= consumer.budget AND results.filtered_by(caller_permissions)
Skip it and
a retrieval system that returns documents the person asking is not allowed to read.
pgvectorPineconeAzure AI Search
Stage 06Agreed for handover
Operate
Monitoring, alerting and the handover. Your team runs a failure with us watching before we step back, so the first real incident is not the first time anyone has seen the runbook.
What gets built
- Monitoring and alert routing
- Runbooks per failure mode
- Rehearsed incident
- Handover or a managed service
Gate
team.resolved(simulated_failure) WITHOUT escalation_to(meronym)
Skip it and
your team inherits a platform without a rehearsed way to respond when support is needed.
Monte CarloOpenLineage
05 / Contracts
Every shared dataset ships with a contract
A data contract is the agreement between the team that produces a dataset and everything that reads it: the fields, the rules each one obeys, how fresh it will be, and how much notice you get before any of that changes. It is checked in, it is tested on every build, and breaking it fails the pipeline rather than the consumer.
This matters especially with an AI consumer. A person reading a report may notice when a column goes null; an automated system needs a test that stops or marks the data stale before the issue reaches an answer. The contract turns that condition into a visible build failure instead.
The specimen below is an illustrative format, not a client document. A live engagement uses your dataset names, owners, guarantees and review process.
| Field | Type | Rule |
|---|---|---|
| order_id | string | not null, unique |
| customer_id | string | not null, references customers.id |
| placed_at | timestamp | not null, not in the future |
| status | enum | one of [placed, paid, shipped, refunded, cancelled] |
| total_cents | integer | greater than or equal to 0 |
| currency | string | ISO 4217, 3 characters |
| refunded_cents | integer | less than or equal to total_cents |
| region | string | not null, drives row level access |
Guarantees
06 / Quality
What we test, and what happens when a test fails
We describe data quality through the checks that run and the action each failure triggers. Each check either halts a run or raises an alert for a named owner, making the control visible to your engineering and operations teams.
| Dimension | Test that runs | Threshold | On failure |
|---|---|---|---|
| Completeness | Null rate per required column, per load | Above 0.5% on a required column, or any null in a key | Halt the load, serve the last good partition, alert the producing team |
| Freshness | Age of the newest row against the contract | Older than twice the contracted refresh interval | Alert, and mark the dataset stale so consumers can degrade rather than answer |
| Volume | Row count against the trailing distribution | 25% off the same weekday's trailing four-week average | Quarantine the partition and hold for review before it is published |
| Uniqueness | Duplicate rate on the contract's key | Any duplicate on the key | Halt the load. A duplicate key is a modelling fault, not a data blip |
| Referential integrity | Orphan rate against the parent table | Above 0.1% of rows | Alert, and route orphans to a review table rather than dropping them |
| Schema drift | Column set and types against the contract version | Any unannounced change | Fail the build in staging, which is the point of the contract |
| Permission integrity | Sampled retrieval as a restricted user | Zero documents returned outside the user's grant | Halt the index refresh. This one has no tolerance |
07 · Stack
Data infrastructure services: the stack, and when we pick each
Warehouse and lakehouse
- Snowflake
- Databricks
- BigQuery
- Postgres
Snowflake when the workload is SQL and the team is analysts. Databricks when Spark, notebooks or ML training are in scope. BigQuery when you are already on Google Cloud and the pattern is bursty. Postgres when the data is smaller than everyone assumes, which is more often than the vendors would like.
Transformation and orchestration
- dbt
- Airflow
- Dagster
- Spark
dbt for anything expressible in SQL, because the tests and lineage come with it. Dagster over Airflow on a new build when assets matter more than tasks; Airflow when your team already runs it and the cost of a second orchestrator is higher than the benefit.
Streaming
- Kafka
- Kinesis
- Pub/Sub
Only when a consumer genuinely cannot wait for the next batch. Streaming roughly doubles the operational surface, so we ask what breaks at an hour of latency before we agree to seconds.
Quality and lineage
- Great Expectations
- dbt tests
- OpenLineage
- Monte Carlo
dbt tests cover most common checks without another licence. Great Expectations helps when those checks outgrow SQL. A paid observability tool becomes useful once the dependency graph is too large for the team to review manually.
Retrieval stores
- pgvector
- Pinecone
- Azure AI Search
- Elasticsearch
pgvector first, because most corpora are smaller than a dedicated vector database needs and one fewer system is worth a lot. A dedicated store when the index outgrows the primary database or the filtering gets complicated.
08 / Migration
Migrating a platform with a calm, reversible cutover
- 01Set after inventory
Inventory and dependency map
Every table, job and report on the old platform, and what reads each one. The list is always longer than the one you were given, and the surprises are found here rather than at cutover.
- 02Set after inventory
Build alongside
The new platform is built while the old one keeps running. Nothing is switched off, and nothing depends on the new platform yet.
- 03Agreed from reconciliation results
Dual running and reconciliation
Both platforms run the same loads and we compare the outputs row by row until they agree. Disagreements often reveal a business rule that was not documented, which is exactly what this phase is designed to find.
- 04Phased by consumer
Cutover by consumer
Readers are moved one at a time, lowest risk first, with the old platform still warm. There is no single switch, so there is no single thing that can fail.
- 05After an agreed quiet period
Decommission
The old platform is switched off only after an agreed quiet period with nothing reading it. That period is when the last forgotten report turns up.
Rollback position: until decommission, the old platform is running and reconciled, so reverting a consumer is a configuration change rather than a recovery. Each consumer moves only after its own reconciliation checks pass.
09 / Cost
What data engineering services cost
Build and monthly platform costs are quoted separately because they behave differently: the build is a project, while cloud and platform costs continue with usage. Showing both helps engineering and finance plan together.
Engagement sizes
- Data assessment and architecture$4k–$18k
Consumer requirements, source inventory, target architecture, data contracts and a cost model for the platform. Ends in a fixed quote for the build, and is credited against it if you go ahead.
- AI-ready data layer$12k–$59k
Pipelines, models, indexes and permission filters for a defined set of consumers, with tests, lineage and alerting.
- Platform build or migration$26k–$153k
A warehouse or lakehouse from scratch, or a migration off one you have outgrown, including dual running and cutover.
- Dedicated data team$9k–$25k per month
Ongoing delivery against a roadmap, billed monthly. The usual shape once the first build lands and the backlog keeps growing.
10 / Team
Who is on a data engineering engagement
The row that matters is the last one. An AI engineer sits on the consumer side of every data build we run, which is how the layer ends up fitting what reads it rather than what produced it.
| Role | Owns | Allocation |
|---|---|---|
| Data engineer | Pipelines, modelling, tests, lineage | Set in scope |
| Platform engineer | Infrastructure, security, cost and the warehouse bill | Set in scope |
| Delivery lead | Scope, the gates between stages, and your weekly report | Set in scope |
| AI engineer, consumer side | What the agents and models need, and whether the layer delivers it | Set in scope |
11 / Work
Data work we have shipped
- Web3 data · Real-time ingestion and indexingTransactions processed with few or no stops, and the indexed history retrieved quickly as it growsEthereum transactions, wallet activity and contract events taken in as they happen through a queue system, with the full history of the chain indexed and ready to query.Read the case study
- Dental practices · Business intelligenceUp to 40 to 50% less time producing and reviewing business performance data, against manual reportingOperational and financial data from purchasing, services and vendors brought into one platform, with dashboards on performance and profitability for each practice.Read the case study
12 / Industries
Data engineering services by sector
- Banking and financeRetention rules that outlive the systems holding the data, and a lineage requirement that comes from the regulator rather than from engineering.
- InsurancePolicy and claims data spread across platforms bought a decade apart, where the same customer exists three times under slightly different names.
- Logistics and supply chainEvent volume that makes batch too slow and streaming expensive, with partner data arriving in formats your team cannot control.
- Retail and e-commerceSKU hierarchies that change faster than the models reading them, and a returns picture that only resolves weeks after the sale.
- ManufacturingHigh-frequency sensor data that does not need to be stored forever, next to ERP data with much longer retention needs.
- HealthcarePermissions that have to survive every hop, because the access question is not a preference here.
Before you buy data engineering services
All guides- Further readingAI agent costWhat the surrounding AI work costs, broken down by the things that actually move the number.
- Further readingSelf-hosted AI agentsAgents you can run on your own servers or cloud, and what each one sends outside your environment, which is the data question most teams ask first.
- Further readingAI agents for businessWhat teams put agents on first, which tells you which datasets to make ready first.
FAQs
Data engineering services FAQ
Clear answers to the questions that commonly come up before a first conversation about data engineering services.
What are data engineering services?
The work of moving data from the systems that produce it into a form something else can rely on: pipelines, models, storage, and the quality tests, lineage and permissions around them. The service ends where the model begins. We build the layer; training and deploying models is machine learning development.
How much do data engineering services cost?
An assessment and architecture engagement is typically $4k–$18k, credited against the build if you go ahead. An AI-ready data layer is typically $12k–$59k, and a platform build or migration $26k–$153k. A dedicated data team is typically $9k–$25k per month. Cloud and platform charges are modelled separately because they continue with usage.
How long do data engineering services take?
The assessment usually takes two to three weeks and ends with a fixed quote and project plan. An AI-ready data layer is commonly six to ten weeks, while a full platform build or migration may take three to six months. Stage gates still need to pass before the team moves forward, even when a target date has been agreed.
When does data engineering outsourcing make sense, and what does a data engineering company add?
Your in-house team may be fully capable of doing this work. An external team can add delivery capacity, experience from similar migrations, reusable contract and test formats, and an AI engineer who represents the systems that will consume the data. If your team already has those capabilities and availability, we will say so during the assessment.
What is included in data engineering consulting services?
The consulting half is the assessment and the architecture: what will read the data, what it needs, what to build, what it will cost to run, and the contracts between producers and consumers. Some clients stop there and build it themselves with the architecture in hand. That is a legitimate outcome and it is priced as its own engagement.
Do you offer data platform consulting for Snowflake, Databricks and BigQuery?
Yes. Choosing the platform, sizing it and modelling what it will cost to run are part of our data engineering services, in the data assessment and architecture engagement. For a platform you already run, including one that grew without a plan, data warehouse consulting covers modelling, performance and cost.
How does data pipeline development handle legacy source systems?
The same way as anything else, with one extra step: we profile the source before we agree a contract, because legacy systems rarely match their documentation. Where change data capture is not available we fall back to incremental extracts with reconciliation, and the reconciliation test is what makes that safe rather than hopeful.
How do you keep permissions intact when data reaches an AI system?
Permissions are carried as attributes on the record through every hop and applied as a filter at read time, before ranking, not after. We test it by running sampled retrieval as a restricted user and checking that nothing outside their grant comes back. That test has no tolerance, and a single failure halts the index refresh.
What happens to our data platform after you hand over?
You own all of it: the code, the models, the contracts, the runbooks. Before we step back, your team runs a simulated failure with us watching, so the first real incident is not the first time anyone has opened the runbook. If you would rather we kept running it, the dedicated team engagement is the shape for that.
Start with the data assessment
Tell us what you want to read the data and we will tell you what stands between you and it, what to build, and what it will cost to run. The assessment ends in a target architecture, the contracts for every shared dataset, and a fixed quote for the build.
Reply within two business days