Service · Web development

Custom data warehouse development as the foundation for analytics and AI.

A central data layer on which your BI, dashboards, machine-learning models and RAG applications can run. Built vendor-independent on Snowflake, BigQuery, Databricks or your own lakehouse: modular, with code ownership on your side, and GDPR-compliant from day one.

Snowflake / BigQueryLakehousedbt + AirflowELT with AirbyteAI-ready

A data warehouse is not a reporting database

A data warehouse is the central storage and compute layer where you bring together all relevant data from your operational systems, transform it into reliable models, and make it available for analytics, BI, dashboarding and, increasingly, AI and machine learning applications. It is not a copy of your operational database; it is a purpose-built data model in which definitions are explicit, history is preserved, and queries stay fast even across billions of rows.

We build data warehouses for mid-market and enterprise organisations that have multiple source systems (CRM, ERP, e-commerce, product analytics, marketing stacks) and want to consolidate their reporting and AI position on a single, reliable foundation. The question often starts with "our numbers don't add up" or "every department calculates revenue differently" and ends with a data model in which business definitions are captured once and carried consistently through every dashboard, KPI report and LLM prompt.

A data warehouse rarely stands alone. It is usually the heart of a broader data engineering platform, with ingestion, orchestration, transformation and observability as the surrounding layers. For the visualisation on top, one of two routes usually makes sense: a commercial BI tool such as Power BI or Tableau, or a custom BI layer that runs in your own branding and can be embedded in your product. In our first conversation we advise which route fits, based on who will use it and which compliance requirements apply.

Four architectures for a data warehouse.

Traditional data warehouse (Inmon or Kimball) — the classic approach with normalised data vault or dimensional star schemas. Suited to stable business processes where the data models last for years and governance requirements are strict (financial reporting, annual reports, regulatory data). Kimball with fact and dimension tables remains a strong choice for analytics-heavy workloads where speed and consistency matter more than flexibility. We build this on Postgres with Citus for on-premises, or on a cloud warehouse such as Snowflake or Azure Synapse as data volumes grow.

Modern data stack (ELT + dbt) — the standard of recent years: raw data loaded into a cloud warehouse via Airbyte, Fivetran or Meltano, transformations in dbt, orchestration via Airflow, Dagster or Prefect, and a BI layer on top. The ELT approach (extract-load-transform rather than extract-transform-load) suits cloud warehouses that have the compute capacity to run transformations at query time. dbt has democratised the transformation layer: SQL with version control, tests, lineage and documentation as a by-product. For most mid-market organisations, this is the right starting architecture.

Lakehouse (Iceberg or Delta) — the architecture that combines the best of a data lake and a data warehouse: open file formats (Parquet) beneath a transactional table-format layer (Apache Iceberg, Delta Lake, Apache Hudi) that provides ACID guarantees without vendor lock-in. Suited to organisations with both structured and unstructured data that want to run multiple compute engines on the same storage. Databricks delivers this with Delta Lake, Snowflake with Iceberg tables, or you can run your own open-source stack on S3 or GCS with Trino, Spark or DuckDB.

Data mesh (federated) — for enterprise organisations with multiple business units or subsidiaries that want to keep managing their own data but must be able to collaborate in a federated way. Each domain team owns its own data product, and a central team provides governance, the data catalogue and the federated query layer. It works well when the organisation is mature enough to carry decentralised ownership; for mid-sized companies it is usually overkill. We build mesh implementations on Snowflake Data Sharing, BigQuery Analytics Hub or a custom Trino layer.

Which route fits depends on your scale, use cases and organisational maturity. We deliberately don't always recommend the same thing — one architecture for everyone is the wrong answer.

Three flavours of data warehouse.

Most warehouse projects fall into one of these three profiles. Which one fits depends on the number of source systems, the data volume, and how much you want to manage yourself. In the first conversation we discuss which route is right and where the right cloud choice lies.

Compact project · fixed sprint budget

Single-cloud warehouse on BigQuery or Snowflake

A warehouse project in which we set up a single cloud warehouse (BigQuery, Snowflake or Azure Synapse) with three to five source systems connected via Airbyte or Fivetran, version-controlled and tested dbt transformations, and Airflow or Dagster as the orchestrator. Suited to mid-market organisations consolidating their data for the first time or building a first reporting layer. We deliver a central data model with core tables for your most important entities (customers, orders, products, leads) plus the first mart layer for BI purposes. From there, your team can keep building on its own, or you can have us continue the development under a managed service.

BigQuery / Snowflakedbt + AirflowAirbyte / FivetranPower BI / MetabaseGDPR-compliant
Mid-sized project · fixed sprint budget

Lakehouse with AI readiness

For organisations that have both structured and semi-structured or unstructured data (logs, JSON events, documents, images) and want to set up their warehouse straight away as a foundation for AI and LLM applications. We build a lakehouse on Databricks Delta Lake, Snowflake with Iceberg tables, or an open-source stack on S3 or Google Cloud Storage with Trino and DuckDB. On top come vector tables for RAG applications, feature stores for ML models, and a metadata catalogue that your AI development team can use without having to build data pipelines themselves. AI Act compliance layer included.

Databricks / IcebergDelta LakeVector storeFeature storeRAG-ready
Larger project · fixed sprint budget

Enterprise data mesh with federated governance

For enterprise organisations with multiple business units, subsidiaries or strict regulatory regimes (banks, insurers, healthcare groups, government). We build a federated architecture in which each domain team owns its own data product, with a central governance layer that manages the data catalogue, lineage, access control and quality monitoring. Components such as Snowflake Data Sharing, BigQuery Analytics Hub, or a Trino layer across multiple warehouses are combined with a data catalogue (DataHub, OpenMetadata, Collibra) and observability (Monte Carlo, Soda). Full GDPR and DPIA documentation, EU data residency, and an audit trail per query.

Snowflake / TrinoDataHub / CollibraMonte Carlo / SodaEU residencyData mesh

What you get at the end.

A production-ready data warehouse that your team manages itself, plus everything around it to keep building without vendor lock-in. Code ownership is the starting point — all SQL, all DAGs, all infrastructure-as-code and all documentation sit in your Git.

  • The warehouse itself, in productionProduction and staging environments in your cloud (GCP, AWS, Azure) or managed by us. Including CI/CD, automated tests, monitoring, backups and a documented recovery procedure.
  • Source connectors and ingestion pipelinesConnected to your source systems: CRM (HubSpot, Salesforce, Pipedrive), ERP (SAP, NetSuite, Exact, AFAS), e-commerce (Shopify, Magento), product analytics (Mixpanel, Amplitude), marketing stacks (Meta, Google Ads), operational databases. Via Airbyte, Fivetran or Meltano, or with custom connectors where needed.
  • Transformation layer in dbtA complete dbt project with staging, intermediate and mart models, plus tests, exposures and documentation. Business definitions (revenue, margin, customer retention, CAC) are defined once in macros, so they stay consistent through every report, dashboard and LLM prompt.
  • Orchestration and data qualityAirflow, Dagster or Prefect as the orchestrator, with DAGs for all ELT jobs. Soda or Monte Carlo for data observability: freshness, volume checks, null rates, schema drift and alerting via Slack or Teams when something breaks between source and mart.
  • Semantic layer and data catalogueA central place where business terms, column definitions and KPI formulas are recorded, usable by your analysts, your BI tool and any text-to-SQL or LLM layer. Linked to DataHub, OpenMetadata or a lighter alternative, with lineage from source to dashboard.
  • BI and AI integrationsConnections to the BI tool of your choice (Power BI, Tableau, Looker, Metabase) or to a custom BI tool. For AI applications: feature store and vector tables ready for RAG, fine-tuning or classic ML models.
  • Compliance package and governancePII redaction at the staging layer, retention policies per table, EU residency configuration where relevant, audit logging on every query, and encryption in transit and at rest. DPIA documentation for GDPR, plus the extra governance documents for sector-specific regimes (NEN 7510 in healthcare, DORA in finance, the AI Act for ML models).
  • Full codebase and documentationSource code, infrastructure as code (Terraform), dbt project, Airflow DAGs, architecture diagrams, ERDs, data dictionary and an incident runbook. Everything in Git, with code ownership on your side, so another team can take it over without vendor lock-in.
  • Training and ongoing supportSessions for your data engineers and analysts, a hands-on dbt workshop for the analytics team, and short videos for BI end users. Optional support contract covering monitoring, security patches, cost optimisation and further development at a fixed monthly price.

When your own data warehouse is the right choice.

Six patterns in which organisations come to us for a warehouse project. If you recognise one, a conversation is usually worthwhile. We are honest about when a lighter alternative will do: sometimes a central dashboard is enough and a full warehouse is over-investment.

Source system sprawl

Four or more systems that don't talk to each other

You have a CRM, ERP, e-commerce platform, product analytics and marketing stacks, and every department exports to Excel separately. Definitions drift apart (sales calculates revenue differently from finance), reports differ per team, and nobody trusts the numbers any more. A central data layer that brings everything together in a single data model solves this at its root.

AI readiness

You want to put LLMs or ML to work on your data

RAG applications, customer-specific chat assistants, predictive models for churn or fraud. All of them need a reliable data foundation to run feature stores and vector tables on. A warehouse is the typical prerequisite for serious AI development: without one, there is no reproducibility, no lineage and no trust in model outputs.

BI overlap

Dashboards for multiple departments

Operations, finance, marketing and management each want their own dashboard with their own KPIs. A central warehouse with a semantic layer is the natural foundation for a custom KPI dashboard per department, with consistently calculated metrics. No more diverging definitions between reports.

Real-time requirements

Operational dashboards on live data

Logistics, energy, e-commerce, gaming: environments where a daily refresh is too slow and you need to steer on live status. A warehouse is then often not enough: you need a real-time analytics platform alongside or on top of the warehouse, fed by event streams on Kafka or Materialize.

Compliance

GDPR, NEN 7510 or DORA requirements

Healthcare organisations with patient data, financial institutions with DORA reporting, energy companies with regulatory logging. The governance, audit and data residency requirements become so demanding that ad-hoc spreadsheets are no longer legally viable. A warehouse with PII redaction, retention policies and EU residency then becomes a necessity rather than a luxury.

Vendor independence

You want out of a closed platform

You're stuck with a SaaS reporting tool that won't give your data back, or with a legacy data warehouse whose licence costs rise faster every year than the value it delivers. A modular setup built on open formats (Parquet, Iceberg, dbt) gives you the option to switch compute engines in future without losing your data model.

Not sure about a large project yet?

Test your idea first: a working prototype in 1 day

With OneDayBuild, we make your idea tangible in a single day for €1,150, so you know whether further development is worth the investment. Decide to go ahead with the full build? Then we deduct the cost in full.

View OneDayBuild →

How a warehouse project works.

1

Introduction and use-case discovery

A conversation to understand which decisions your organisation wants to support with data, which sources are involved, and which compliance requirements apply. We also check straight away whether a full warehouse is the right investment — sometimes a lighter alternative is enough, and that is more honest advice than a heavier project.

2

Architecture and source inventory

A workshop with your team plus interviews with the people who will use the data (analysts, finance, operations). We draw up an architecture blueprint (warehouse choice, ELT pipeline, semantic layer, governance layer) and an initial data model for the key entities. At the end you have a scope, a tech stack choice and a realistic estimate of production cloud costs.

3

Foundation build in sprints

A working build every two weeks. We start with one source, one data model layer (staging, intermediate and the first mart) and the basic orchestration. From sprint three, your team can already see the first results in Power BI, Metabase or another BI tool. In subsequent sprints we add sources and extend the data model.

4

Quality layer, observability and governance

dbt tests on all models, freshness and volume checks via Soda or Monte Carlo, lineage and a data catalogue set up, and PII redaction and retention policies configured. For AI applications we also add the feature store and vector tables, with access control that fits your position under the AI Act.

5

Compliance audit and pen test

Before go-live: a pen test on the cloud environment, a review of row-level security and encryption, an audit trail check, and a data flow mapping for the DPIA. For sector-specific regimes we provide the additional documentation your data protection officer or regulator expects.

6

Rollout, training and further development

A phased rollout to user groups, a hands-on dbt workshop for your analytics team, and ongoing management of security, performance and cost optimisation. Cloud warehouses can become significantly more expensive if nobody monitors query patterns — as standard we provide the tooling and runbook to keep this under control.

The stack we use — and why we don't stick to one vendor.

Our stack choices are pragmatic rather than religious. For ingestion, Airbyte and Fivetran are both good options, and the choice depends on your source portfolio and budget: Fivetran is more expensive but needs less maintenance, while Airbyte is open source and works excellently self-hosted. For custom-built connectors, we work with Meltano or our own Python layer. For orchestration, Airflow remains the de facto standard, but Dagster and Prefect have strengths in data asset tracking and developer experience respectively. We advise based on your team's skills and operational needs.

For transformation, dbt is our starting point — almost all modern warehouse projects use dbt for SQL models with tests, lineage and documentation. For Python-based transformations (denoising, feature engineering, NLP) we use dbt-python, Polars or Spark, depending on scale. For the warehouse layer itself, we have extensive experience with Snowflake, BigQuery, Azure Synapse, Databricks, Redshift and ClickHouse, plus on-prem Postgres with Citus for situations where cloud is not an option.

For BI on top, we can work with any mainstream tool (Power BI, Tableau, Looker, Metabase) or build a custom BI tool when you need embedded analytics or multi-tenancy. For observability, we use Monte Carlo or Soda, and for data catalogues DataHub or OpenMetadata. For AI integrations, we work with LangChain, LlamaIndex or a custom RAG orchestration, and we build vector tables on pgvector, Weaviate, Pinecone or Snowflake Cortex.

The idea behind these choices is always the same: you get a warehouse built on open standards (SQL, Parquet, Iceberg, dbt), so your team can switch vendors in future without losing your data model. Vendor independence isn't a marketing slogan but an architectural principle: we build warehouses that you can maintain yourself, or that another team can take over from you.

Frequently asked questions.

What clients usually want to know before we start on a warehouse project.

What exactly is a data warehouse?
A data warehouse is a central data system designed specifically for analytical workloads rather than operational transactions. You pull data from your operational systems (CRM, ERP, e-commerce, etc.) and transform it into a data model that makes analytical questions quick to answer, often in a star schema with fact and dimension tables, or in a data vault model. The difference from a data lake is that data in a warehouse is explicitly modelled and transformed; the difference from your operational database is that the warehouse retains history and is optimised for analytical queries rather than point operations.
Which cloud warehouse do you usually recommend?
That depends on where the rest of your stack lives. For organisations already deep in Google Cloud, BigQuery is usually the right choice: serverless, no capacity planning, native integration with the rest of GCP. For AWS organisations, Snowflake almost always competes in the same category, alongside Redshift and Databricks. For Microsoft organisations, Azure Synapse or Databricks on Azure makes sense. For heavily operational workloads with high-cardinality dashboards (real-time analytics), ClickHouse is often the better choice. And for specific compliance requirements around data residency, we sometimes opt for on-prem Postgres with Citus.
What about GDPR and data residency?
In our setup, PII (personal data) gets a redaction layer on staging as standard: the raw ingestion captures everything, but every field marked as PII is hashed, pseudonymised or fully redacted before it lands in the model layer. For EU data residency, we configure the warehouse in an EU region (eu-west-3 for BigQuery, EU-Frankfurt for Snowflake, west-europe for Azure). Retention policies are set at column level, with automatic data deletion after the retention period. For a DPIA, we provide a data-flow mapping and a record of processing activities. For AI applications where PII ends up in features or vector stores, we extend this with an AI Act-compliant audit trail.
What's the difference between a data lake and a lakehouse?
A data lake is raw file storage (Parquet, JSON, CSV, logs) on object storage with no schema layer: you put everything in and decide later what to do with it. A data warehouse is the opposite: everything is explicitly modelled, transformed and optimised for analytical queries. A lakehouse combines the two: open file formats on object storage (Parquet) with a transactional table-format layer (Iceberg, Delta Lake, Hudi) that provides ACID guarantees and warehouse-like query performance. For most mid-market organisations, a pure warehouse on BigQuery or Snowflake is the simplest choice; for organisations with large volumes of unstructured data or a strong AI/ML focus, a lakehouse architecture becomes interesting.
What about a data warehouse versus a data engineering platform?
A data warehouse is the central storage and compute layer. A data engineering platform is the broader context around it: ingestion tooling, orchestration, transformation frameworks, observability, governance, and the developer experience for your data team. In practice the two overlap heavily: a serious warehouse project almost always has a platform layer around it. Whether you tackle that as one project or as two consecutive phases depends on your starting position. Do you already have a working warehouse but no proper ELT pipeline? Then we're talking about a platform project. Starting from scratch? Then a single combined project is usually more efficient.
How big does your organisation need to be to have its own warehouse?
There's no fixed answer, but as a rough guide: your own warehouse makes economic sense when you have multiple source systems with overlapping entities, a data team or a strong wish to professionalise your data work, and a use case that goes beyond purely ad hoc reporting. Below that scale, a lighter solution (a central dashboard, a lightweight ETL into an operational Postgres, or a managed BI tool with direct source connectors) is often a better investment. We're honest about this: not every company needs a warehouse.
What about cloud costs?
Cloud warehouses generally work on a usage-based model: you pay for compute and for storage. Snowflake and BigQuery price compute sharply, so your query patterns largely determine what you pay. The biggest pitfall is an unregulated team running queries that make costs explode. That's why we build in cost monitoring as standard (BigQuery slot reservations, Snowflake resource monitors, per-team query cost attribution) and run a monthly review of the most expensive queries. For every project we deliver a cost model for the first production year, so there are no surprises.
Do you also build AI/ML models on the warehouse?
The warehouse itself is usually the right place to manage feature stores and vector tables. The models on top of it, whether classic ML (churn prediction, propensity scoring, fraud detection) or LLM applications (RAG, agentic flows, document classification), we build through our AI development practice. For classic ML we typically run Python frameworks (scikit-learn, XGBoost, LightGBM) on the warehouse data via dbt-python or Databricks. For LLM applications we connect the warehouse as a knowledge source to a RAG architecture with a vector store. Both require reliable data lineage and reproducibility, which is exactly what a good warehouse delivers.
What if we already have a warehouse that no longer meets our needs?
Many of our projects are warehouse migrations rather than greenfield builds. Typical scenarios: a legacy on-prem warehouse on SQL Server or Oracle that is becoming too slow and too expensive to maintain, a Redshift cluster whose licence costs and operational burden have become heavy, or a first cloud warehouse that was set up without a modelling layer or governance and is now a mess. We run an audit (where the pain is, what the query patterns are, what the current model layer looks like) and propose a migration path with parallel running, so you are never without reporting during the switchover. The migration often includes a data model clean-up as well, because the old setup has typically grown ad hoc over the years.
How long until we can go live?
For a first working warehouse on a single cloud with three to five source systems, the foundation is in place after a few sprints, and from that point your team can run its first dashboards in BI tools. A fully production-ready warehouse with a semantic layer, observability, governance, AI readiness and multi-source integrations takes several further sprints on top of that. We work in two-week sprints with a working build every sprint, so you see early on where things are heading and can adjust along the way. For multi-tenant or mesh architectures the lead time is longer, because the governance and operating-model layer requires extra work.

Talk to us about your warehouse question.

A no-obligation introductory call of half an hour. We listen to who uses the data, which sources are involved and which compliance requirements apply, and we tell you honestly whether your own warehouse is the right route or whether a lighter alternative is enough. No sales funnel, just genuine sparring.

Fabian van Dijk
Business developer · Appfront
fabian.vandijk@appfront.nl
Share LinkedIn Email

Edit content