Work with Cognite Data Fusion

1.1 Four kinds of extractor

1.2 The big three for OT: PI, OPC UA, DB

All Cognite extractors share one design: a YAML config file with a cognite section (where to send, how to authenticate), a source section (where to read), and an extractor section (behaviour: state store, buffering, batch sizes). Here is an illustrative DB-extractor config that lands an equipment table in RAW — the exact keys are in each extractor's docs, but the shape is always this:

# db-extractor config (illustrative — check the extractor docs for exact keys)
version: 3
logger:
  console: { level: INFO }

cognite:
  project: hnm-dev
  host: https://westeurope-1.cognitedata.com
  idp-authentication:                       # service principal, never a personal login
    token-url: https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token
    client-id: ${CLIENT_ID}
    secret: ${CLIENT_SECRET}
    scopes: [ https://westeurope-1.cognitedata.com/.default ]
  extraction-pipeline: { external-id: ep_cmms_equipment }   # ties runs to a pipeline (1.4)

extractor:
  state-store: { local: { path: state.json } }             # remembers "where was I" between runs

databases:
  - name: cmms
    connection-string: "Driver={ODBC Driver 18 for SQL Server};Server=cmms-db;Database=maint;Trusted_Connection=yes"

queries:
  - name: equipment_master
    database: cmms
    query: >
      SELECT tag, description, func_loc, parent_func_loc, updated_at
      FROM equipment WHERE updated_at > '{incremental_field}' ORDER BY updated_at
    incremental-field: updated_at         # only rows changed since last run
    initial-start: "2000-01-01"
    destination: { type: raw, database: cmms, table: equipment_master }

1.3 RAW from the inside

RAW is a three-level store: database → table → rows. Each row has a key, a lastUpdatedTime, and a columns JSON object. There is no schema — every row can have different columns — which is exactly why it is the right landing zone: the extractor never has to know what the source will look like next year.

When to bypass RAW: time-series datapoints (billions of small values — they go straight to the time-series store), files (binary — straight to the file store), and hosted-extractor streams that are already clean and mapped. Everything tabular from IT systems should go through RAW.

1.4 Extraction pipelines — the monitoring wrapper

An extraction pipeline is a CDF object that represents "this extractor, on this source". The extractor reports to it, and it gives you:

The status page in Integrate → Extraction pipelines is the first screen an integration engineer opens every morning. If a pipeline has not reported seen for longer than its schedule, the data is stale — before anyone in the plant notices.

1.5 Data sets and lineage

Every resource an extractor creates should carry a data set (dataSetId). A data set answers "which system does this come from, and who owns it?" — e.g. ds_cmms_equipment, ds_pi_unit2. Two reasons this matters now, not later: (1) Module 8 scopes access by data set, and (2) the data catalog shows lineage per data set. Decide the naming before the first extractor runs.

2.1 Transformations — SQL that writes to CDF

A transformation is a scheduled Spark SQL query whose result set is written to a destination. You choose the destination (assets, time series, events, RAW, or data-model nodes/edges); the destination decides which output columns are required. RAW tables are queried as plain tables; CDF resources appear under _cdf.*; data-model instances via cdf_nodes(...).

Two real queries. First, the classic asset hierarchy (destination: assets) — this is what the Academy's "Transform data into asset hierarchy" course does:

-- Destination: Assets · Action: upsert
select
  cast(tag as STRING)                          as externalId,
  cast(tag as STRING)                          as name,
  cast(description as STRING)                  as description,
  cast(parent_func_loc as STRING)              as parentExternalId,   -- builds the tree
  dataset_id('ds_cmms_equipment')              as dataSetId,
  to_metadata(func_loc, serial)                as metadata            -- leftovers into metadata
from cmms.equipment_master
where is_new('cmms_equipment_master', lastUpdatedTime)              -- only changed rows since last successful run

Second, the modern way (destination: data model nodes, view Pump in Module 3). Nodes need space + externalId; relations are written with node_reference:

-- Destination: Nodes · view hnm_power:Pump/1 · instance space hnm_power_data
select
  'hnm_power_data'                             as space,
  cast(tag as STRING)                          as externalId,
  cast(tag as STRING)                          as name,
  cast(description as STRING)                  as description,
  node_reference('hnm_power_data', parent_func_loc) as asset,        -- direct relation: which asset this equipment sits in
  cast(serial as STRING)                       as serialNumber,
  cast(tag as STRING)                          as sourceId
from cmms.equipment_master
where tag like 'BFP-%'

2.2 Building the asset hierarchy properly

The tree Plant → Unit → System → Equipment must be created parents first. Two patterns:

• Single query, self-referencing. The assets destination resolves parentExternalId within the same batch, so one query can write the whole tree as long as every parent exists in the result set or already in CDF.
• Layered queries. One transformation per level (locations → systems → equipment), chained by a workflow (2.3). Easier to debug on large plants.

For the Core Data Model the equivalent is CogniteAsset.parent (a direct relation) — the path and root properties are then maintained by CDF automatically.

2.3 Data Workflows — the orchestrator

A workflow is a directed graph of tasks. Task types: transformation, function, CDF request (any API call), dynamic (tasks generated at run time), subworkflow, and simulation. Tasks declare dependsOn, and per task you set retries, timeout and onFailure (abortWorkflow or skipTask). Outputs flow between tasks with references like ${extract.output.rows}.

The same graph as a workflow definition (what the API/Toolkit store; the UI builds it visually):

{
  "workflowExternalId": "wf_nightly_unit2", "version": "v1",
  "workflowDefinition": { "tasks": [
    { "externalId": "check",  "type": "cdf",
      "parameters": { "cdfRequest": { "resourcePath": "/extpipes/runs/list", "method": "POST",
                       "body": { "filter": { "externalId": "ep_cmms_equipment" }, "limit": 1 } } } },
    { "externalId": "assets", "type": "transformation", "dependsOn": [{ "externalId": "check" }],
      "parameters": { "transformation": { "externalId": "tr_assets", "concurrencyPolicy": "fail" } },
      "retries": 3, "timeout": 1800, "onFailure": "abortWorkflow" },
    { "externalId": "wos",    "type": "transformation", "dependsOn": [{ "externalId": "check" }],
      "parameters": { "transformation": { "externalId": "tr_workorders" } }, "retries": 3 },
    { "externalId": "hours",  "type": "function", "dependsOn": [{ "externalId": "assets" }, { "externalId": "wos" }],
      "parameters": { "function": { "externalId": "fn_running_hours", "data": { "unit": "U2" } } }, "timeout": 600 },
    { "externalId": "notify", "type": "cdf", "dependsOn": [{ "externalId": "hours" }], "onFailure": "skipTask",
      "parameters": { "cdfRequest": { "resourcePath": "/extpipes/runs", "method": "POST",
                       "body": { "items": [{ "externalId": "ep_nightly_unit2", "status": "success" }] } } } }
  ] }
}

Triggers: a cron schedule, or a data-modeling trigger that fires when instances of a view change ("run when new work orders arrive"). Workflow definitions are versioned; each run records inputs, outputs, timings and errors per task.

2.4 Cognite Functions — Python inside CDF

When SQL is not enough (calling an external API, a calculation with pandas, writing datapoints), you deploy a Python function. It runs serverless inside CDF with an authenticated client already injected. The contract is one function named handle:

# handler.py  — deployed as Function "fn_running_hours"
import pandas as pd

def handle(client, data, secrets=None, function_call_info=None):
    """client: pre-authenticated CogniteClient · data: dict passed by the caller/workflow
       secrets: dict of secrets set on the function · function_call_info: ids of this call"""
    unit = data.get("unit", "U2")
    # motor current > 5 A means "running"; compute daily running hours for every BFP in the unit
    ts = client.time_series.list(external_id_prefix=f"{unit}.BFP", limit=None)
    current = [t.external_id for t in ts if t.external_id.endswith(".MOTOR.CURRENT")]
    df = client.time_series.data.retrieve_dataframe(
        external_id=current, start="30d-ago", end="now", aggregates=["average"], granularity="1h")
    # column names differ by SDK major version: v8 = MultiIndex (external_id, aggregate, …), v7 = "external_id|average"
    df.columns = (df.columns.get_level_values(0) if isinstance(df.columns, pd.MultiIndex)
                  else [c.split("|")[0] for c in df.columns])
    running = (df > 5).resample("1D").sum()          # hours per day where avg current > 5 A
    running.columns = [c.replace(".MOTOR.CURRENT", ".RUNNING_HOURS") for c in running.columns]
    client.time_series.data.insert_dataframe(running)   # target series must exist (created in 5.3 / by a transformation)
    return {"pumps": len(current), "days": len(running)}

# requirements.txt (bundled with the function)
cognite-sdk>=7
pandas

• Deploy from the UI (upload a zip / point at a folder) or with client.functions.create(...); Level 3 does it from Git with the Toolkit.
• Run on demand (client.functions.call(external_id=..., data={...})), on a schedule (cron), or from a workflow task.
• Secrets (API keys for external systems) are stored on the function and arrive in the secrets dict — never in code.
• Limits exist on memory and run time (they differ per cloud — see the docs' technical page); design functions to finish in minutes, not hours, and let workflows fan out.

2.5 Monitoring the whole chain

Three screens, one story: Extraction pipelines (did the data arrive?), Transformations (did it get shaped? rows written / errors), Workflows (did the chain complete? per-task timing). Set notifications on the pipeline and on the workflow; leave transformations to report through the workflow. Level 3 adds metrics and dashboards on top.

3.1 The building blocks, precisely

3.2 The Core Data Model in depth

The CDM lives in the system space cdf_cdm (views at version v1). It is built from small mix-in interfaces that you saw in Level 1 — CogniteDescribable (name, description, tags, aliases), CogniteSourceable (sourceId, sourceContext, source timestamps), CogniteSchedulable (start/end times), CogniteVisualizable (object3D) — and concrete types that implement them. The rule for your own model is simple:

3.3 Building a model in GraphQL

In Build solutions → Data models → Create, you write the schema in Cognite's GraphQL data modeling language (DML). Types become views (and, unless told otherwise, matching containers). Start with the editor's Import → Core Data Model dialog: it pastes the CDM types you pick as interfaces, complete with all their fields. Then write your own types. The full PowerPlant model, following the pattern in Cognite's "Extending the core data model" guide:

# Data model: hnm_power : PowerPlant / v1

# 1) Imported from the Core Data Model (space cdf_cdm, version v1). The editor inserts these as
#    interfaces WITH all their fields — abbreviated here. @import = "use the existing view, don't create one".
interface CogniteDescribable   @view(space: "cdf_cdm", version: "v1") @import { name: String  description: String  tags: [String]  aliases: [String] }
interface CogniteSourceable    @view(space: "cdf_cdm", version: "v1") @import { sourceId: String  sourceContext: String  # … }
interface CogniteVisualizable  @view(space: "cdf_cdm", version: "v1") @import { object3D: Cognite3DObject }
interface CogniteAsset      implements CogniteVisualizable & CogniteDescribable & CogniteSourceable
                               @view(space: "cdf_cdm", version: "v1") @import { # … all CogniteAsset fields }
interface CogniteEquipment  implements CogniteDescribable & CogniteSourceable
                               @view(space: "cdf_cdm", version: "v1") @import { # … all CogniteEquipment fields }
interface CogniteTimeSeries implements CogniteDescribable & CogniteSourceable
                               @view(space: "cdf_cdm", version: "v1") @import { # … all CogniteTimeSeries fields }

# 2) Our types. GraphQL requires an implementing type to repeat every interface field —
#    the editor does that for you (Ctrl+Space → "implement interface fields"); shown abbreviated.
#    No @import here (we own these); @view(version) pins a version you can reference from SQL and the SDK.
type Unit implements CogniteAsset & CogniteVisualizable & CogniteDescribable & CogniteSourceable
    @view(version: "1") {
  # … all CogniteAsset fields (name, description, parent, path, root, …)
  unitNumber: Int
  ratedMW: Float
}

type Pump implements CogniteEquipment & CogniteDescribable & CogniteSourceable
    @view(version: "1") {
  # … all CogniteEquipment fields (name, description, serialNumber, manufacturer, asset: CogniteAsset, …)
  ratedFlow: Float                    # m3/h
  ratedHead: Float                    # m
  pumpType: String
  feeds: [Pump] @relation(type: { space: "hnm_power", externalId: "feeds" })   # an edge relation with a type
}

type Sensor implements CogniteTimeSeries & CogniteDescribable & CogniteSourceable
    @view(version: "1") {
  # … all CogniteTimeSeries fields (name, isStep, type, unit, assets: [CogniteAsset], equipment: [CogniteEquipment], …)
  measurementType: String             # pressure · temperature · current …
  alarmHigh: Float
}

What the directives mean: @view(space, version) identifies a view — required on imported types, optional on your own (defaults: the model's space and an auto-generated version); @import reuses an existing view instead of creating one; implements inherits the interface's fields and lets Cognite tools treat a Pump as a CogniteEquipment; @relation makes an edge-backed list; a plain type reference (asset: CogniteAsset) is a direct relation. Publish → the editor validates, creates the containers for your new properties, and exposes a GraphQL endpoint for PowerPlant/v1.

3.4 Populating the model

Three ways, all writing nodes and edges through the same instances API:

1. Transformation with destination Nodes (view Pump, instance space hnm_power_data) — the query from Module 2.1. For edges use destination Edges with startNode/endNode columns via node_reference.
2. Python SDK — client.data_modeling.instances.apply(nodes=[…], edges=[…]) (Module 5).
3. The UI — Data models → your model → a view → "Add instance" for quick tests.

Order matters only for relations: a direct relation may point at a node that does not exist yet (it resolves when it arrives), but an edge requires both endpoints to exist. So: nodes first, edges second — which is what the workflow in Figure 2.2 enforces.

3.5 Querying

Every data model gets a generated GraphQL API. The generated root fields are list<Type>, get<Type>ById, search<Type> and aggregate<Type>. Nested relations are just nested selections:

query PumpsInUnit2 {
  listPump(filter: { asset: { externalId: { eq: "U2" } } }, first: 100) {
    items {
      externalId
      name
      ratedFlow
      asset { name }
      timeSeries(first: 20) {            # reverse relation from CogniteEquipment
        items { externalId name unit { symbol } }
      }
    }
  }
}

Behind GraphQL sits the REST instances API: /models/instances/query (graph traversal with filters and cursors), /search (full text), /aggregate, and /sync — a cursor you keep calling to receive changes since last time, which is how downstream systems stay in step without polling everything.

3.6 Managing with spaces

• Schema space vs. instance space. Keep hnm_power (views, containers, model) separate from hnm_power_data (nodes, edges). You can then grant a team write on the data space but not on the schema.
• One instance space per site or business unit when access differs (hnm_plantA_data, hnm_plantB_data) — all sharing one schema space.
• Data sets still exist for classic resources (files, classic time series); for data-model instances the space is the access boundary (Module 8).

4.1 Entity matching

The problem: the historian has U2.BFP1A.DISCH.PRESS; SAP has 21-PA-001A; your model has BFP-1A. Entity matching is a machine-learning service that proposes source → target pairs with a confidence score, based on string features of names and descriptions.

The same operation from Python (the UI does exactly this under the hood):

sources = [{"id": t.id, "externalId": t.external_id, "name": t.name, "description": t.description or ""}
           for t in client.time_series.list(external_id_prefix="U2.", limit=None)]
targets = [{"id": a.id, "externalId": a.external_id, "name": a.name}
           for a in client.assets.list(external_id_prefix="BFP", limit=None)]

model = client.entity_matching.fit(sources=sources, targets=targets,
                                   match_fields=[("name", "name"), ("description", "name")],
                                   feature_type="bigram")
job = model.predict(sources=sources, targets=targets, num_matches=1, score_threshold=0.6)
for m in job.result["items"]:
    src = m["source"]["externalId"]
    for cand in m["matches"]:
        print(f'{src:32s} → {cand["target"]["externalId"]:10s}  {cand["score"]:.2f}')

4.2 Interactive engineering diagrams (P&ID parsing)

Diagram parsing reads a P&ID (PDF or image), finds every text that looks like a tag, matches it against a list of entities you supply, and writes an annotation per hit: page, bounding box (normalised 0–1 coordinates), text, linked resource, confidence, status. Once approved, the drawing becomes clickable in Search and Canvas, and drawings link to each other through their off-page connectors.

# The detect job from Python (what the UI runs)
job = client.diagrams.detect(
    entities=[{"name": a.name, "externalId": a.external_id, "aliases": [a.name.split("-", 1)[-1]]}
              for a in client.assets.list(external_id_prefix="21-", limit=None)],
    search_field="name",
    file_external_ids=["21-PID-004"],
    min_tokens=2, partial_match=True)
for page in job.result["items"][0]["annotations"]:
    print(page["text"], page["confidence"], [e["externalId"] for e in page["entities"]])

4.3 Document contextualization

Beyond P&IDs: datasheets, manuals, inspection reports. CDF extracts text (OCR when needed) for full-text search, classifies documents into file categories (the CDM CogniteFileCategory), and can extract structured fields (a datasheet's "design pressure" → a property). Linking a manual to its equipment is either a metadata rule in the transformation (equipment_tag column → equipment relation) or entity matching on file names. This is what Copilot cites when it answers "what is the maximum design temperature?" in Level 1 Figure 5.2.

4.4 3D contextualization

1. Upload a CAD model (many formats) or point cloud → CDF processes it into a streamable 3D model + revision.
2. Map nodes to assets. CAD node names usually contain tags (/21-PA-001A/PUMP_CASING); the 3D contextualization tool proposes mappings the same way entity matching does; you approve; each mapping becomes a Cognite3DObject link (Level 1 Figure 4.2).
3. Use. Click the pump in 3D → its data; search an asset → "Show in 3D"; InField shows the operator where the valve is.

360° image collections (site walk-throughs) follow the same pattern with image annotations instead of CAD nodes.

4.5 Quality & the approval workflow

5.1 Authentication

# pip install "cognite-sdk<8" msal python-dotenv   # pin 7.x if the same environment also uses cognite-extractor-utils or cognite-toolkit (both still require SDK < 8)
import os
from cognite.client import CogniteClient

# A) interactive — you (Level 1 · 5.5 used this against the sandbox)
client = CogniteClient.default_oauth_interactive(
    project="hnm-dev", cdf_cluster="westeurope-1",
    tenant_id=os.environ["CDF_TENANT_ID"], client_id=os.environ["CDF_CLIENT_ID"],
    client_name="hnm-notebook")

# B) client credentials — a service principal (never commit the secret; use env vars / a vault)
client = CogniteClient.default_oauth_client_credentials(
    project="hnm-dev", cdf_cluster="westeurope-1",
    tenant_id=os.environ["CDF_TENANT_ID"], client_id=os.environ["CDF_CLIENT_ID"],
    client_secret=os.environ["CDF_CLIENT_SECRET"], client_name="hnm-job")

print(client.iam.token.inspect().projects)   # which projects and capabilities this token has

5.2 Reading — the calls you will live in

import pandas as pd

# Assets: one, a subtree, a filtered list
bfp = client.assets.retrieve(external_id="BFP-1A")
unit2_tree = client.assets.retrieve_subtree(external_id="U2")                 # AssetList
pumps = client.assets.list(external_id_prefix="BFP", limit=None).to_pandas()

# Time series metadata, then datapoints as a DataFrame (aggregates keep it small)
ts = client.time_series.list(asset_external_ids=["BFP-1A"], limit=None)
df = client.time_series.data.retrieve_dataframe(
        external_id=[t.external_id for t in ts],
        start="30d-ago", end="now",
        aggregates=["average", "max"], granularity="1h")                    # SDK 8: MultiIndex columns (id, aggregate, …); SDK 7: "id|average"

# Events / activities linked to the pump (classic events API)
wos = client.events.list(asset_external_ids=["BFP-1A"], type="workorder", limit=None).to_pandas()

# Files: metadata + download
files = client.files.list(asset_external_ids=["BFP-1A"], limit=None)
client.files.download(directory="./downloads", external_id="21-PID-004")

# RAW rows straight into pandas
raw = client.raw.rows.retrieve_dataframe("cmms", "equipment_master", limit=None)

# Data-model instances through a view
from cognite.client.data_classes.data_modeling import ViewId
pump_view = ViewId("hnm_power", "Pump", "1")
nodes = client.data_modeling.instances.list(instance_type="node", sources=pump_view,
                                            space="hnm_power_data", limit=None)
pumps_dm = nodes.to_pandas()

5.3 Writing

from cognite.client.data_classes import TimeSeriesWrite, EventWrite
from cognite.client.data_classes.data_modeling import NodeApply, NodeOrEdgeData, EdgeApply, DirectRelationReference

# Classic resources: upsert by external ID
client.time_series.upsert([TimeSeriesWrite(external_id="U2.BFP1A.RUNNING_HOURS", name="BFP-1A running hours",
                                           unit="h", is_step=True, asset_id=bfp.id, data_set_id=ds_id)])
client.time_series.data.insert_dataframe(running_hours_df)      # index = timestamps, columns = external IDs
client.events.upsert([EventWrite(external_id="WO-4711", type="workorder", start_time=..., end_time=...,
                                 asset_ids=[bfp.id], metadata={"description": "Bearing change"})])
client.files.upload("BFP-1A_datasheet.pdf", external_id="DS-BFP-1A", asset_ids=[bfp.id], data_set_id=ds_id)

# Data-model instances: nodes with properties per view, and an edge
node = NodeApply(space="hnm_power_data", external_id="BFP-1A",
                 sources=[NodeOrEdgeData(source=pump_view, properties={
                     "name": "BFP-1A", "ratedFlow": 1200.0, "ratedHead": 2100.0,
                     "asset": DirectRelationReference("hnm_power_data", "U2")})])
edge = EdgeApply(space="hnm_power_data", external_id="BFP-1A.feeds.BFP-1B",
                 type=DirectRelationReference("hnm_power", "feeds"),
                 start_node=DirectRelationReference("hnm_power_data", "BFP-1A"),
                 end_node=DirectRelationReference("hnm_power_data", "BFP-1B"))
client.data_modeling.instances.apply(nodes=[node], edges=[edge])

5.4 The REST API directly

Everything above is a POST to https://{cluster}.cognitedata.com/api/v1/projects/{project}/… with a bearer token. Reasons to go raw: a feature the SDK doesn't wrap yet, Postman exploration, or a language without an SDK. The reference is api-docs.cognite.com; every endpoint shows request/response JSON. Two conventions: list endpoints are POST …/list with a filter and a cursor for paging; and externalId lookups are POST …/byids with {"items":[{"externalId":"…"}]}.

# Raw call through the SDK's HTTP client (keeps auth and retries)
resp = client.post(f"/api/v1/projects/{client.config.project}/assets/list",
                   json={"filter": {"externalIdPrefix": "BFP"}, "limit": 10})
print(resp.json()["items"][0]["name"])

5.5 Jupyter inside CDF

Build solutions → Jupyter opens a notebook that runs in your browser (JupyterLite) with a pre-authenticated client (from cognite.client import CogniteClient; client = CogniteClient() — no credentials needed), pandas and plotting available, and notebooks saved in CDF and shareable with colleagues. Ideal for analysis you want a domain expert to open without installing anything. Heavy compute (large ML training) still belongs on your own machine or a Function.

5.6 Streamlit apps

The same area hosts Streamlit: a Python file becomes a small web app, authenticated with the viewer's CDF identity. A pump-health page is ~40 lines: pick a pump from listPump, plot 30 days of pressure with work orders shaded, show the latest running hours. Level 3 turns this into a production app.

6.1 Charts

Charts is the trending tool operators and engineers will use daily. Three features turn it from "a trend" into "analysis":

• Calculations (no code). Add a calculation node: Discharge pressure − Suction pressure = differential head; Motor current × 0.69 × √3 × 0.9 / 1000 ≈ kW; rolling mean, resample, unit conversion. Calculations are built from a function palette (arithmetic, smoothing, resampling, logical) and can be saved as new scheduled calculations that write a time series back.
• Units. Because time series carry a unit (CDM CogniteUnit), Charts converts on the fly — bar ↔ kPa, °C ↔ °F — and refuses to add apples to oranges.
• Activities overlay. Work orders and events appear as shaded bands on the time axis ("bearing change here") — the visual version of Deliverable 5.

Alerts & monitoring: define a threshold (or a calculated signal crossing a limit) with a minimum duration, choose subscribers, and CDF emails/notifies when it fires; alerts have a history you can review. This is condition monitoring without a separate product.

6.2 Industrial Canvas

A root-cause analysis in Canvas, step by step:

1. Frame it. New canvas "BFP-1A trip 12 Sep". Add a text box with the event description.
2. Drop the evidence. Search BFP-1A → drag the asset; from its panel drag the P&ID, the last work order, the datasheet. Add the discharge pressure and motor current time series → they render as charts on the canvas; align their time ranges to the trip.
3. Add the physical view. "Show in 3D" → a 3D card focused on the pump. Add the site photo from the last inspection.
4. Cause map. Use the cause-map template (sticky notes with arrows): trip ← high bearing temp ← lube-oil flow low ← filter ΔP high ← filter change overdue (work order postponed).
5. Ask Copilot. "Which documents mention the bearing lubrication interval?" — it answers with page citations from the datasheet on the canvas.
6. Collaborate. Comment with @mentions; the maintenance planner replies inline; export a summary.

6.3 InField

InField is the tablet app for operators. Setting it up is a data-engineering job:

The payoff is the "office ↔ field" loop: an engineer sees a rising bearing temperature in Charts, creates an observation task for tonight's round, the operator sees it on the tablet at the pump (with the P&ID and the 3D position), logs the local reading and a photo, and the engineer sees it in the morning — on the same asset.

6.4 Maintain

Maintain is for turnaround and campaign planning: it pulls work orders from SAP/Maximo (through the same extractors), lets planners group them into scopes, simulates duration, crew and cost, checks against the asset hierarchy and 3D for clashes, and pushes the agreed scope back. Prerequisites: a clean work-order extraction (Module 1) mapped to CogniteActivity with functional locations resolved to assets (Module 2).

6.5 Search & 3D

• Search ranks across all resource types; filters by type, data set, time range, metadata; supports saved searches. Aliases in the CDM make PT-1019 find 21-PT-1019.
• 3D viewer streams the processed model; "Show in 3D" from any asset; clicking an object opens its data; measurements and slicing; point clouds and 360° images side by side.

7.1 Power BI

Two connectors exist. The older OData connector ("Cognite Data Fusion (OData)" in Power BI) exposes tables — Assets, Timeseries, Events, Files, Datapoints aggregates, data-model views — at https://{cluster}.cognitedata.com/odata/{version}/projects/{project}; Cognite now marks it deprecated: keep it only for existing reports. All new reports should use the "Cognite Data Fusion (REST API)" connector, which issues GET/POST and GraphQL queries directly, supports every identity provider CDF supports, performs better and reaches all CDF features.

1. Get Data → Cognite Data Fusion → enter project and cluster → sign in.
2. Pick Datapoints aggregates, not raw datapoints, and filter in the query step (asset subtree, time range, granularity) so the filter is pushed to CDF instead of downloading everything.
3. Join to Assets / data-model views by external ID for hierarchy slicers.
4. Schedule refresh in the Power BI service with a service-principal login; keep refresh windows aligned with the nightly workflow.

7.2 Grafana

Install the Cognite Data Source plugin from the Grafana catalog, configure project + cluster + OAuth2 client credentials (a service principal with read capabilities). Query editors exist for time series (by external ID, by asset subtree, with aggregates), events (as annotations), and data-model GraphQL. The trick for scale: a dashboard variable that lists pumps from a query, so one dashboard serves every pump — combined with Grafana alerting, this is a cheap ops console.

7.3 Spark, Databricks, Fabric

For bulk analytics and data science, CDF offers a Spark data source (read/write CDF resources as DataFrames) and the Microsoft Fabric connector (two-way, OneLake Delta tables ↔ CDF, no copies in RAW). Pattern: model and contextualise in CDF; train models in Databricks/Fabric on CDF data; write predictions back as time series so Charts and Canvas can show them next to the sensors.

8.1 The chain

The common ACLs: assetsAcl, timeSeriesAcl, eventsAcl, filesAcl, rawAcl, transformationsAcl, extractionPipelinesAcl, functionsAcl, workflowOrchestrationAcl, dataModelsAcl (schema), dataModelInstancesAcl (data), datasetsAcl, groupsAcl, projectsAcl. Actions are per ACL (READ, WRITE, LIST, OWNER…). Scopes: all, datasetScope, spaceIdScope, tableScope (RAW), idScope, assetRootIdScope.

// A CDF group as the API sees it (the UI builds this form; the Toolkit stores it as YAML in Level 3)
{
  "name": "unit2-engineers",
  "sourceId": "3f9c1c2e-…",                                  // Entra ID group object id
  "capabilities": [
    { "timeSeriesAcl":         { "actions": ["READ"],          "scope": { "datasetScope": { "ids": [4501] } } } },
    { "assetsAcl":             { "actions": ["READ"],          "scope": { "datasetScope": { "ids": [4501, 4502] } } } },
    { "filesAcl":              { "actions": ["READ"],          "scope": { "datasetScope": { "ids": [4502] } } } },
    { "eventsAcl":             { "actions": ["READ"],          "scope": { "datasetScope": { "ids": [4503] } } } },
    { "dataModelsAcl":         { "actions": ["READ"],          "scope": { "spaceIdScope": { "spaceIds": ["hnm_power", "cdf_cdm"] } } } },
    { "dataModelInstancesAcl": { "actions": ["READ", "WRITE"], "scope": { "spaceIdScope": { "spaceIds": ["hnm_power_data"] } } } },
    { "projectsAcl":           { "actions": ["LIST"],          "scope": { "all": {} } } }
  ]
}

8.2 Designing roles for a plant

8.3 Data-model access

Two ACLs, two questions: dataModelsAcl — may you read/change the schema (spaces, containers, views, models)? dataModelInstancesAcl — may you read/write the data (nodes, edges)? Both are scoped by space. This is why Module 3 separated hnm_power (schema) from hnm_power_data (data): engineers get WRITE on the data space and READ on the schema; only data engineers change the schema. Remember to include cdf_cdm in READ scopes or CDM-based views will not resolve.

8.4 Governance habits

• Naming conventions written down: data sets ds_<source>_<unit>, external IDs <source-prefix>:<id> or the plant tag, spaces <org>_<domain>[_data], groups <unit>-<role>.
• Data-set descriptions filled in (owner, source system, refresh, contact) — the data catalog is only as good as these.
• Quality checks in transformations (reject rows with null tags into a quarantine RAW table) and a weekly coverage number: % time series linked to an asset, % drawings parsed.
• Change log for the data model: version bumps, what changed, who consumes each view.
• Audit: review admins quarterly; review all-scoped capabilities monthly (there should be none outside admins).

8.5 Level 2 checkpoint quiz

★ Level 2 capstone — the Unit 2 feedwater mini-solution

If you did every deliverable, you already have it. Tick what exists in your dev project; each line maps to one module:

• RAW table cmms.equipment_master fed by an extractor (or CSV stand-in), registered as extraction pipeline ep_cmms_equipment with failure notifications (M1)
• Transformation tr_assets (idempotent), Function fn_running_hours, Workflow wf_nightly_unit2 (M2)
• Data model PowerPlant v1 with Unit / Pump / Sensor implementing CDM types; instances for U2, BFP-1A/1B, sensors; one edge (M3)
• Time series linked to pumps by entity matching; one P&ID parsed and ≥10 annotations approved (M4)
• A notebook that plots 30 days of three sensors with work orders shaded (M5)
• Charts workspace with calculation + alert; Canvas RCA; InField checklist (M6)
• Power BI report and Grafana dashboard on aggregates (M7)
• Six groups with data-set/space scopes; a scoped service principal; verified with a test user (M8)

Certificates & badges for this level

Glossary additions for Level 2