This page was generated from docs/examples/DataSet/InMemoryDataSet.ipynb. Interactive online version: .

In memory dataset

This notebooks explains an alternative way of measuring where the raw data is not written directly to a sqlite database file but only kept in memory with the ability to export the data after the measurement is completed. This may significantly speed up measurements where a lot of data is acquired but there is no protection against any data lose that may happen during a measurement. (Power loss, computer crash etc.) However, there may be situations where this trade-off is worthwhile. Please do only use the in memory dataset for measurements if you understand the risks.

%matplotlib inline
import numpy as np

import qcodes as qc
from qcodes.dataset import (
    DataSetType,
    Measurement,
    initialise_or_create_database_at,
    load_by_run_spec,
    load_or_create_experiment,
    plot_dataset,
)
from qcodes.instrument_drivers.mock_instruments import (
    DummyInstrument,
    DummyInstrumentWithMeasurement,
)

Logging hadn't been started.
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/runner/.qcodes/logs/command_history.log
Mode           : append
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active
Qcodes Logfile : /home/runner/.qcodes/logs/240503-14508-qcodes.log

Here we set up two mock instruments and a database to measure into:

# preparatory mocking of physical setup

dac = DummyInstrument("dac", gates=["ch1", "ch2"])
dmm = DummyInstrumentWithMeasurement(name="dmm", setter_instr=dac)

station = qc.Station(dmm, dac)

initialise_or_create_database_at("./in_mem_example.db")
exp = load_or_create_experiment(experiment_name="in_mem_exp", sample_name="no sample")

And run a standard experiment writing data to the database:

meas = Measurement(exp=exp)
meas.register_parameter(dac.ch1)  # register the first independent parameter
meas.register_parameter(dmm.v1, setpoints=(dac.ch1,))  # now register the dependent oone

meas.write_period = 0.5

with meas.run() as datasaver:
    for set_v in np.linspace(0, 25, 10):
        dac.ch1.set(set_v)
        get_v = dmm.v1.get()
        datasaver.add_result((dac.ch1, set_v), (dmm.v1, get_v))

    dataset1D = datasaver.dataset  # convenient to have for data access and plotting

Starting experimental run with id: 1.

ax, cbax = plot_dataset(dataset1D)

The in memory measurement looks nearly identical with the only difference being that we explicitly pass in an Enum to select the dataset class that we want to use as a parameter to measurement.run

The DataSetType Enum currently has 2 members representing the two different types of dataset supported.

with meas.run(dataset_class=DataSetType.DataSetInMem) as datasaver:
    for set_v in np.linspace(0, 25, 10):
        dac.ch1.set(set_v)
        get_v = dmm.v1.get()
        datasaver.add_result((dac.ch1, set_v), (dmm.v1, get_v))
    datasetinmem = datasaver.dataset

Starting experimental run with id: 2.

ax, cbax = plot_dataset(datasetinmem)

datasetinmem.run_id

2

When the measurement is performed in this way the data is not written to the database but the metadata (run_id, timestamps, snapshot etc.) is.

To preserve the raw data it must be exported it to another file format. See Exporting QCoDes Datasets for more information on exporting including how this can be done automatically.

datasetinmem.export("netcdf", path=".")

The export_info attribute contains information about locations where the file was exported to. We will use this below to show how the data may be reloaded from a netcdf file.

path_to_netcdf = datasetinmem.export_info.export_paths["nc"]
path_to_netcdf

'/home/runner/work/Qcodes/Qcodes/docs/examples/DataSet/qcodes_2_91f01d82-0000-0000-0000-018f3e2bb26c.nc'

As expected we can see this file in the current directory.

!dir

Accessing-data-in-DataSet.ipynb
Benchmarking.ipynb
Cache
DataSet-class-walkthrough.ipynb
Database.ipynb
Datasaver_Builder.ipynb
Dataset_Performance.ipynb
Exporting-data-to-other-file-formats.ipynb
Extracting-runs-from-one-DB-file-to-another.ipynb
InMemoryDataSet.ipynb
Linking\ to\ parent\ datasets.ipynb
Measuring\ X\ as\ a\ function\ of\ time.ipynb
Offline\ Plotting\ Tutorial.ipynb
Offline\ plotting\ with\ categorical\ data.ipynb
Offline\ plotting\ with\ complex\ data.ipynb
Paramtypes\ explained.ipynb
Pedestrian\ example\ of\ subscribing\ to\ a\ DataSet.ipynb
Performing-measurements-using-qcodes-parameters-and-dataset.ipynb
QCoDeS_example.db
Real_instruments
Saving_data_in_the_background.ipynb
The-Experiment-Container.ipynb
Threaded\ data\ acquisition.ipynb
Using_doNd_functions_in_comparison_to_Measurement_context_manager_for_performing_measurements.ipynb
Using_register_name.ipynb
Working\ with\ snapshots.ipynb
Working-With-Pandas-and-XArray.ipynb
data_2018_01_17
dataset_performance.db
export_example.db
extract_runs_notebook_source.db
extract_runs_notebook_source_aux.db
extract_runs_notebook_target.db
import-data-from-legacy-dat-files.ipynb
in_mem_example.db
in_mem_example.db-shm
in_mem_example.db-wal
measurement_extensions.db
qcodes_2_47015d4e-0000-0000-0000-018f3e2b710e.nc
qcodes_2_91f01d82-0000-0000-0000-018f3e2bb26c.nc
reimport_example.db
somefile.nc
subscriber\ json\ exporter.ipynb

Note that you can interact with the dataset via the cache attribute of the dataset in the same way as you can with a regular dataset. However the in memory dataset does not implement methods that provide direct access to the data from the dataset object it self (get_parameter_data etc.) since these read data from the database.

Reloading data from db and exported file

ds = load_by_run_spec(captured_run_id=datasetinmem.captured_run_id)

plot_dataset(ds)

([<Axes: title={'center': 'Run #2, Experiment in_mem_exp (no sample)'}, xlabel='Gate ch1 (V)', ylabel='Gate v1 (V)'>],
 [None])

When a dataset is loaded using load_by_run_spec and related functions QCoDeS will first check if the data is available in the database. If not if will check if the data has been exported to netcdf and then try to load the data from the last known export location. If this fails a warning will be raised and the dataset will be loaded without any raw data.

A dataset can also be loaded directly from the netcdf file. See Exporting QCoDes Datasets for more information on how this is done. Including information about how you can change the netcdf location.