This page was generated from
docs/examples/DataSet/Working with snapshots.ipynb.
Interactive online version:
.
Working with snapshots
Here, the following topics are going to be covered:
What is a snapshot
How to create it
How it is saved next to the measurement data
How to extract snapshot from the dataset
Useful imports
[1]:
import json # for converting JSON data into python 'dict'
from pathlib import Path
from pprint import pprint # for pretty-printing python variables like 'dict'
from qcodes.dataset import (
Measurement,
initialise_or_create_database_at,
load_or_create_experiment,
)
from qcodes.instrument_drivers.mock_instruments import DummyInstrument
from qcodes.parameters import Parameter
from qcodes.station import Station
Logging hadn't been started.
Activating auto-logging. Current session state plus future input saved.
Filename : /home/runner/.qcodes/logs/command_history.log
Mode : append
Output logging : True
Raw input log : False
Timestamping : True
State : active
Qcodes Logfile : /home/runner/.qcodes/logs/240503-17279-qcodes.log
What is a snapshot?
Often times experiments comprise a complex network of interconnected instruments. Their numerous settings define the overall behavior of the experimental setup. Obviously, the experimental setup has a direct impact on the measured data which is of prime interest for researchers. In order to capture this link, the measured data should have metadata associated with it. An important part of that metadata is a captured state of the experimental setup. In QCoDeS terms, this is called snapshot.
How to create a snapshot?
All QCoDeS instruments and parameters (and some other objects too, like InstrumentChannels) support snapshotting, which means that they provide a method to retrieve their state.
Let’s look at snapshots of various objects.
Snapshot example for Parameter object
Let’s create a Parameter, call its snapshot method, and then inspect the output of that method.
The returned snapshot is a python dictionary that reflects all the important properties of that parameter (check the name, label, unit, and even value).
[2]:
p = Parameter('p', label='Parameter P', unit='kg', set_cmd=None, get_cmd=None)
p.set(123)
snapshot_of_p = p.snapshot()
pprint(snapshot_of_p)
{'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'p',
'inter_delay': 0,
'label': 'Parameter P',
'name': 'p',
'post_delay': 0,
'raw_value': 123,
'ts': '2024-05-03 11:19:46',
'unit': 'kg',
'validators': [],
'value': 123}
In case you want to use the snapshot object in your code, you can refer to its contents in the same way as you work with python dictionaries, for example:
[3]:
print(f"Value of {snapshot_of_p['label']} was {snapshot_of_p['value']} (when it was snapshotted).")
Value of Parameter P was 123 (when it was snapshotted).
Note that the implementation of a particular QCoDeS object defines which attributes are snapshotted and how. For example, Parameter implements a keyword argument snapshot_value which allows to choose if the value of the parameter is snapshotted (the reasons for this are out of scope of this article). (Another interesting keyword argument of Parameter that is realated to snapshotting is snapshot_get - refer to Parameters’s docstring for more information.)
Below is a demonstration of the snapshot_value keyword argument, notice that the value of the parameter is not part of the snapshot.
[4]:
q = Parameter('q', label='Parameter Q', unit='A', snapshot_value=False, set_cmd=None, get_cmd=None)
p.set(456)
snapshot_of_q = q.snapshot()
pprint(snapshot_of_q)
{'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'q',
'inter_delay': 0,
'label': 'Parameter Q',
'name': 'q',
'post_delay': 0,
'ts': None,
'unit': 'A',
'validators': []}
Snapshot of an Instrument
Now let’s have a brief look at snapshots of instruments. For the sake of exercise, we are going to use a “dummy” instrument.
[5]:
# A dummy instrument with two parameters, "input" and "output", plus a third one we'll use later.
instr = DummyInstrument('instr', gates=['input', 'output', 'gain'])
instr.gain(11)
[6]:
snapshot_of_instr = instr.snapshot()
pprint(snapshot_of_instr, indent=4)
{ '__class__': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'functions': {},
'label': 'instr',
'name': 'instr',
'parameters': { 'IDN': { '__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_IDN',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'IDN',
'name': 'IDN',
'post_delay': 0,
'raw_value': None,
'ts': None,
'unit': '',
'validators': ['<Anything>'],
'vals': '<Anything>',
'value': None},
'gain': { '__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_gain',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate gain',
'name': 'gain',
'post_delay': 0,
'raw_value': 11,
'ts': '2024-05-03 11:19:46',
'unit': 'V',
'validators': ['<Numbers -800<=v<=400>'],
'vals': '<Numbers -800<=v<=400>',
'value': 11},
'input': { '__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_input',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate input',
'name': 'input',
'post_delay': 0,
'raw_value': 0,
'ts': '2024-05-03 11:19:46',
'unit': 'V',
'validators': ['<Numbers -800<=v<=400>'],
'vals': '<Numbers -800<=v<=400>',
'value': 0},
'output': { '__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_output',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate output',
'name': 'output',
'post_delay': 0,
'raw_value': 0,
'ts': '2024-05-03 11:19:46',
'unit': 'V',
'validators': ['<Numbers -800<=v<=400>'],
'vals': '<Numbers -800<=v<=400>',
'value': 0}},
'submodules': {}}
Station and its snapshot
Experimental setups are large, and instruments tend to be quite complex in that they comprise many parameters and other stateful parts. It would be very time-consuming for the user to manually go through every instrument and parameter, and collect the snapshot data.
Here is where the concept of station comes into play. Instruments, parameters, and other submodules can be added to a Station object (nbviewer.jupyter.org link). In turn, the station has its snapshot method that allows to create a collective, single snapshot of all the instruments, parameters, and submodules.
Note that in this article the focus is on the snapshot feature of the QCoDeS Station, while it has some other features (also some legacy once).
Let’s create a station, and add a parameter, instrument, and submodule to it. Then we will print the snapshot. Notice that the station is aware of insturments and stand-alone parameters, and classifies them into dedicated lists within the snapshot.
[7]:
station = Station()
station.add_component(p)
station.add_component(instr)
# Note that it is also possible to add components
# to a station via arguments of its constructor, like this:
# station = Station(p, instr)
[7]:
'instr'
[8]:
snapshot_of_station = station.snapshot()
pprint(snapshot_of_station)
{'components': {},
'config': None,
'instruments': {'instr': {'__class__': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'functions': {},
'label': 'instr',
'name': 'instr',
'parameters': {'IDN': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_IDN',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'IDN',
'name': 'IDN',
'post_delay': 0,
'raw_value': {'firmware': 'NA',
'model': '<class '
"'qcodes.instrument_drivers.mock_instruments.DummyInstrument'>",
'serial': 'NA',
'vendor': 'QCoDeS'},
'ts': '2024-05-03 11:19:46',
'unit': '',
'validators': ['<Anything>'],
'vals': '<Anything>',
'value': {'firmware': 'NA',
'model': '<class '
"'qcodes.instrument_drivers.mock_instruments.DummyInstrument'>",
'serial': 'NA',
'vendor': 'QCoDeS'}},
'gain': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_gain',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate gain',
'name': 'gain',
'post_delay': 0,
'raw_value': 11,
'ts': '2024-05-03 11:19:46',
'unit': 'V',
'validators': ['<Numbers '
'-800<=v<=400>'],
'vals': '<Numbers '
'-800<=v<=400>',
'value': 11},
'input': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_input',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate input',
'name': 'input',
'post_delay': 0,
'raw_value': 0,
'ts': '2024-05-03 11:19:46',
'unit': 'V',
'validators': ['<Numbers '
'-800<=v<=400>'],
'vals': '<Numbers '
'-800<=v<=400>',
'value': 0},
'output': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_output',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate output',
'name': 'output',
'post_delay': 0,
'raw_value': 0,
'ts': '2024-05-03 '
'11:19:46',
'unit': 'V',
'validators': ['<Numbers '
'-800<=v<=400>'],
'vals': '<Numbers '
'-800<=v<=400>',
'value': 0}},
'submodules': {}}},
'parameters': {'p': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'p',
'inter_delay': 0,
'label': 'Parameter P',
'name': 'p',
'post_delay': 0,
'raw_value': 456,
'ts': '2024-05-03 11:19:46',
'unit': 'kg',
'validators': [],
'value': 456}}}
Saving snapshot next to the measurement data
With the power of the station object, it is now possible to conveniently associate the snapshot information with the measured data.
In order to do so, a station needs to be created, and then that station needs to be provided to the Measurement object. If no station is explicitly provided, the Measurement object will use the default station, Station.default (refer to Measurement and Station objects docstrings for more information). At the moment the new measurement run is started, a snapshot of the whole station will be taken, and added next to the measured data.
Note that the snapshot gets stored in a JSON format (an automatic convertion from python dictionary to JSON takes place). This is done in order to ensure that the snapshot can be read in environments other than python. JSON is an extemely popular data format, and all platforms/environments/languages/frameworks have means to read JSON-formatted data.
Here is how it looks in the code. We will create a new experiment. Then we are going to reuse the station object created above, and create a new measurement object. Then we will perform a dummy measurement. After that we are going to extract the snapshot from the resulting dataset, and print it.
[9]:
# Let's initialize a database to ensure that it exists
initialise_or_create_database_at(Path.cwd() / "snapshot_example.db")
# Let's create a new experiment
experiment = load_or_create_experiment('snapshot_experiment', 'no_sample_yet')
[10]:
measurement = Measurement(experiment, station)
measurement.register_parameter(instr.input)
measurement.register_parameter(instr.output, setpoints=[instr.input])
[10]:
<qcodes.dataset.measurements.Measurement at 0x7f96a4a38950>
[11]:
with measurement.run() as data_saver:
input_value = 111
instr.input.set(input_value)
instr.output.set(222) # assuming that the instrument measured this value on the output
data_saver.add_result((instr.input, input_value),
(instr.output, instr.output()))
# For convenience, let's work with the dataset object directly
dataset = data_saver.dataset
Starting experimental run with id: 1.
Extracting snapshot from dataset
Now we have a dataset that contains data from the measurement run. It also contains the snapshot.
In order to access the snapshot, use the DataSet’s properties called snapshot and snapshot_raw. As their docstrings declare, the former returns the snapshot of the run as a python dictionary, while the latter returns it as JSON string (in other words, in exactly the same format as it is stored in the experiments database).
[12]:
snapshot_of_run = dataset.snapshot
[13]:
snapshot_of_run_in_json_format = dataset.snapshot_raw
To prove that these snapshots are the same, use json.loads or json.dumps to assert the values of the variables:
[14]:
assert json.loads(snapshot_of_run_in_json_format) == snapshot_of_run
[15]:
assert json.dumps(snapshot_of_run) == snapshot_of_run_in_json_format
Finally, let’s pretty-print the snapshot. Notice that the values of the input and output parameters of the instr instrument have 0s as values, and not 111 and 222 that were set during the measurement run.
[16]:
pprint(snapshot_of_run)
{'station': {'components': {},
'config': None,
'instruments': {'instr': {'__class__': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'functions': {},
'label': 'instr',
'name': 'instr',
'parameters': {'IDN': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_IDN',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'IDN',
'name': 'IDN',
'post_delay': 0,
'raw_value': {'firmware': 'NA',
'model': '<class '
"'qcodes.instrument_drivers.mock_instruments.DummyInstrument'>",
'serial': 'NA',
'vendor': 'QCoDeS'},
'ts': '2024-05-03 '
'11:19:46',
'unit': '',
'validators': ['<Anything>'],
'vals': '<Anything>',
'value': {'firmware': 'NA',
'model': '<class '
"'qcodes.instrument_drivers.mock_instruments.DummyInstrument'>",
'serial': 'NA',
'vendor': 'QCoDeS'}},
'gain': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_gain',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate '
'gain',
'name': 'gain',
'post_delay': 0,
'raw_value': 11,
'ts': '2024-05-03 '
'11:19:46',
'unit': 'V',
'validators': ['<Numbers '
'-800<=v<=400>'],
'vals': '<Numbers '
'-800<=v<=400>',
'value': 11},
'input': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_input',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate '
'input',
'name': 'input',
'post_delay': 0,
'raw_value': 0,
'ts': '2024-05-03 '
'11:19:46',
'unit': 'V',
'validators': ['<Numbers '
'-800<=v<=400>'],
'vals': '<Numbers '
'-800<=v<=400>',
'value': 0},
'output': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'instr_output',
'instrument': 'qcodes.instrument_drivers.mock_instruments.DummyInstrument',
'instrument_name': 'instr',
'inter_delay': 0,
'label': 'Gate '
'output',
'name': 'output',
'post_delay': 0,
'raw_value': 0,
'ts': '2024-05-03 '
'11:19:46',
'unit': 'V',
'validators': ['<Numbers '
'-800<=v<=400>'],
'vals': '<Numbers '
'-800<=v<=400>',
'value': 0}},
'submodules': {}}},
'parameters': {'p': {'__class__': 'qcodes.parameters.parameter.Parameter',
'full_name': 'p',
'inter_delay': 0,
'label': 'Parameter P',
'name': 'p',
'post_delay': 0,
'raw_value': 456,
'ts': '2024-05-03 11:19:46',
'unit': 'kg',
'validators': [],
'value': 456}}}}
Note that the snapshot that we have just loaded from the dataset is almost the same as the snapshot that we directly obtained from the station above. The only difference is that the snapshot loaded from the dataset has a top-level station field. If you do not trust me, have a look at the following assert statement for the proof.
[17]:
assert {'station': snapshot_of_station} == snapshot_of_run
Comparing how two DataSets were taken
Suppose something went wrong in an experiment, and you’d like to compare what changed since a known-good run. QCoDeS lets you do this by taking a diff between the snapshots for two DataSet instances.
[18]:
measurement = Measurement(experiment, station)
measurement.register_parameter(instr.input)
measurement.register_parameter(instr.output, setpoints=[instr.input])
[18]:
<qcodes.dataset.measurements.Measurement at 0x7f96a4a50c10>
[19]:
instr.gain(400) # Oops!
with measurement.run() as data_saver:
input_value = 111
instr.input.set(input_value)
instr.output.set(222) # assuming that the instrument measured this value on the output
data_saver.add_result((instr.input, input_value),
(instr.output, instr.output()))
# For convenience, let's work with the dataset object directly
bad_dataset = data_saver.dataset
Starting experimental run with id: 2.
The diff_param_values function tells us about the parameters that changed betw
[20]:
from qcodes.utils.metadata import diff_param_values
[21]:
diff_param_values(dataset.snapshot, bad_dataset.snapshot).changed
[21]:
{('instr', 'gain'): (11, 400),
('instr', 'output'): (0, 222),
('instr', 'input'): (0, 111)}
[ ]: