This page was generated from
docs/examples/DataSet/Dataset_Performance.ipynb.
Interactive online version:
.
DataSet Performance
This notebook shows the trade-off between inserting data into a database row-by-row and as binary blobs. Inserting the data row-by-row means that we have direct access to all the data and may perform queries directly on the values of the data. On the other hand, as we shall see, this is much slower than inserting the data directly as binary blobs.
First, we choose a new location for the database to ensure that we don’t add a bunch of benchmarking data to the default one.
[1]:
import os
cwd = os.getcwd()
import qcodes as qc
qc.config["core"]["db_location"] = os.path.join(cwd, 'testing.db')
Logging hadn't been started.
Activating auto-logging. Current session state plus future input saved.
Filename : /home/runner/.qcodes/logs/command_history.log
Mode : append
Output logging : True
Raw input log : False
Timestamping : True
State : active
Qcodes Logfile : /home/runner/.qcodes/logs/240426-13063-qcodes.log
[2]:
%matplotlib inline
import time
from pathlib import Path
import matplotlib.pyplot as plt
import numpy as np
import qcodes as qc
from qcodes.dataset import (
Measurement,
initialise_or_create_database_at,
load_or_create_experiment,
)
from qcodes.parameters import ManualParameter
[3]:
initialise_or_create_database_at(Path.cwd() / "dataset_performance.db")
exp = load_or_create_experiment(experiment_name='tutorial_exp', sample_name="no sample")
Here, we define a simple function to benchmark the time it takes to insert n points with either numeric or array data type. We will compare both the time used to call add_result and the time used for the full measurement.
[4]:
def insert_data(paramtype, npoints, nreps=1):
meas = Measurement(exp=exp)
x1 = ManualParameter('x1')
x2 = ManualParameter('x2')
x3 = ManualParameter('x3')
y1 = ManualParameter('y1')
y2 = ManualParameter('y2')
meas.register_parameter(x1, paramtype=paramtype)
meas.register_parameter(x2, paramtype=paramtype)
meas.register_parameter(x3, paramtype=paramtype)
meas.register_parameter(y1, setpoints=[x1, x2, x3],
paramtype=paramtype)
meas.register_parameter(y2, setpoints=[x1, x2, x3],
paramtype=paramtype)
start = time.perf_counter()
with meas.run() as datasaver:
start_adding = time.perf_counter()
for i in range(nreps):
datasaver.add_result((x1, np.random.rand(npoints)),
(x2, np.random.rand(npoints)),
(x3, np.random.rand(npoints)),
(y1, np.random.rand(npoints)),
(y2, np.random.rand(npoints)))
stop_adding = time.perf_counter()
run_id = datasaver.run_id
stop = time.perf_counter()
tot_time = stop - start
add_time = stop_adding - start_adding
return tot_time, add_time, run_id
Comparison between numeric/array data and binary blob
Case1: Short experiment time
[5]:
sizes = [1,500,1000,2000,3000,4000,5000]
t_numeric = []
t_numeric_add = []
t_array = []
t_array_add = []
for size in sizes:
tn, tna, run_id_n = insert_data('numeric', size)
t_numeric.append(tn)
t_numeric_add.append(tna)
ta, taa, run_id_a = insert_data('array', size)
t_array.append(ta)
t_array_add.append(taa)
Starting experimental run with id: 1.
Starting experimental run with id: 2.
Starting experimental run with id: 3.
Starting experimental run with id: 4.
Starting experimental run with id: 5.
Starting experimental run with id: 6.
Starting experimental run with id: 7.
Starting experimental run with id: 8.
Starting experimental run with id: 9.
Starting experimental run with id: 10.
Starting experimental run with id: 11.
Starting experimental run with id: 12.
Starting experimental run with id: 13.
Starting experimental run with id: 14.
[6]:
fig, ax = plt.subplots(1,1)
ax.plot(sizes, t_numeric, 'o-', label='Inserting row-by-row')
ax.plot(sizes, t_numeric_add, 'o-', label='Inserting row-by-row: add_result only')
ax.plot(sizes, t_array, 'd-', label='Inserting as binary blob')
ax.plot(sizes, t_array_add, 'd-', label='Inserting as binary blob: add_result only')
ax.legend()
ax.set_xlabel('Array length')
ax.set_ylabel('Time (s)')
fig.tight_layout()
As shown in the latter figure, the time to setup and and close the experiment is approximately 0.4 sec. In case of small array sizes, the difference between inserting values of data as arrays and inserting them row-by-row is relatively unimportant. At larger array sizes, i.e. above 10000 points, the cost of writing data as individual datapoints starts to become important.
Case2: Long experiment time
[7]:
sizes = [1,500,1000,2000,3000,4000,5000]
nreps = 100
t_numeric = []
t_numeric_add = []
t_numeric_run_ids = []
t_array = []
t_array_add = []
t_array_run_ids = []
for size in sizes:
tn, tna, run_id_n = insert_data('numeric', size, nreps=nreps)
t_numeric.append(tn)
t_numeric_add.append(tna)
t_numeric_run_ids.append(run_id_n)
ta, taa, run_id_a = insert_data('array', size, nreps=nreps)
t_array.append(ta)
t_array_add.append(taa)
t_array_run_ids.append(run_id_a)
Starting experimental run with id: 15.
Starting experimental run with id: 16.
Starting experimental run with id: 17.
Starting experimental run with id: 18.
Starting experimental run with id: 19.
Starting experimental run with id: 20.
Starting experimental run with id: 21.
Starting experimental run with id: 22.
Starting experimental run with id: 23.
Starting experimental run with id: 24.
Starting experimental run with id: 25.
Starting experimental run with id: 26.
Starting experimental run with id: 27.
Starting experimental run with id: 28.
[8]:
fig, ax = plt.subplots(1,1)
ax.plot(sizes, t_numeric, 'o-', label='Inserting row-by-row')
ax.plot(sizes, t_numeric_add, 'o-', label='Inserting row-by-row: add_result only')
ax.plot(sizes, t_array, 'd-', label='Inserting as binary blob')
ax.plot(sizes, t_array_add, 'd-', label='Inserting as binary blob: add_result only')
ax.legend()
ax.set_xlabel('Array length')
ax.set_ylabel('Time (s)')
fig.tight_layout()
However, as we increase the length of the experiment, as seen here by repeating the insertion 100 times, we see a big difference between inserting values of the data row-by-row and inserting it as a binary blob.
Loading the data
[9]:
from qcodes.dataset import load_by_id
As usual you can load the data by using the load_by_id function but you will notice that the different storage methods are reflected in shape of the data as it is retrieved.
[10]:
run_id_n = t_numeric_run_ids[0]
run_id_a = t_array_run_ids[0]
[11]:
ds = load_by_id(run_id_n)
ds.get_parameter_data('x1')
[11]:
{'x1': {'x1': array([0.87108794, 0.87108794, 0.93910358, 0.93910358, 0.9492624 ,
0.9492624 , 0.96340368, 0.96340368, 0.54875972, 0.54875972,
0.61703917, 0.61703917, 0.90704604, 0.90704604, 0.41438542,
0.41438542, 0.96814864, 0.96814864, 0.35127244, 0.35127244,
0.57862737, 0.57862737, 0.71251407, 0.71251407, 0.82601009,
0.82601009, 0.11950571, 0.11950571, 0.08702638, 0.08702638,
0.33118004, 0.33118004, 0.43064493, 0.43064493, 0.24269267,
0.24269267, 0.33967312, 0.33967312, 0.84200736, 0.84200736,
0.81791061, 0.81791061, 0.13245808, 0.13245808, 0.0249503 ,
0.0249503 , 0.83877468, 0.83877468, 0.1487899 , 0.1487899 ,
0.4543279 , 0.4543279 , 0.24920185, 0.24920185, 0.42739199,
0.42739199, 0.63543421, 0.63543421, 0.25518955, 0.25518955,
0.54095251, 0.54095251, 0.68850963, 0.68850963, 0.59844259,
0.59844259, 0.07842034, 0.07842034, 0.60078966, 0.60078966,
0.64153149, 0.64153149, 0.521788 , 0.521788 , 0.05306588,
0.05306588, 0.99919404, 0.99919404, 0.2305931 , 0.2305931 ,
0.8306734 , 0.8306734 , 0.35251291, 0.35251291, 0.82886264,
0.82886264, 0.49456706, 0.49456706, 0.65759632, 0.65759632,
0.66795333, 0.66795333, 0.05183711, 0.05183711, 0.31962715,
0.31962715, 0.85379809, 0.85379809, 0.30425052, 0.30425052,
0.8306853 , 0.8306853 , 0.53613724, 0.53613724, 0.48882484,
0.48882484, 0.85448159, 0.85448159, 0.41315423, 0.41315423,
0.5759593 , 0.5759593 , 0.21545999, 0.21545999, 0.07675106,
0.07675106, 0.31532978, 0.31532978, 0.71574996, 0.71574996,
0.38031678, 0.38031678, 0.94874709, 0.94874709, 0.14116921,
0.14116921, 0.95345647, 0.95345647, 0.34951193, 0.34951193,
0.97062273, 0.97062273, 0.35808013, 0.35808013, 0.23806148,
0.23806148, 0.81873624, 0.81873624, 0.83654468, 0.83654468,
0.31680219, 0.31680219, 0.80887936, 0.80887936, 0.21743484,
0.21743484, 0.45188699, 0.45188699, 0.09702649, 0.09702649,
0.97343307, 0.97343307, 0.29039638, 0.29039638, 0.58495043,
0.58495043, 0.88215712, 0.88215712, 0.24387462, 0.24387462,
0.98567144, 0.98567144, 0.02476672, 0.02476672, 0.29423741,
0.29423741, 0.75723888, 0.75723888, 0.86305228, 0.86305228,
0.85053256, 0.85053256, 0.54458345, 0.54458345, 0.32412747,
0.32412747, 0.3761167 , 0.3761167 , 0.13458721, 0.13458721,
0.43070077, 0.43070077, 0.67359323, 0.67359323, 0.60881627,
0.60881627, 0.5881546 , 0.5881546 , 0.3671089 , 0.3671089 ,
0.21442669, 0.21442669, 0.59089839, 0.59089839, 0.27874587,
0.27874587, 0.40575875, 0.40575875, 0.88532458, 0.88532458])}}
And a dataset stored as binary arrays
[12]:
ds = load_by_id(run_id_a)
ds.get_parameter_data('x1')
[12]:
{'x1': {'x1': array([[8.68297228e-01],
[8.68297228e-01],
[7.07237194e-01],
[7.07237194e-01],
[2.86098584e-01],
[2.86098584e-01],
[7.79645160e-01],
[7.79645160e-01],
[4.18088354e-01],
[4.18088354e-01],
[3.63082774e-01],
[3.63082774e-01],
[7.18825016e-01],
[7.18825016e-01],
[1.24623122e-01],
[1.24623122e-01],
[4.16409696e-01],
[4.16409696e-01],
[7.89078655e-02],
[7.89078655e-02],
[7.20000318e-01],
[7.20000318e-01],
[4.77823211e-01],
[4.77823211e-01],
[7.51959446e-01],
[7.51959446e-01],
[8.27590238e-01],
[8.27590238e-01],
[4.18412023e-01],
[4.18412023e-01],
[3.03847857e-01],
[3.03847857e-01],
[2.23895626e-01],
[2.23895626e-01],
[7.76875738e-01],
[7.76875738e-01],
[6.01474369e-01],
[6.01474369e-01],
[7.11309092e-01],
[7.11309092e-01],
[2.12440294e-01],
[2.12440294e-01],
[1.06427441e-01],
[1.06427441e-01],
[8.70421503e-01],
[8.70421503e-01],
[2.94853219e-01],
[2.94853219e-01],
[2.82133659e-01],
[2.82133659e-01],
[4.46358851e-02],
[4.46358851e-02],
[3.09019248e-02],
[3.09019248e-02],
[5.44402184e-01],
[5.44402184e-01],
[9.78533602e-01],
[9.78533602e-01],
[4.86929299e-01],
[4.86929299e-01],
[3.64564957e-01],
[3.64564957e-01],
[9.03555698e-01],
[9.03555698e-01],
[8.80261038e-02],
[8.80261038e-02],
[4.88970676e-01],
[4.88970676e-01],
[4.75915122e-01],
[4.75915122e-01],
[2.28669040e-01],
[2.28669040e-01],
[7.68617046e-02],
[7.68617046e-02],
[7.12227959e-01],
[7.12227959e-01],
[8.55480061e-01],
[8.55480061e-01],
[7.63355115e-01],
[7.63355115e-01],
[9.95475856e-02],
[9.95475856e-02],
[9.30757859e-01],
[9.30757859e-01],
[1.15510396e-02],
[1.15510396e-02],
[5.62833243e-01],
[5.62833243e-01],
[7.74772506e-01],
[7.74772506e-01],
[9.32969737e-01],
[9.32969737e-01],
[4.87541014e-01],
[4.87541014e-01],
[8.67304328e-01],
[8.67304328e-01],
[2.83064940e-01],
[2.83064940e-01],
[8.56077464e-01],
[8.56077464e-01],
[8.16765044e-01],
[8.16765044e-01],
[3.26859680e-01],
[3.26859680e-01],
[1.71125950e-02],
[1.71125950e-02],
[8.16313319e-01],
[8.16313319e-01],
[2.97899309e-04],
[2.97899309e-04],
[7.05062781e-01],
[7.05062781e-01],
[2.08276896e-01],
[2.08276896e-01],
[1.51774122e-02],
[1.51774122e-02],
[2.53767172e-02],
[2.53767172e-02],
[9.40595546e-01],
[9.40595546e-01],
[5.68515611e-01],
[5.68515611e-01],
[4.25741752e-01],
[4.25741752e-01],
[1.49648786e-01],
[1.49648786e-01],
[7.67290167e-03],
[7.67290167e-03],
[5.03891144e-01],
[5.03891144e-01],
[2.94079930e-01],
[2.94079930e-01],
[2.68559493e-01],
[2.68559493e-01],
[9.29646794e-01],
[9.29646794e-01],
[4.27370033e-02],
[4.27370033e-02],
[8.42764418e-01],
[8.42764418e-01],
[8.26854608e-01],
[8.26854608e-01],
[5.66516417e-01],
[5.66516417e-01],
[6.46039081e-02],
[6.46039081e-02],
[8.10080300e-01],
[8.10080300e-01],
[2.47094376e-01],
[2.47094376e-01],
[5.72682183e-02],
[5.72682183e-02],
[3.51137655e-01],
[3.51137655e-01],
[2.45539922e-01],
[2.45539922e-01],
[1.05254783e-01],
[1.05254783e-01],
[5.78808580e-01],
[5.78808580e-01],
[3.87040218e-01],
[3.87040218e-01],
[8.81964040e-01],
[8.81964040e-01],
[4.88683386e-02],
[4.88683386e-02],
[6.54434332e-01],
[6.54434332e-01],
[9.37725760e-01],
[9.37725760e-01],
[6.61201117e-01],
[6.61201117e-01],
[2.66522210e-01],
[2.66522210e-01],
[8.07864880e-01],
[8.07864880e-01],
[1.97221030e-01],
[1.97221030e-01],
[7.77958912e-01],
[7.77958912e-01],
[4.52098314e-01],
[4.52098314e-01],
[9.04486763e-01],
[9.04486763e-01],
[5.68063011e-01],
[5.68063011e-01],
[7.67483735e-03],
[7.67483735e-03],
[6.68773066e-01],
[6.68773066e-01],
[7.61746021e-01],
[7.61746021e-01],
[9.12816863e-02],
[9.12816863e-02],
[4.02187223e-01],
[4.02187223e-01],
[1.77741881e-02],
[1.77741881e-02],
[9.60388002e-01],
[9.60388002e-01]])}}
[ ]: