This page was generated from docs/examples/logging/logging_example.ipynb. Interactive online version: Binder badge.

Logging

TL;DR

  • There is a QCoDeS logging module: qcodes.logger

  • Call logger.start_all_logging at the start of every script/session to make sure all log messages get stored to the .qcodes/logs folder in your home directory, or set logger.start_logging_on_import to True in your qcodesrc.json configuration.

  • For debugging purposes you can log messages of an individual instrument (and see the VISA dialog).

  • You can obtain all logging messages in a pandas.DataFrame for further analysis.

Contents

Introduction

Python offers an excellent logging framework that is widely used in QCoDeS. As a user, you might be interested in reading the log messages that are generated by QCoDeS for debugging and also profiling purposes.

All log records(i.e., messages and metadata) are created using a python logging.logger object. In QCoDeS there is a logger for every module with the same name as the module itself, i.e., you can use logging.getLogger('qcodes.instrument.base') for getting the logger of the QCoDeS Instrument. From the dots in the names of the loggers a hierarchy is defined, at the top of which is the so called root logger that catches all messages. The log records created in a logger are processed by the handlers that have been attached to this specific logger. The handlers can, for example, be used to output the log messages into different locations, like the console or a log file.

Every logger, handler and log record has an associated level. The default levels are: CRITICAL, ERROR, WARNING, INFO and DEBUG. The levels determine how messages are passed: a logger passes only records that exceeds its minimal passing level to its handlers and the handlers in turn only output those messages that exceed their minimal passing level.

For a more detailed description of Python logging refer to the official manual (This is all well summed up in this chart)

Instrument communication loggers

Not all the used loggers in QCoDeS follow the previously presented rule of naming the logger after the module. For instruments there are the following loggers: - qcodes.instrument.base.com: This logger logs all the communication of the Instrument and its descendants with the actual physical instruments, e.g., the messages generated by the write and query calls. - qcodes.instrument.base.com.visa: This is a sublogger that logs all the communication related messages of the VisaInstrument and its descendants.

Set up logging in QCoDeS

At the beginning of every experiment script file you should include the following lines:

[1]:

from qcodes.logger import start_all_logging

start_all_logging()

Logging hadn't been started.
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/runner/.qcodes/logs/command_history.log
Mode           : append
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active
Qcodes Logfile : /home/runner/.qcodes/logs/241121-5745-qcodes.log

The call to start_all_logging does two things: 1. Create handlers to output logging messages from the root logger to the console and to a log file. 2. Enable logging of the IPython commands into a separate log file.

Alternatively we can configure qcodes to automatically start logging on every import of qcodes, by running the following code once. (This will persist the current configuration in ~\qcodesrc.json)

[2]:

from qcodes import config

config.logger.start_logging_on_import = "always"
config.save_to_home()

QCoDeS default handlers

The effect of the console handler can be seen in the two printed lines with a red background below the previous code cell. Every line corresponds to a log record. The fields of the records are separated by ¦ and have the following meaning: 0. time stamp 1. logger name, here the module name 2. record log level 3. name of the function that created the log message 4. line at which the message has been issued 5. actual log message

The same format is used for the file handler. By default QCoDeS logs to the file <user directory>/.qcodes/logs/qcodes.log. To avoid endlessly long files, the files roll over at midnight and get the date appended to their name where qcodes.log without date always gives you the log messages of the same day.

To configure the levels you want to log/handle, you need to edit the qcodesrc.json file in you home directory. The default values are:

"logger":{
    "console_level": "WARNING",
    "file_level": "INFO",
    "logger_levels":{
        "pyvisa": "INFO"
    }

While console_level and file_level describe the levels of the two default handlers described previously, logger_levels is a dictionary that can be used to limit the messages passed on by module level loggers. This means with these defaults, the pyvisa module will not log messages on a DEBUG level. Therefore setting the console_level to DEBUG will not show the pyvisa debug messages (you certainly don’t want to see them as they log every single character that is passed to an instrument with an individual message). On the other hand setting the console level to warning will still suppress pyvisa info messages.

IPython command history

The output above that follows these logging messages comes from IPython. It tells us that all issued IPython commands will be logged to a file. The file lives in the same log directory as the python logs and is called command_history.log. It will not roll over.

To change the command history logging directory you will need to call start_command_history_logger with the new path as an argument.

Temporarily elevating the logging level

Sometimes you might wish to catch more than that meets to the eye. To this end, one can temporarily elevate the logging level to catch all debug messages by using a context manager. This can simply be done via:

[3]:

import logging

from qcodes import logger

log = logging.getLogger("example_logger")

log.debug("This message will not be visible as the logging level is set to `DEBUG`")

with logger.console_level(logging.DEBUG):
    log.debug(
        "This message is visible as the logging level is temporarily elevated to `DEBUG`"
    )

log.debug("The level is back to what it used to be")

2024-11-21 06:54:11,743 ¦ example_logger ¦ DEBUG ¦ 2067591600 ¦ <module> ¦ 10 ¦ This message is visible as the logging level is temporarily elevated to `DEBUG`

For other handlers than the console handler, you can do the same thing using the handler_level context manager.

Filtering log messages by instrument

A very common use case for logging in QCoDeS is to analyze the communication with a VISA instrument. For this purpose the logger module provides a context manager to filter the logging messages by instrument.

Generally one could of course simply attach a handler to the logger of qcodes.instrument_drivers.Vendor.Type but this would not show the messages that are generated by the visa communication in qcodes.instrument.visa. Additionally with the approach implemented here one can not only filter messages for a given type of instrument but also of a specific instance.

To demonstrate this consider the following mock up of the AMI430 Magnet controller that consists of individual VISA instruments for each axis, here with the names x, y and z:

[4]:

import numpy as np

from qcodes.instrument_drivers.american_magnetics import AMIModel430, AMIModel4303D

mag_x = AMIModel430(
    "x", address="GPIB::1::INSTR", pyvisa_sim_file="AMI430.yaml", terminator="\n"
)
mag_y = AMIModel430(
    "y", address="GPIB::2::INSTR", pyvisa_sim_file="AMI430.yaml", terminator="\n"
)
mag_z = AMIModel430(
    "z", address="GPIB::3::INSTR", pyvisa_sim_file="AMI430.yaml", terminator="\n"
)

field_limit = [
    lambda x, y, z: x == 0 and y == 0 and z < 3,
    lambda x, y, z: np.linalg.norm([x, y, z]) < 2,
]

driver = AMIModel4303D("AMI430_3D", mag_x, mag_y, mag_z, field_limit)

Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.03s
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.00s
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.00s

In the above messages you can see the prefix [<instrument_name>(<instrument_type>)]. To analyze the visa communication one can use filter_instrument:

[5]:

driver.cartesian((0, 0, 0))
with logger.console_level("DEBUG"):
    with logger.filter_instrument(mag_x):
        driver.cartesian((0, 0, 1))

2024-11-21 06:54:11,806 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [x(AMIModel430)] Querying: STATE?
2024-11-21 06:54:11,807 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [x(AMIModel430)] Response: 2
2024-11-21 06:54:11,808 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [x(AMIModel430)] Querying: FIELD:MAG?
2024-11-21 06:54:11,809 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [x(AMIModel430)] Response: 0.0
2024-11-21 06:54:11,810 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [x(AMIModel430)] Querying: FIELD:MAG?
2024-11-21 06:54:11,810 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [x(AMIModel430)] Response: 0.0

The output on the console shows as expected only messages from the x instrument. For multiple instruments pass a sequence of instruments to filter_instrument we write:

[6]:

driver.cartesian((0, 0, 0))
with logger.console_level("DEBUG"):
    with logger.filter_instrument((mag_x, mag_y)):
        driver.cartesian((0, 0, 1))

2024-11-21 06:54:11,822 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [x(AMIModel430)] Querying: STATE?
2024-11-21 06:54:11,823 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [x(AMIModel430)] Response: 2
2024-11-21 06:54:11,824 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [y(AMIModel430)] Querying: STATE?
2024-11-21 06:54:11,825 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [y(AMIModel430)] Response: 2
2024-11-21 06:54:11,825 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [x(AMIModel430)] Querying: FIELD:MAG?
2024-11-21 06:54:11,826 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [x(AMIModel430)] Response: 0.0
2024-11-21 06:54:11,826 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [y(AMIModel430)] Querying: FIELD:MAG?
2024-11-21 06:54:11,827 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [y(AMIModel430)] Response: 0.0
2024-11-21 06:54:11,828 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [x(AMIModel430)] Querying: FIELD:MAG?
2024-11-21 06:54:11,828 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [x(AMIModel430)] Response: 0.0
2024-11-21 06:54:11,829 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 389 ¦ [y(AMIModel430)] Querying: FIELD:MAG?
2024-11-21 06:54:11,830 ¦ qcodes.instrument.instrument_base.com.visa ¦ DEBUG ¦ visa ¦ ask_raw ¦ 391 ¦ [y(AMIModel430)] Response: 0.0

Capturing Pandas.DataFrame

To process the logs, especially with the timestamps, the pandas module is highly suited. With logger.log_to_dataframe or logger.logfile_to_dataframe you can convert a log(file) to a pandas DataFrame. See this notebook for an example.

You can also use a context manager to capture the logs directly into a DataFrame.

[7]:

from qcodes.logger import capture_dataframe

[8]:

with logger.console_level(logging.WARN):
    driver.cartesian((0, 0, 0))
    with capture_dataframe(level="DEBUG") as (handler, get_dataframe):
        driver.cartesian((0, 0, 1))
        df = get_dataframe()
        driver.cartesian((0, 0, 2))
        df2 = get_dataframe()  # this is the cummulative log

[9]:

df

[9]:
asctime name levelname module funcName lineno message
0 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base DEBUG AMI430_visa _adjust_child_instruments 1088 [AMI430_3D(AMIModel4303D)] Checking whether fi...
1 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: STATE?
2 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 2
3 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [y(AMIModel430)] Querying: STATE?
4 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [y(AMIModel430)] Response: 2
5 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: STATE?
6 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2
7 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base DEBUG AMI430_visa _adjust_child_instruments 1103 [AMI430_3D(AMIModel4303D)] Field values OK, pr...
8 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base DEBUG parameter _get_manual_parameter 194 [AMI430_3D(AMIModel4303D)] Getting raw value o...
9 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: FIELD:MAG?
10 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 0.0
11 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [y(AMIModel430)] Querying: FIELD:MAG?
12 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [y(AMIModel430)] Response: 0.0
13 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: FIELD:MAG?
14 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 0.0
15 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: FIELD:MAG?
16 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 0.0
17 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [y(AMIModel430)] Querying: FIELD:MAG?
18 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [y(AMIModel430)] Response: 0.0
19 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: FIELD:MAG?
20 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 0.0
21 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base DEBUG parameter _get_manual_parameter 194 [AMI430_3D(AMIModel4303D)] Getting raw value o...
22 2024-11-21 06:54:11,845 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: COIL?
23 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2.0
24 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: CURR:LIMIT?
25 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 80
26 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: QU?
27 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 0
28 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: PERS?
29 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 0
30 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: STATE?
31 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2
32 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa write_raw 372 [z(AMIModel430)] Writing: PAUSE
33 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa write_raw 372 [z(AMIModel430)] Writing: CONF:FIELD:TARG 1.0
34 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: PS:INST?
35 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 0
36 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa write_raw 372 [z(AMIModel430)] Writing: RAMP
37 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base DEBUG AMI430_visa set_field 494 [z(AMIModel430)] Starting blocking ramp of z t...
38 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: STATE?
39 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2
40 2024-11-21 06:54:11,846 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: STATE?
41 2024-11-21 06:54:11,847 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2
42 2024-11-21 06:54:11,847 qcodes.instrument.instrument_base DEBUG AMI430_visa set_field 496 [z(AMIModel430)] Finished blocking ramp 
[10]:

df2

[10]:
asctime name levelname module funcName lineno message
0 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base DEBUG AMI430_visa _adjust_child_instruments 1088 [AMI430_3D(AMIModel4303D)] Checking whether fi...
1 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: STATE?
2 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 2
3 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [y(AMIModel430)] Querying: STATE?
4 2024-11-21 06:54:11,844 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [y(AMIModel430)] Response: 2
... ... ... ... ... ... ... ...
81 2024-11-21 06:54:12,057 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: STATE?
82 2024-11-21 06:54:12,057 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2
83 2024-11-21 06:54:12,057 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [z(AMIModel430)] Querying: STATE?
84 2024-11-21 06:54:12,057 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [z(AMIModel430)] Response: 2
85 2024-11-21 06:54:12,057 qcodes.instrument.instrument_base DEBUG AMI430_visa set_field 496 [z(AMIModel430)] Finished blocking ramp

86 rows × 7 columns

You can of course combine the context managers:

[11]:

with logger.console_level(logging.WARN):
    driver.cartesian((0, 0, 0))
    with capture_dataframe(level="DEBUG") as (handler, get_dataframe):
        with logger.filter_instrument(mag_x, handler=handler):
            driver.cartesian((0, 0, 1))
            df = get_dataframe()
df

[11]:
asctime name levelname module funcName lineno message
0 2024-11-21 06:54:12,090 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: STATE?
1 2024-11-21 06:54:12,090 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 2
2 2024-11-21 06:54:12,090 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: FIELD:MAG?
3 2024-11-21 06:54:12,090 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 0.0
4 2024-11-21 06:54:12,091 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 389 [x(AMIModel430)] Querying: FIELD:MAG?
5 2024-11-21 06:54:12,091 qcodes.instrument.instrument_base.com.visa DEBUG visa ask_raw 391 [x(AMIModel430)] Response: 0.0

For an analysis of the timestamps please also refer to the log analysis example notebook.

Telemetry

The QCoDeS Measurement Context manager, DataSaverBuilder and DoND functions are instrumented as OpenTelemetry traces. This enables users to correlate log messages with the the measurement that they were performed within. See the (OpenTelemetry)[https://opentelemetry.io/] documentation for examples of how this can be used.

[ ]: