This page was generated from
docs/examples/logging/logging_example.ipynb.
Interactive online version:
.
Logging
TL;DR
There is a QCoDeS logging module:
qcodes.loggerCall
logger.start_all_loggingat the start of every script/session to make sure all log messages get stored to the.qcodes/logsfolder in your home directory, or setlogger.start_logging_on_importtoTruein yourqcodesrc.jsonconfiguration.For debugging purposes you can log messages of an individual instrument (and see the VISA dialog).
You can obtain all logging messages in a
pandas.DataFramefor further analysis.
Contents
Introduction
Python offers an excellent logging framework that is widely used in QCoDeS. As a user, you might be interested in reading the log messages that are generated by QCoDeS for debugging and also profiling purposes.
All log records(i.e., messages and metadata) are created using a python logging.logger object. In QCoDeS there is a logger for every module with the same name as the module itself, i.e., you can use logging.getLogger('qcodes.instrument.base') for getting the logger of the QCoDeS Instrument. From the dots in the names of the loggers a hierarchy is defined, at the top of which is the so called root logger that catches all messages. The log records created in a logger are processed by
the handlers that have been attached to this specific logger. The handlers can, for example, be used to output the log messages into different locations, like the console or a log file.
Every logger, handler and log record has an associated level. The default levels are: CRITICAL, ERROR, WARNING, INFO and DEBUG. The levels determine how messages are passed: a logger passes only records that exceeds its minimal passing level to its handlers and the handlers in turn only output those messages that exceed their minimal passing level.
For a more detailed description of Python logging refer to the official manual (This is all well summed up in this chart)
Instrument communication loggers
Not all the used loggers in QCoDeS follow the previously presented rule of naming the logger after the module. For instruments there are the following loggers: - qcodes.instrument.base.com: This logger logs all the communication of the Instrument and its descendants with the actual physical instruments, e.g., the messages generated by the write and query calls. - qcodes.instrument.base.com.visa: This is a sublogger that logs all the communication related messages of the
VisaInstrument and its descendants.
Set up logging in QCoDeS
At the beginning of every experiment script file you should include the following lines:
[1]:
from qcodes.logger import start_all_logging
start_all_logging()
Logging hadn't been started.
Activating auto-logging. Current session state plus future input saved.
Filename : /home/runner/.qcodes/logs/command_history.log
Mode : append
Output logging : True
Raw input log : False
Timestamping : True
State : active
Qcodes Logfile : /home/runner/.qcodes/logs/240508-5238-qcodes.log
The call to start_all_logging does two things: 1. Create handlers to output logging messages from the root logger to the console and to a log file. 2. Enable logging of the IPython commands into a separate log file.
Alternatively we can configure qcodes to automatically start logging on every import of qcodes, by running the following code once. (This will persist the current configuration in ~\qcodesrc.json)
[2]:
from qcodes import config
config.logger.start_logging_on_import = 'always'
config.save_to_home()
QCoDeS default handlers
The effect of the console handler can be seen in the two printed lines with a red background below the previous code cell. Every line corresponds to a log record. The fields of the records are separated by ¦ and have the following meaning: 0. time stamp 1. logger name, here the module name 2. record log level 3. name of the function that created the log message 4. line at which the message has been issued 5. actual log message
The same format is used for the file handler. By default QCoDeS logs to the file <user directory>/.qcodes/logs/qcodes.log. To avoid endlessly long files, the files roll over at midnight and get the date appended to their name where qcodes.log without date always gives you the log messages of the same day.
To configure the levels you want to log/handle, you need to edit the qcodesrc.json file in you home directory. The default values are:
"logger":{
"console_level": "WARNING",
"file_level": "INFO",
"logger_levels":{
"pyvisa": "INFO"
}
While console_level and file_level describe the levels of the two default handlers described previously, logger_levels is a dictionary that can be used to limit the messages passed on by module level loggers. This means with these defaults, the pyvisa module will not log messages on a DEBUG level. Therefore setting the console_level to DEBUG will not show the pyvisa debug messages (you certainly don’t want to see them as they log every single character that is
passed to an instrument with an individual message). On the other hand setting the console level to warning will still suppress pyvisa info messages.
IPython command history
The output above that follows these logging messages comes from IPython. It tells us that all issued IPython commands will be logged to a file. The file lives in the same log directory as the python logs and is called command_history.log. It will not roll over.
To change the command history logging directory you will need to call start_command_history_logger with the new path as an argument.
Temporarily elevating the logging level
Sometimes you might wish to catch more than that meets to the eye. To this end, one can temporarily elevate the logging level to catch all debug messages by using a context manager. This can simply be done via:
[3]:
import logging
from qcodes import logger
log = logging.getLogger('example_logger')
log.debug('This message will not be visible as the logging level is set to `DEBUG`')
with logger.console_level(logging.DEBUG):
log.debug('This message is visible as the logging level is temporarily elevated to `DEBUG`')
log.debug('The level is back to what it used to be')
For other handlers than the console handler, you can do the same thing using the handler_level context manager.
Filtering log messages by instrument
A very common use case for logging in QCoDeS is to analyze the communication with a VISA instrument. For this purpose the logger module provides a context manager to filter the logging messages by instrument.
Generally one could of course simply attach a handler to the logger of qcodes.instrument_drivers.Vendor.Type but this would not show the messages that are generated by the visa communication in qcodes.instrument.visa. Additionally with the approach implemented here one can not only filter messages for a given type of instrument but also of a specific instance.
To demonstrate this consider the following mock up of the AMI430 Magnet controller that consists of individual VISA instruments for each axis, here with the names x, y and z:
[4]:
from qcodes.instrument_drivers.american_magnetics import AMIModel430, AMIModel4303D
mag_x = AMIModel430('x', address='GPIB::1::INSTR', pyvisa_sim_file="AMI430.yaml",
terminator='\n')
mag_y = AMIModel430('y', address='GPIB::2::INSTR', pyvisa_sim_file="AMI430.yaml",
terminator='\n')
mag_z = AMIModel430('z', address='GPIB::3::INSTR', pyvisa_sim_file="AMI430.yaml",
terminator='\n')
import numpy as np
field_limit = [
lambda x, y, z: x == 0 and y == 0 and z < 3,
lambda x, y, z: np.linalg.norm([x, y, z]) < 2
]
driver = AMIModel4303D("AMI430_3D", mag_x, mag_y, mag_z, field_limit)
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.03s
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.00s
Connected to: QCoDeS AMI430_simulation (serial:1337, firmware:0.0.01) in 0.00s
In the above messages you can see the prefix [<instrument_name>(<instrument_type>)]. To analyze the visa communication one can use filter_instrument:
[5]:
driver.cartesian((0, 0, 0))
with logger.console_level('DEBUG'):
with logger.filter_instrument(mag_x):
driver.cartesian((0, 0, 1))
The output on the console shows as expected only messages from the x instrument. For multiple instruments pass a sequence of instruments to filter_instrument we write:
[6]:
driver.cartesian((0, 0, 0))
with logger.console_level('DEBUG'):
with logger.filter_instrument((mag_x, mag_y)):
driver.cartesian((0, 0, 1))
Capturing Pandas.DataFrame
To process the logs, especially with the timestamps, the pandas module is highly suited. With logger.log_to_dataframe or logger.logfile_to_dataframe you can convert a log(file) to a pandas DataFrame. See this notebook for an example.
You can also use a context manager to capture the logs directly into a DataFrame.
[7]:
from qcodes.logger import capture_dataframe
[8]:
with logger.console_level(logging.WARN):
driver.cartesian((0, 0, 0))
with capture_dataframe(level='DEBUG') as (handler, get_dataframe):
driver.cartesian((0, 0, 1))
df = get_dataframe()
driver.cartesian((0, 0, 2))
df2 = get_dataframe() # this is the cummulative log
[9]:
df
[9]:
| asctime | name | levelname | module | funcName | lineno | message | |
|---|---|---|---|---|---|---|---|
| 0 | 2024-05-08 11:36:44,574 | qcodes.instrument.instrument_base | DEBUG | AMI430_visa | _adjust_child_instruments | 977 | [AMI430_3D(AMIModel4303D)] Checking whether fi... |
| 1 | 2024-05-08 11:36:44,574 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: STATE? |
| 2 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 2 |
| 3 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [y(AMIModel430)] Querying: STATE? |
| 4 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [y(AMIModel430)] Response: 2 |
| 5 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: STATE? |
| 6 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2 |
| 7 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base | DEBUG | AMI430_visa | _adjust_child_instruments | 993 | [AMI430_3D(AMIModel4303D)] Field values OK, pr... |
| 8 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base | DEBUG | parameter | _get_manual_parameter | 191 | [AMI430_3D(AMIModel4303D)] Getting raw value o... |
| 9 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: FIELD:MAG? |
| 10 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 0.0 |
| 11 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [y(AMIModel430)] Querying: FIELD:MAG? |
| 12 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [y(AMIModel430)] Response: 0.0 |
| 13 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: FIELD:MAG? |
| 14 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 0.0 |
| 15 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: FIELD:MAG? |
| 16 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 0.0 |
| 17 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [y(AMIModel430)] Querying: FIELD:MAG? |
| 18 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [y(AMIModel430)] Response: 0.0 |
| 19 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: FIELD:MAG? |
| 20 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 0.0 |
| 21 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base | DEBUG | parameter | _get_manual_parameter | 191 | [AMI430_3D(AMIModel4303D)] Getting raw value o... |
| 22 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: COIL? |
| 23 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2.0 |
| 24 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: CURR:LIMIT? |
| 25 | 2024-05-08 11:36:44,576 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 80 |
| 26 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: QU? |
| 27 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 0 |
| 28 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: PERS? |
| 29 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 0 |
| 30 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: STATE? |
| 31 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2 |
| 32 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | write_raw | 314 | [z(AMIModel430)] Writing: PAUSE |
| 33 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | write_raw | 314 | [z(AMIModel430)] Writing: CONF:FIELD:TARG 1.0 |
| 34 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: PS:INST? |
| 35 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 0 |
| 36 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | write_raw | 314 | [z(AMIModel430)] Writing: RAMP |
| 37 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base | DEBUG | AMI430_visa | set_field | 409 | [z(AMIModel430)] Starting blocking ramp of z t... |
| 38 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: STATE? |
| 39 | 2024-05-08 11:36:44,577 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2 |
| 40 | 2024-05-08 11:36:44,578 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: STATE? |
| 41 | 2024-05-08 11:36:44,578 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2 |
| 42 | 2024-05-08 11:36:44,578 | qcodes.instrument.instrument_base | DEBUG | AMI430_visa | set_field | 411 | [z(AMIModel430)] Finished blocking ramp |
[10]:
df2
[10]:
| asctime | name | levelname | module | funcName | lineno | message | |
|---|---|---|---|---|---|---|---|
| 0 | 2024-05-08 11:36:44,574 | qcodes.instrument.instrument_base | DEBUG | AMI430_visa | _adjust_child_instruments | 977 | [AMI430_3D(AMIModel4303D)] Checking whether fi... |
| 1 | 2024-05-08 11:36:44,574 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: STATE? |
| 2 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 2 |
| 3 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [y(AMIModel430)] Querying: STATE? |
| 4 | 2024-05-08 11:36:44,575 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [y(AMIModel430)] Response: 2 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 81 | 2024-05-08 11:36:44,787 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: STATE? |
| 82 | 2024-05-08 11:36:44,787 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2 |
| 83 | 2024-05-08 11:36:44,787 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [z(AMIModel430)] Querying: STATE? |
| 84 | 2024-05-08 11:36:44,787 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [z(AMIModel430)] Response: 2 |
| 85 | 2024-05-08 11:36:44,787 | qcodes.instrument.instrument_base | DEBUG | AMI430_visa | set_field | 411 | [z(AMIModel430)] Finished blocking ramp |
86 rows × 7 columns
You can of course combine the context managers:
[11]:
with logger.console_level(logging.WARN):
driver.cartesian((0, 0, 0))
with capture_dataframe(level='DEBUG') as (handler, get_dataframe):
with logger.filter_instrument(mag_x, handler=handler):
driver.cartesian((0, 0, 1))
df = get_dataframe()
df
[11]:
| asctime | name | levelname | module | funcName | lineno | message | |
|---|---|---|---|---|---|---|---|
| 0 | 2024-05-08 11:36:44,827 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: STATE? |
| 1 | 2024-05-08 11:36:44,827 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 2 |
| 2 | 2024-05-08 11:36:44,828 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: FIELD:MAG? |
| 3 | 2024-05-08 11:36:44,828 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 0.0 |
| 4 | 2024-05-08 11:36:44,828 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 328 | [x(AMIModel430)] Querying: FIELD:MAG? |
| 5 | 2024-05-08 11:36:44,828 | qcodes.instrument.instrument_base.com.visa | DEBUG | visa | ask_raw | 330 | [x(AMIModel430)] Response: 0.0 |
For an analysis of the timestamps please also refer to the log analysis example notebook.
Telemetry
The QCoDeS Measurement Context manager, DataSaverBuilder and DoND functions are instrumented as OpenTelemetry traces. This enables users to correlate log messages with the the measurement that they were performed within. See the (OpenTelemetry)[https://opentelemetry.io/] documentation for examples of how this can be used.
[ ]: