pyspark.logger.PySparkLogger

class pyspark.logger.PySparkLogger(name='PySparkLogger')

Custom logging.Logger wrapper for PySpark that logs messages in a structured JSON format.

PySparkLogger extends the standard Python logging.Logger class, allowing seamless integration with existing logging setups. It customizes the log output to JSON format, including additional context information, making it more useful for PySpark applications.

New in version 4.0.0.

Methods

addFilter(filter)	Add the specified filter to this handler.
addHandler(handler)	Add the specified handler to this logger in structured JSON format.
callHandlers(record)	Pass a record to all relevant handlers.
critical(msg, *args, **kwargs)	Log 'msg % args' with severity 'CRITICAL'.
debug(msg, *args, **kwargs)	Log 'msg % args' with severity 'DEBUG'.
error(msg, *args, **kwargs)	Log 'msg % args' with severity 'ERROR' in structured JSON format.
exception(msg, *args, **kwargs)	Convenience method for logging an ERROR with exception information.
fatal(msg, *args, **kwargs)	Log 'msg % args' with severity 'CRITICAL'.
filter(record)	Determine if a record is loggable by consulting all the filters.
findCaller([stack_info, stacklevel])	Find the stack frame of the caller so that we can note the source file name, line number and function name.
getChild(suffix)	Get a logger which is a descendant to this one.
getEffectiveLevel()	Get the effective level for this logger.
getLogger([name])	Return a PySparkLogger with the specified name, creating it if necessary.
handle(record)	Call the handlers for the specified record.
hasHandlers()	See if this logger has any handlers configured.
info(msg, *args, **kwargs)	Log 'msg % args' with severity 'INFO' in structured JSON format.
isEnabledFor(level)	Is this logger enabled for level 'level'?
log(level, msg, *args, **kwargs)	Log 'msg % args' with the integer severity 'level'.
makeRecord(name, level, fn, lno, msg, args, ...)	A factory method which can be overridden in subclasses to create specialized LogRecords.
removeFilter(filter)	Remove the specified filter from this handler.
removeHandler(hdlr)	Remove the specified handler from this logger.
setLevel(level)	Set the logging level of this logger.
warn(msg, *args, **kwargs)
warning(msg, *args, **kwargs)	Log 'msg % args' with severity 'WARNING' in structured JSON format.

Attributes

manager
root