mutcleaner.cleaners.base_config#

Classes

BaseCleanerConfig([pipeline_name, ...])

Base configuration class for all dataset cleaners
class mutcleaner.cleaners.base_config.BaseCleanerConfig(pipeline_name='base_cleaner', num_workers=16, validate_config=True)[source]#

Bases: ABC

Base configuration class for all dataset cleaners

This abstract base class provides common configuration functionality that can be inherited by specific cleaner configurations.

Attributes:
pipeline_namestr

Name of the cleaning pipeline

num_workersint

Default number of worker processes

validate_configbool

Whether to validate configuration before use

Methods

from_dict(config_dict)

Create configuration object from dictionary

from_json(json_path)

Load configuration from JSON file

get_summary()

Get a human-readable summary of the configuration

merge(partial_config)

Merge partial configuration with current configuration

to_dict([exclude_callables])

Convert configuration to dictionary

to_json(json_path, **json_kwargs)

Save configuration to JSON file

validate()

Validate the configuration
classmethod from_dict(config_dict)[source]#

Create configuration object from dictionary

Parameters:

config_dict (Dict[str, Any]) – Dictionary containing configuration parameters

Return type:

TypeVar(CleanerConfigType, bound= BaseCleanerConfig)

Returns:

Configuration object

classmethod from_json(json_path)[source]#

Load configuration from JSON file

Parameters:

json_path (Union[str, Path]) – Path to JSON configuration file

Return type:

TypeVar(CleanerConfigType, bound= BaseCleanerConfig)

Returns:

Configuration object

Raises:

FileNotFoundError – If configuration file does not exist

get_summary()[source]#

Get a human-readable summary of the configuration

Return type:

str

Returns:

String summary of the configuration

merge(partial_config)[source]#

Merge partial configuration with current configuration

Parameters:

partial_config (Dict[str, Any]) – Dictionary containing configuration values to update

Return type:

TypeVar(CleanerConfigType, bound= BaseCleanerConfig)

Returns:

New configuration object with merged values

num_workers: int = 16#
pipeline_name: str = 'base_cleaner'#
to_dict(exclude_callables=True)[source]#

Convert configuration to dictionary

Parameters:

exclude_callables (bool) – Whether to exclude callable objects (functions, lambdas), by default True

Return type:

Dict[str, Any]

Returns:

Dictionary representation of the configuration

to_json(json_path, **json_kwargs)[source]#

Save configuration to JSON file

Parameters:

  • json_path (Union[str, Path]) – Path where to save the JSON file

  • **json_kwargs – Additional arguments passed to json.dump

Return type:

None

abstractmethod validate()[source]#

Validate the configuration

This method should be implemented by subclasses to perform specific validation logic.

Raises:

ValueError – If configuration is invalid

Return type:

None

validate_config: bool = True#