matbench_genmetrics.core package

Subpackages

Submodules

matbench_genmetrics.core.metrics module

Core functionality for matbench-genmetrics (generative materials benchmarking)

class matbench_genmetrics.core.metrics.GenMatcher(test_structures, gen_structures: List[Structure] | None = None, test_comp_fingerprints: ndarray | None = None, gen_comp_fingerprints: ndarray | None = None, test_struct_fingerprints: ndarray | None = None, gen_struct_fingerprints: ndarray | None = None, verbose=True, match_type='cdvae_coverage', **match_kwargs)[source]

Bases: object

A class for matching generated structures to test structures.

Parameters:

  • test_structures (List[pymatgen.Structure]) – A list of test structures.

  • gen_structures (List[pymatgen.Structure]) – A list of generated structures.

  • match_type (str, optional) – The type of matching algorithm to use. Default is “StructureMatcher”.

  • match_kwargs (dict, optional) – Additional keyword arguments to pass to the matching algorithm.

match_matrix

A matrix of match scores between test and generated structures.

Type:

numpy.ndarray

property duplicity_count
property duplicity_counts

Get number of duplicates within the match matrix ignoring self-comparison.

property duplicity_rate

Get number of duplicates divided by number of possible duplicate locations.

A set with 4 instances of the same structure will score lower than 2 repeat instances each for 2 structures. In other words, the metric favors a larger of unique structures, even if repeat structures exist.

property match_count
property match_counts
property match_matrix

A matrix of match scores between test and generated structures.

match_matrixnumpy.ndarray

The element at position (i, j) represents the match score between the ith test structure and the jth generated structure. The match score is calculated using the specified matching algorithm and any additional keyword arguments passed to the GenMatcher class.

>>> from pymatgen.core.structure import Structure
>>> from pymatgen.core.lattice import Lattice
>>> test_structures = [
...     Structure(
...         Lattice.cubic(3.0), ["Si", "Si"], [[0, 0, 0], [0.5, 0.5, 0.5]]
...     ),
...     Structure(
...         Lattice.cubic(3.0), ["Si", "Si"], [[0, 0, 0], [0.5, 0.5, 0.5]]
...     ),
... ]
>>> gen_structures = [
...     Structure(
...         Lattice.cubic(3.0), ["Si", "Si"], [[0, 0, 0], [0.5, 0.5, 0.5]]
...     ),
...     Structure(
...         Lattice.cubic(3.0), ["Si", "Si"], [[0, 0, 0], [0.5, 0.5, 0.5]]
...     ),
... ]
>>> matcher = GenMatcher(test_structures, gen_structures)
>>> matcher.match_matrix
array([[1., 1.],
    [1., 1.]])
property match_rate
class matbench_genmetrics.core.metrics.GenMetrics(train_structures: List[Structure], test_structures: List[Structure], gen_structures: List[Structure], train_comp_fingerprints=None, test_comp_fingerprints=None, train_struct_fingerprints=None, test_struct_fingerprints=None, train_test_spg=None, train_test_modpetti_df=None, test_pred_structures=None, verbose=True, match_type='cdvae_coverage', **match_kwargs)[source]

Bases: object

Evaluate the performance of a generative model on a set of training and test data.

Parameters:

  • train_structures (List[pymatgen.Structure]) – A list of training structures.

  • test_structures (List[pymatgen.Structure]) – A list of test structures.

  • gen_structures (List[pymatgen.Structure]) – A list of generated structures.

  • test_pred_structures (List[pymatgen.Structure]) – A list of test structures predicted by the machine learning model.

  • match_type (str, optional) – The type of matching algorithm to use. Default is “StructureMatcher”.

  • match_kwargs (dict, optional) – Additional keyword arguments to pass to the matching algorithm.

train_test_match_matrix

A matrix of match scores between training and test structures. The element at position (i, j) represents the match score between the ith training structure and the jth test structure. The match score is calculated using the specified matching algorithm and any additional keyword arguments passed to the GenMetrics class.

Type:

numpy.ndarray

test_pred_match_matrix

A matrix of match scores between test structures and predicted test structures. The element at position (i, j) represents the match score between the ith test structure and the jth predicted test structure. The match score is calculated using the specified matching algorithm and any additional keyword arguments passed to the GenMetrics class.

Type:

numpy.ndarray

train_gen_match_matrix

A matrix of match scores between training structures and generated structures. The element at position (i, j) represents the match score between the ith training structure and the jth generated structure. The match score is calculated using the specified matching algorithm and any additional keyword arguments passed to the GenMetrics class.

Type:

numpy.ndarray

test_gen_match_matrix

A matrix of match scores between test structures and generated structures. The element at position (i, j) represents the match score between the ith test structure and the jth generated structure. The match score is calculated using the specified matching algorithm and any additional keyword arguments passed to the GenMetrics class.

Type:

numpy.ndarray

property coverage

Match rate between test structures and generated structures.

property metrics

Return validity, coverage, novelty, and uniqueness metrics as a dict.

property novelty

One minus match rate between train structures and generated structures.

property uniqueness

One minus duplicity rate within generated structures.

property validity

Scaled Wasserstein distance between real (train/test) and gen structures.

class matbench_genmetrics.core.metrics.MPTSMetrics(dummy=False, verbose=True, num_gen=None, save_dir='results', match_type='cdvae_coverage', **match_kwargs)[source]

Bases: object

Evaluate the performance of a crystal generative model using MP Time Split (MPTS).

Parameters:

  • dummy (bool, optional) – Whether to use dummy data for testing purposes. Default is False.

  • verbose (bool, optional) – Whether to print out the results of the evaluation. Default is True.

  • num_gen (int, optional) – The number of generated structures to use for the evaluation. Default is None.

  • save_dir (str, optional) – The directory to save the generated structures to. Default is “results”.

  • match_type (str, optional) – The type of matching algorithm to use. Default is “StructureMatcher”.

train_scores

A list of dictionaries containing the evaluation results for each fold of the training data.

Type:

List[Dict[str, float]]

val_scores

A list of dictionaries containing the evaluation results for each fold of the validation data.

Type:

List[Dict[str, float]]

test_scores

A list of dictionaries containing the evaluation results for each fold of the test data.

Type:

List[Dict[str, float]]

gen_scores

A list of dictionaries containing the evaluation results for each fold of the generated data.

Type:

List[Dict[str, float]]

test_pred_scores

A list of dictionaries containing the evaluation results for each fold of the predicted test data.

Type:

List[Dict[str, float]]

\*\*match_kwargs

Keyword arguments passed to GenMetrics.

Type:

Dict[str, Any]

Notes

This class assumes that the data is split into training, validation, and test sets using the MP Time Split (MPTS) protocol. The get_train_and_val_data method is used to retrieve the training and validation data for a given fold, and the evaluate_and_record method is used to evaluate the performance of the model on the generated structures and record the results. The evaluation results are stored in the train_scores, val_scores, test_scores, gen_scores, and test_pred_scores attributes.

evaluate_and_record(fold: int, gen_structures, test_pred_structures=None)[source]

Evaluate generated structures and record metrics.

Parameters:

  • fold (int) – Fold number.

  • gen_structures (list of pymatgen Structure)

  • structures. (List of generated)

  • test_pred_structures (list of pymatgen Structure, optional) – List of predicted structures for the test set. If not provided, the test set is assumed to be the same as the validation set.

get_train_and_val_data(fold: int, return_test=False)[source]

Get the MPTimeSplit fingerprints, sp.grp numbers, and modPetti info.

Parameters:

  • fold (int) – Which of the 5 folds to use for training and validation (0-4)

  • return_test (bool, optional) – Whether to return the test data in addition to the training and validation data, by default False

Returns:

  • DataFrame – Training and validation inputs

  • DataFrame – Test inputs. Only returned if return_test is True.

Examples

>>> mptm = MPTSMetrics()
>>> train_and_val_inputs = mptm.get_train_and_val_data(fold, return_test=False)
load(fpath)[source]
load_fingerprints(dummy=False)[source]

Load precalculated fingerprints from FigShare.

Parameters:

dummy (bool, optional) – Whether to load a small, dummy dataset, by default False

Returns:

Compositional fingerprints

Return type:

DataFrame

Examples

>>> mptm = MPTSMetrics()
>>> comp_fingerprints_df, struct_fingerprints_df = mptm.load_fingerprints(
...     dummy=False
... )
load_space_group_and_mod_petti(dummy=False)[source]

Load space groups and modified pettifor encodings from FigShare.

Parameters:

dummy (bool, optional) – Whether to load a small, dummy dataset, by default False

Returns:

  • DataFrame – space group numbers

  • DataFrame – modified pettifor encodings

Examples

>>> mptm = MPTSMetrics()
>>> spg_df, modpetti_df = mptm.load_space_group_and_mod_petti(dummy=False)
save(fpath_stem)[source]
class matbench_genmetrics.core.metrics.MPTSMetrics10(dummy=False, verbose=True)[source]

Bases: MPTSMetrics

Benchmark class for MPTSMetrics with 10 generated structures.

class matbench_genmetrics.core.metrics.MPTSMetrics100(dummy=False, verbose=True)[source]

Bases: MPTSMetrics

Benchmark class for MPTSMetrics with 100 generated structures.

class matbench_genmetrics.core.metrics.MPTSMetrics1000(dummy=False, verbose=True)[source]

Bases: MPTSMetrics

Benchmark class for MPTSMetrics with 1000 generated structures.

class matbench_genmetrics.core.metrics.MPTSMetrics10000(dummy=False, verbose=True)[source]

Bases: MPTSMetrics

Benchmark class for MPTSMetrics with 10000 generated structures.

Module contents

Model benchmarking using validity, coverage, novelty, and uniqueness metrics