HAL8999 6/100
- Watched the “Learn how to Learn” Google talk on youtube
- Updated the jupyter notebooks for handson-ml from github and read through the Ch2 notebook to address the CategoricalEncoder issue from yesterday
- Looked at a basic transformer
Part of building a data pipeline is likely to include the creation of custom transformer classes to perform operations specific to the project or data source. For example, one of the products I work on stores xml data in a database with the newlines encoded as ‘\n’. When the data is pulled from the database those ‘\n’ sequences are converted to newline characters before the xml is passed to the parser. It’s a very simple operation but without it the data would fail xml validation.
The scikit-learn package provides a structure for building transformers for a data pipeline that is based on duck typing i.e. “looks like a duck, walks like a duck, etc” rather then through object inheritance. Essentially, if your class has fit(X) and transform(X) methods, it counts as a transformer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
from sklearn.base import BaseEstimator, TransformerMixin rooms_idx, bedrooms_idx, population_idx, household_idx = 3, 4, 5, 6 class CombinedAttributesAdder(BaseEstimator, TransformerMixin): """Adds combined attributes rooms_per_household, population_per_household, and bedrooms_per_room""" def __init__(self, add_bedrooms_per_room=True): self.add_bedrooms_per_room = add_bedrooms_per_room def fit(self, X, y=None): """Fit is required even though it does nothing Parameters ---------- X : array-like, shape [n_samples, n_feature] Returns ------- self """ return self def transform(self, X, y=None): """Transform X adding additional attributes. Parameters ---------- X : array-like, shape [n_samples, n_features] The data to encode. Returns ------- np.c_ : concatenation of X and additional attribute columns """ rooms_per_household = X[:, rooms_idx] / X[:, household_idx] population_per_household = X[:, population_idx] / X[:, household_idx] if self.add_bedrooms_per_room: bedrooms_per_room = X[:, bedrooms_idx] / X[:, rooms_idx] return np.c_[X, rooms_per_household, population_per_household, bedrooms_per_household] else: return np.c_[X, rooms_per_household, population_per_household] attr_adder = CombinedAttributesAdder() housing_xtra_attrs = attr_adder.transform(X) |
