Learnwares Search

Learnware Searcher is a key module within the Learnware Market that identifies and recommends helpful learnwares to users according to their user information. The learnware package currently provide two types of learnware searchers:

• homogeneous searchers conduct homogeneous learnware identification and return helpful learnware(s) within the same feature space as the user’s task;

• heterogenous searchers preliminarily support heterogenous learnware identification for tabular tasks, which broaden the search scope and return targeted learnware(s) from different feature spaces.

All the searchers are implemented as a subclass of BaseSearcher. When initializing, an organizer should be assigned to it. The introduction of organizer is shown in COMPONENTS: Market - Framework. Then, these searchers can be invoked with user information provided in BaseUserInfo, and they will return SearchResults containing identification results.

Homo Search

The homogeneous search of helpful learnwares can be divided into two stages: semantic specification search and statistical specification search. Both of them needs BaseUserInfo as input. The following codes shows how to use the searcher to search for helpful learnwares from a market easy_market for a user. The introduction of EasyMarket is in COMPONENTS: Market.

from learnware.market import BaseUserInfo, instantiate_learnware_market
from learnware.specification import generate_semantic_spec, generate_stat_spec

easy_market = instantiate_learnware_market(market_id="demo", name="easy", rebuild=True)

# generate BaseUserInfo(semantic_spec + stat_info)
user_semantic = generate_semantic_spec(
    task_type="Classification",
    scenarios=["Business"],
)
rkme_table = generate_stat_spec(type="table", X=train_x)
user_info = BaseUserInfo(
    semantic_spec=user_semantic, stat_info={rkme_table.type: rkme_table}
)
search_result = easy_market.search_learnware(user_info)

In the above code, search_result is of type dict, with the following specific structure ("single" and "multiple" correspond to the search results for a single learnware and multiple learnwares, respectively):

search_result = {
    "single": {
        "learnware_ids": List[str],
        "semantic_specifications": List[dict],
        "matching": List[float],
    },
    "multiple": {
        "learnware_ids": List[str],
        "semantic_specifications": List[dict],
        "matching": float,
    },
}

Hetero Search

For tabular tasks, homogeneous searchers like EasySearcher may fail to recommend learnwares if no table learnware shares the same feature space as the user’s task, resulting in empty returns. The learnware package preliminarily supports the search of learnwares from different feature spaces through heterogeneous searchers. The process is as follows:

• Learnware markets such as Hetero Market integrate different tabular specification islands into a unified “specification world” by generating new system specifications for learnwares. This allows heterogeneous searchers like HeteroSearcher to recommend tabular learnwares from the entire learnware collection.

• Based on their statistical specifications, users receive new specifications assigned by searchers, which employ methods from the respective organizers. For instance, in Hetero Market, HeteroOrganizer.generate_hetero_map_spec is used to generate new specifications for users.

• Finally searchers conduct statistical specification search across the unified “specification world” based on users’ new specifications and return potentially targeted heterogeneous learnwares.

To activate heterogeneous learnware search, UserInfo needs to include both semantic and statistical specifications. Furthermore, the semantic specification should meet the following requirements:

• The task type should be Classification or Regression.

• The data type should be Table.

• There should be a description for at least one feature dimension.

• The feature dimension mentioned here must align with that in the statistical specification.

The code below demonstrates how to search for potentially useful heterogeneous learnwares from a market hetero_market for a user. For more information about HeteroMarket, see COMPONENTS: Hetero Market.

# initiate a Hetero Market
hetero_market = initiate_learnware_market(market_id="demo", name="hetero", rebuild=True)

# user_semantic should meet the above requirements
input_description = {
    "Dimension": 2,
    "Description": {
        "0": "leaf width",
        "1": "leaf length",
    },
}
user_semantic = generate_semantic_spec(
    data_type="table",
    task_type="Classification",
    scenarios=["Business"],
    input_description=input_description,
)
user_spec = generate_stat_spec(type="table", X=train_x)
user_info = BaseUserInfo(
    semantic_spec=user_semantic,
    stat_info={user_spec.type: user_spec}
)

# search for heterogeneous learnwares in hetero_market
search_result = hetero_market.search_learnware(user_info)