Usage | RMIS Benchmark

Path Setup & Meta-Data Generation

After downloading the code, the data (convert to 10s wavs) and model checkpoints, fill up the paths to these datasets and checkpoints in conf/basic.yaml. Then generate the meta-data for all datasets by running:

                    # Generate meta-data for all RMIS datasets

                    python -m runner.meta_data.rmis --ds all

Test Integrated Models

Now you can test integrated models by easily running one command!

Standard Evaluation (RMIS Score)

Calculate the RMIS score across all datasets.

                        python -m rmis.scripts.reg_all \

                        --model_conf rmis/model_conf/{model}/{variant}.yaml \

                        --rel_exp_dir {rel_exp_dir} \

                        --gpu 0,1,2,3

Additional Multi-Split Experiment

Evaluate using multi-split ratios to test generalization.

                        python -m rmis.scripts.multi_split_all \

                            --model_conf rmis/model_conf/{model}/{variant}.yaml \

                            --rel_exp_dir {rel_exp_dir} \

                            --gpu 0,1,2,3

Note: RMIS is designed for Linux. Windows may encounter some pathing compatibility issues.

Test Your Model

The RMIS codebase is designed to readily integrate new models. You can test your model on the RMIS benchmark in just two steps!

1

Implement your model

Create a folder for your model models/your-model, add the source code of your model inside.
Then define a model wrapper in models/your-model/main.py` to wrap your model for RMIS:

                        class MyAwesomeModel(nn.Module):

                        def __init__(self, **kwargs):

                            self.model = mymodel(**kwargs)

                        def forward(self, x: torch.Tensor):

                            # x is the input signal, can be wav, stft, fbank, etc. specify in model_conf

                            x = self.model(x)

                            return {'embedding': x}

Then, register your model in models/model_wrapper.py. You can refer to how integrated models are implemented.

                        elif args['model_name'] == 'your-model':

                        from models.your-model.main import MyAwesomeModel

                        net = MyAwesomeModel(**model_conf)

2

Define Inference Configurations

Create a folder in rmis/model_conf and add inference configurations in rmis/model_conf/your-model. It should contain five parts as presented in template.yaml. You can also refer to the configurations of integrated models.

Data

Sample rate, feature type (wav, stft, fbank).

Preprocess

Spectrogram normalization.

Model

Model name, model configurations (for setting up your model).

Inference

Batch size.

Dataset Specific

Adaptive settings for specific datasets within the benchmark.

Ready to Evaluate?

Once you have completed all above steps, you can test your model with a single command!

                    python -m rmis.scripts.reg_all --model_conf rmis/model_conf/{your-model}/{variant}.yaml --gpu 0,1,2,3
                