If you duplicate a brush(at least in 5.x+) it will let you choose the name, library, and category. By default it will use the user library and the same category and name of the brush you duplicated. You can create new category thru this also. If you decide to change the category just click the dropdown arrow to the right of the tool icon under the tools/brush settings and click Edit metadata. click in catalog and select the category or you can type in a new one. Note: you set the subcategories by using forward slash.
The easier way is to go to the asset browser and change it to the library you have the brush set to and then just create categories and subcategories like normal and then drag and drop the brush into the category.
