I have to write front-end form in my component with dropdown list of categories and subcategories. Categories and subcategories amount should be unlimited. Should I use a nested sets model to do that?
If you implement your own categories functionality you can use anything that suits you.
However you can take advantage of Joomla's built-in categories system for your component.
Have a read here:
Developing an MVC Component/Adding categories